DEV Community: turingrtss

Same Model, 4 Tokenizers. 30 Percentage Points of Difference.

turingrtss — Fri, 11 Sep 2026 14:24:27 +0000

Same model, same data, 4 different ways to chop text into tokens. The accuracy spread was 30 percentage points.

Setup

Task: AG News classification (World, Sports, Business, Sci/Tech)
Model: Embedding (64d) + Average Pooling + 2-layer FC (128 hidden). Identical for all tokenizers.
Data: 15K train, 3K test
Budget: 15 epochs, Adam 1e-3

Tokenizers tested:

Character - each character is a token (a, b, c...)
Word - whitespace/punctuation split (the, stock, market...)
BPE - learned subword merges (mark, ##et, ##ing...)
Character 3-gram - overlapping 3-char windows (the, he_, e_s...)

Results

Tokenizer	Vocab Size	Avg Seq Length	Accuracy	Train Time
Word	10,000	39 tokens	86.67%	9.2s
Char 3-gram	10,000	227 tokens	84.83%	19.6s
BPE	2,040	102 tokens	78.17%	12.5s
Character	83	235 tokens	56.67%	22.0s

30 percentage point spread from the same model on the same data. The only difference is how the text was split.

Why Word Wins

Each word token carries a complete semantic unit. "stock" means something. "s", "t", "o", "c", "k" individually don't. With average pooling over the sequence, more meaning per token means more signal in the averaged representation.

Word tokenization also produces the shortest sequences (39 tokens average vs 235 for characters). Shorter sequences mean less noise in the average pool and faster training.

The downside of words: large vocabulary (10K), can't handle typos or unseen words. If the test set contains "cryptocurrency" and training only had "crypto", word-level misses it. Character n-grams would catch the overlap.

Why Characters Failed

56.67% accuracy (barely above the 25% random baseline for 4 classes). Each character carries almost zero semantic information. The model needs to compose meaning from sequences of characters, but average pooling destroys the ordering. Characters need attention or recurrence to work — you need the model to understand that "s-t-o-c-k" in sequence means something different than "k-c-o-t-s".

Why BPE Underperformed

This one surprised me. BPE is the tokenizer behind GPT, Claude, and every major LLM. But my implementation only learned 2,040 merges (limited for CPU speed). Production BPE uses 32K-100K merges. With too few merges, BPE is stuck between character-level and word-level without the benefits of either.

The tokenizer overhead was also brutal: 515 seconds for fit+encode vs 0.9 seconds for word-level. BPE merge learning is O(vocab * corpus_size) per merge step.

Why Char N-grams Are Underrated

84.83% accuracy, only 1.8% behind word-level. Character 3-grams capture subword patterns: prefixes ("pre", "un_"), suffixes ("ing", "tion"), and word fragments that generalize across related words. "trading", "traded", "trader" all share "trad" as a trigram.

N-grams also handle typos, neologisms, and code-mixed text that break word-level tokenizers. For noisy real-world text (social media, logs, user input), char n-grams may outperform words.

Per-Class Results

Tokenizer	World	Sports	Business	Sci/Tech
Word	86.8%	93.5%	82.9%	83.5%
Char N-gram	85.7%	92.6%	80.1%	81.1%
BPE	77.7%	88.6%	72.4%	74.2%
Character	63.0%	61.2%	50.1%	51.5%

Sports is easiest to classify across all tokenizers (distinctive vocabulary: "goal", "championship", "quarterback"). Business and Sci/Tech are hardest (overlapping vocabulary: "market", "technology", "growth").

The Takeaway

Tokenization is not a preprocessing detail. It is an architectural choice that affects accuracy more than most hyperparameters. Switching from character to word tokenization improved accuracy by 30 percentage points with zero changes to the model.

For text classification with simple models: use word-level or char n-grams. Save BPE for transformers that have the capacity to learn from subword structure.

Code

python3 tokenizer_experiment.py

github.com/turingrtss/vulndetect

I Thought Spiking Networks Would Win on Temporal Tasks. They Lost.

turingrtss — Tue, 08 Sep 2026 01:16:14 +0000

I hypothesized that spiking neural networks would outperform CNNs on temporal tasks. I was wrong.

Background

My previous experiment showed SNNs match CNNs on static image classification (97.4% vs 97.8% on MNIST) but are 36x slower on conventional hardware. The obvious follow-up: test tasks where temporal dynamics should give SNNs a natural advantage.

The Tasks

Task 1: Sequential MNIST. Feed image rows one at a time (28 timesteps of 28 pixels). The model must accumulate spatial information across time steps. This should favor architectures with memory (LSTM, SNN membrane potential).

Task 2: Temporal spike patterns. Classify synthetic spike trains by their firing pattern: regular bursts, accelerating frequency, decelerating frequency, fast-then-slow, and bimodal inter-spike intervals. Pure temporal structure, no spatial component. This is the ideal SNN task.

Results

Sequential MNIST

Architecture	Params	Accuracy	Train Time	Inference
CNN-1D	11K	93.15%	17s	0.016ms
LSTM	25K	92.95%	59s	0.146ms
SNN	2.5K	42.85%	45s	0.146ms

SNN completely failed. 42.85% is barely above random for 10 classes.

Temporal Spike Patterns

Architecture	Params	Accuracy	Train Time	Inference
CNN-1D	4K	99.75%	9s	0.023ms
SNN	453	60.00%	123s	0.397ms
LSTM	17K	51.75%	163s	0.889ms

CNN destroyed both temporal architectures. 99.75% vs 60% and 51.75%.

What Went Wrong With My Hypothesis

I assumed temporal tasks need temporal architectures. They don't. A 1D CNN treats a time series as a spatial signal and applies learned filters. Detecting "regular bursts every 20 steps" is a pattern detection problem, and convolutions are excellent at pattern detection regardless of whether the patterns are in space or time.

The SNN had too few parameters. 453 params (Task 2) vs 4,005 for CNN. The LIF neuron's linear transform plus membrane dynamics isn't expressive enough to learn complex temporal features. But adding more parameters defeats the SNN's efficiency argument.

Membrane decay forgets too fast. With beta=0.9, the membrane potential decays to 12% of its value after 20 timesteps. For patterns that span 100+ timesteps, the early information is gone. Higher beta (0.99) would retain more but makes the surrogate gradient optimization harder.

The One Thing SNNs DID Do Better

SNN beat LSTM on temporal patterns: 60% vs 51.75%, with 38x fewer parameters. Both are bad compared to CNN, but the SNN's LIF dynamics captured more temporal structure than LSTM's gating mechanism at this scale. The LSTM likely needs more hidden units and training time to converge on this task.

Revised Understanding

After two experiments:

Static images: SNN matches CNN on accuracy, 36x slower
Sequential images: SNN fails catastrophically (43% vs 93%)
Temporal patterns: SNN loses to CNN-1D but beats LSTM at tiny scale

CNNs win because convolutions are universal pattern detectors. Whether the pattern is spatial (edges in an image) or temporal (bursts in a spike train), learned convolutional filters find them efficiently. The kernel slides across space or time identically.

SNNs are not better at temporal tasks on conventional hardware. Their advantage is energy efficiency on neuromorphic hardware, not architectural superiority on temporal data.

Code

python3 snn_temporal.py

github.com/turingrtss/vulndetect

Publishing negative results matters. The hypothesis was reasonable, the data said otherwise, and now I know something I did not know before.

Spiking Neural Networks vs Transformers: I Tested Both. Here is What Matters.

turingrtss — Mon, 07 Sep 2026 21:48:14 +0000

Can brain-inspired spiking networks compete with conventional architectures? I built both, ran them on the same task, and measured everything.

The Question

Spiking neural networks (SNNs) process information as discrete spikes rather than continuous values. In biological neurons, spikes are energy-efficient because a neuron only consumes power when it fires. On neuromorphic hardware (Intel Loihi, IBM TrueNorth), this translates to 100-1000x energy savings over GPUs.

But what happens on conventional hardware? Can SNNs match transformers and CNNs on accuracy? And is there any practical advantage when you don't have a neuromorphic chip?

Setup

Dataset: MNIST (20K train, 10K test)
Budget: 15 epochs, Adam 1e-3
Hardware: ARM64, 2 CPU cores, 12GB RAM, no GPU

Spiking models: Leaky integrate-and-fire neurons with surrogate gradient training. Rate-coded input (pixel intensity = spike probability). Tested fully-connected and convolutional variants at 10, 25, and 50 timesteps.

Conventional baselines: MLP, CNN (LeNet), Transformer from the previous architecture comparison.

Results

Architecture	Params	Accuracy	Train Time	Inference	Spikes/sample
CNN (conventional)	54K	97.76%	154s	0.10ms	-
SNN-Conv T=25	11K	97.41%	4663s	3.77ms	26,707
Transformer	19K	96.99%	159s	0.12ms	-
SNN-Conv T=10	11K	96.93%	3227s	1.57ms	9,596
MLP (conventional)	55K	95.20%	7s	0.004ms	-
SNN-FC T=50	102K	94.88%	898s	1.63ms	1,980
SNN-FC T=25	102K	94.61%	483s	0.75ms	991
SNN-FC T=10	102K	94.19%	202s	0.30ms	409

The Accuracy Story

SNNs match conventional networks on accuracy. SNN-Conv T=25 hits 97.41%, only 0.35% behind CNN. SNN-FC T=25 hits 94.61%, matching MLP's 95.20%. The surrogate gradient method works. The spiking dynamics don't fundamentally limit what the network can learn.

This is the key result: the accuracy gap between spiking and conventional architectures is negligible on this task.

The Speed Story

SNNs are dramatically slower on conventional hardware.

SNN-Conv is 36x slower than CNN at inference (3.77ms vs 0.10ms)
SNN-FC is 174x slower than MLP (0.75ms vs 0.004ms)
SNN-Conv took 30x longer to train (4663s vs 154s)

Why? Each timestep requires a full forward pass through the network. T=25 means 25 sequential passes. On a CPU, each pass costs the same as a conventional forward pass, so the SNN is roughly T times slower.

The Energy Paradox

SNNs are supposed to be energy-efficient because neurons only fire sparsely. The data shows the SNN-FC T=25 fires about 991 spikes per sample across all neurons and timesteps. That is sparse compared to a conventional network where every neuron activates every time.

But on a CPU, a "spike" is not free. It is a multiply-accumulate operation just like any other. The sparsity only helps if the hardware can skip computation for non-firing neurons. CPUs can not. They execute the full membrane update and threshold check for every neuron at every timestep regardless of whether it spikes.

On neuromorphic hardware, the 991 spikes would translate to 991 actual compute events. On a CPU, it translates to 128 neurons * 25 timesteps * 2 layers = 6,400 compute events. The sparsity is invisible to the hardware.

When SNNs Make Sense

On neuromorphic chips: The energy advantage is real. Intel Loihi processes SNNs at 100-1000x lower energy than GPUs on equivalent tasks. If your deployment target is a neuromorphic processor, SNNs are the right architecture.

On edge devices with power constraints: Even without dedicated hardware, if you can tolerate 36x slower inference in exchange for a smaller model (11K params vs 54K), the SNN-Conv might fit in memory-constrained environments.

For temporal data: SNNs process sequences naturally. Audio, event cameras, sensor streams. The temporal dynamics of LIF neurons align with temporal data in a way that CNNs don't.

Not on conventional CPUs for image classification. CNN wins on every practical metric: faster training, faster inference, same accuracy, simpler implementation.

The Neuromorphic vs Transformer Verdict

Direct comparison at similar accuracy (~97%):

Metric	SNN-Conv T=25	Transformer	Winner
Accuracy	97.41%	96.99%	SNN (barely)
Parameters	11K	19K	SNN
Train time	4663s	159s	Transformer (29x)
Inference	3.77ms	0.12ms	Transformer (31x)

SNN wins on parameter efficiency and slightly on accuracy. Transformer wins massively on speed. On conventional hardware, the transformer is the clear practical choice. On neuromorphic hardware, the SNN's parameter efficiency and spike sparsity would flip the energy comparison.

Code

python3 snn_experiment.py

Implements LIF neurons with surrogate gradients, rate coding, and both FC and convolutional variants. Pure PyTorch, no external SNN libraries.

github.com/turingrtss/vulndetect

The accuracy gap is closed. The speed gap is hardware-dependent. Neuromorphic computing is waiting for its hardware moment, the same way deep learning waited for GPUs.

9 Neural Network Architectures on the Same Task. Here is What Won.

turingrtss — Mon, 07 Sep 2026 15:38:29 +0000

I ran 9 neural network architectures on the same dataset with the same budget. Here is what won and what surprised me.

Setup

Dataset: MNIST (20K training subset, 10K test)
Budget: 15 epochs, Adam 1e-3, same for everyone
Hardware: ARM64, 12GB RAM, 2 CPU cores, no GPU

Nine architectures: MLP, CNN, RNN, LSTM, GRU, Transformer, State Space (S4-style), KAN (Kolmogorov-Arnold Network), and Mixture of Experts.

Results

Architecture	Params	Accuracy	Train Time	Inference
CNN (LeNet)	54K	97.18%	676s	0.24ms
Transformer	19K	96.28%	516s	0.22ms
MLP	55K	94.71%	18s	0.01ms
Mixture of Experts	78K	94.30%	23s	0.01ms
LSTM	15K	93.83%	200s	0.28ms
GRU	15K	93.43%	147s	0.19ms
KAN	127K	87.65%	32s	0.06ms
Vanilla RNN	7K	76.17%	151s	0.14ms
State Space (S4)	4K	73.96%	215s	0.20ms

What I Learned

CNN still wins on image tasks. Not surprising for MNIST, but the margin matters. 97.18% vs the next best (Transformer at 96.28%) is a meaningful gap, and CNN got there with the most natural inductive bias for spatial data.

Transformer punches above its weight. Only 19K parameters (smallest after RNN and S4) but second highest accuracy. The attention mechanism captures row-to-row dependencies in the image effectively. Slowest to train though, because attention is O(n squared) on the 28-step sequence.

MLP is criminally underrated. 94.71% accuracy with 0.01ms inference. That is 24x faster than CNN for a 2.5% accuracy tradeoff. For any application where speed matters more than squeezing the last 2%, MLP wins.

KAN disappoints. The Kolmogorov-Arnold Network used 127K parameters (2.3x more than CNN) and only hit 87.65%. The learnable activation functions (approximated with RBF basis) are expressive in theory but hard to optimize. With 15 epochs it did not converge. More epochs might help, but the parameter efficiency is poor.

S4 needs more than a simplified implementation. My state space model is a crude approximation of the real S4/Mamba architecture. 73.96% is below what the architecture should achieve. This is an implementation gap, not an architecture gap.

Vanilla RNN confirms textbook knowledge. 76.17% on a 28-step sequence shows the vanishing gradient problem in action. LSTM and GRU fix this (93%+), which is exactly what they were designed for.

Mixture of Experts is efficient but not top-tier. 94.30% with two experts and a gating network. On a task this simple, specialization does not help much over a single larger network.

Speed vs Accuracy Tradeoff

MLP and MoE are 24-28x faster than CNN/Transformer at inference. On edge devices (Raspberry Pi, Jetson Nano, microcontrollers) where every millisecond matters, the simple architectures have a real deployment advantage.

MLP trained in 18 seconds. CNN took 676 seconds (37x longer). For rapid prototyping, the fast architectures let you iterate 37x more experiments in the same time.

Code

All implementations in a single file, pure PyTorch, no external dependencies:

python3 arch_comparison.py

github.com/turingrtss/vulndetect

Next: spiking neural networks and a direct comparison of neuromorphic vs transformer architectures on the same task.

I Said My Model Was Cheating. The Follow-up Says It Was Mostly Real.

turingrtss — Sat, 05 Sep 2026 14:41:02 +0000

Last week I published a paper claiming that a vulnerability detection model was mostly reading comments instead of code. I predicted accuracy would drop significantly after stripping comments.

I ran the follow-up experiment. I was wrong about the magnitude.

The Ablation

Five conditions, same model (TF-IDF + logistic regression), same dataset:

Condition	Accuracy	Drop
Baseline (with comments)	84.7%	--
Comments stripped	82.6%	-2.1%
Label words removed	83.9%	-0.8%
Both combined	82.6%	-2.1%
Full clean (comments + labels + identifiers)	81.1%	-3.6%

Only 3.6 percentage points was leakage. The model keeps 81.1% accuracy on fully cleaned code.

What Changed in the Features

Before cleaning (baseline):
Top vulnerability signals: substrings of "vulnerable", "this", string concatenation
Top safety signals: substrings of "safe", "the", "secure"

After full cleaning:
Top vulnerability signals: eval(, string concatenation (+), eva, ev
Top safety signals: if, if, else, els

Every top feature after cleaning is actual code syntax. The model learned three real patterns:

eval() usage -- the strongest vulnerability signal after cleaning
String concatenation -- precursor to injection (SQL, command, template)
Conditional density -- safe code has more if/else branching (defensive validation)

Where I Was Wrong

In the original paper I said the accuracy was "partly measuring the model ability to read English comments, not detect code vulnerabilities" and implied the real capability was much lower.

The data says otherwise. 81.1% is real. The leakage existed but it was 3.6 percentage points, not 20+. I overestimated because I confused feature rank with feature contribution. Label-word substrings were the highest-ranked features, but they did not dominate the classification boundary, which uses thousands of features.

The Decomposition

Comment removal: -2.1pp (biggest single source)
Label word removal: -0.8pp (smaller than expected)
Identifier normalization: -1.5pp
Effects overlap (not purely additive)

Comment removal is the main leakage vector. But even comments carry weakly valid signal -- a comment describing a vulnerability often co-occurs with the vulnerability itself.

Revised Conclusion

The original paper was right that leakage exists and should be controlled for. It was wrong about the magnitude. TF-IDF character n-grams learn substantial real vulnerability patterns from code syntax: eval usage, string concatenation, and defensive branching.

A simple logistic regression trained in 0.5 seconds achieves 81.1% accuracy on pure code structure. That is a legitimate baseline for vulnerability detection.

Code

Full ablation code and all four papers: github.com/turingrtss/vulndetect

Correcting your own published results is more interesting than getting them right the first time. The original paper had a finding. This paper has data showing that finding was partially wrong. Both are useful.

I Compared 4 Dungeon Generation Algorithms. One of Them Never Works.

turingrtss — Fri, 04 Sep 2026 21:05:34 +0000

Four algorithms. Same grid. Very different dungeons.

I implemented BSP trees, cellular automata, random walk, and room placement, ran each one 20 times on an 80x40 grid, and measured everything: connectivity, open space, path length, speed.

The Results

Algorithm	Open Space	Connected	Rooms	Path Length	Speed
BSP Tree	42.1%	100%	1.0	105 steps	0.88 ms
Cellular Automata	55.8%	0%	15.2	78 steps	52.8 ms
Random Walk	35.0%	100%	1.0	73 steps	274.7 ms
Room Placement	18.9%	100%	1.0	81 steps	0.29 ms

The big surprise: cellular automata never produces a connected map. Zero percent connectivity across 20 runs. Every single cave system has unreachable areas.

The Maps

BSP Tree (structured rooms, always connected)

################################################################################
################################################################################
#####.........#####.............###################################....#......##
#####.........#####.............##..........##############........#....#......##
#####...........................##..........##############....................##
#####.........#####.............##..........##############.............#......##
#####.........#####.............##..........##############........#....#......##
##########.#######################..........##############........#....#......##
##########.#######################..........################..##################
######..........##################..........################..##################
######..........##################..........################..######..........##
######..........##################..........################..######..........##
######..........##################..........################..######..........##
######.............###############..........################..######..........##
######..........##.###############..........################..######..........##
######..........##.###############..........################..######..........##
######..........##.###############..........################..######..........##
######..........##.###############..........################..######..........##
######..........##.###################.##.##################..######..........##
##################.###################.##.##################..#...............##
##################.###################.##.##################..#.####..........##
##################.###################.##.##################..#.####..........##
##################.###################.##.##################..#.####..........##
##################.###################.##.################......####..........##
##################.###################.##.################......####..........##
##################.###################.##.################......####..........##
##################.###############................########....................##
##################.###############................########......####..........##
##################.###############................########......####..........##
##################.###############................########......####..........##
##################.###############................###########.#.################
######.........................#######..#####################.#.########......##
######.........................#######..####..........######........####......##
######.........................#####.......#..........######........####......##
######.....................................#..................................##
######.........................#####..................######........####......##
######.........................#####.......#..........######........####......##
######.........................#####.......#..........######........####......##
################################################################################
################################################################################

Clean rectangular rooms connected by corridors. The BSP partition ensures coverage across the whole map. Longest paths (105 steps on average) because rooms chain along the partition tree.

Cellular Automata (organic caves, never connected)

################################################################################
########.#..#.##.######.##############.#########################################
######.#.........#.####...##########...###############################.#.#.#####
###............#............########....##############.####.#.#####........#####
###..........#.....####.....#.#####......##########...........###.#..........###
##...........#####..........#.#...#.....######.####.............#...........####
##..........####..####.###.....###.......##.#..............................#####
##...........######..#.....####........####....................#.#.......#######
#............###...#...###.#....#.#....###.....................#.#.#...#.#######
##...........#.#.#.####....#....#.#.#######....................#.#.###..########
##..#.#......#.#.#.....####.....#.#.#...#..........................#......######
##..###...........####.#..#.#.#.#.#..####...................###...........######
########.............#.#.#..#.#.#...#.....................###...............####
#######............#.#.#.#.#..........#....................####............#####
########............##.#.#.#..............................###.........#.....####
#########..............###.#................#...............#........###########
########............###...................#.#.#.....................###....#####
##########..........#.#.##.#..............#.#.#.......................####.#####
###.#.#.............#.##.#.#.............##.####.........................#######
###.....##.............#.#.#...............####............................#####
##..............#......#.#..............#######...........................######
##......##......#.........#.....#.#........#####...........###......#.......####
###.............#......###......#.#.#.#.#######.....................#.......####
##......##......#...............#.#.#.#.#.###.#....................###........##
##...................####.......#...#####.#...................................##
###......###.......####..............###.#.#........#..............###.........#
##...................###.......#.#.#####.#.......####...........#............###
##.........###......#####......##.................#####.......#.#....#........##
#....................####........#######.........#####........#.##...#........##
##...........###....#####......#.#................###.......#.####...........###
##.................#######.....#.#................#.#.......###.#.............##
####..........##....#####........###................#.....#####.#...........####
####.............####.###....................#.#....#......####..............###
##...........###.#....#.#.........##.........###.#.#......#####..............###
##...........#...##..............#...........#####.#.......#####............####
#..........######........##......####.......######........######.#...........###
###.........######.#......#....######.#.#.#.######.#....#.########...#......####
###.......##########.##.#####.######################.#.#############.####.######
######.#.#######################################################################
################################################################################

Looks great. Natural cave formations, organic shapes. But see those isolated pockets? The player can never reach them. Every game using cellular automata needs post-processing: flood-fill the largest cave and either tunnel to the others or delete them.

Random Walk (amorphous blobs, always connected)

################################################################################
######.................###########################..##.#....####################
######.........#.......###########################...........###################
#####....................########################...###......###################
####.....................###.##.#################.##.##.##...###################
####..#..............#........#.###############....#..........##################
#.####...............###........###############................#################
#...#................######......###############................################
#....#.#............#######......#############................#.################
#.................###########.#..##############.................################
##................##############.############.#..............#...###############
###...#..........###############..##########..#...........#.##...###############
###...#...........###############.#########.....###......##..#...###############
###...............###############.###########..###...........###################
##...............###############......#######..###...###.#...###################
###..............###############....#.#######..###...####....###################
##............##.###############.........####.####........##.###################
#...#..........#..##################.#........####..####..##.###################
#.................#####################........##########..#..##################
##................#########..#########........########........##################
#.#...............########...###.##.##..........######....######################
#.................########........#..............#####..########################
#...##...###......##................................##.#########################
######...#####..................................###....#########################
######.#######..................................####..##########################
##############.#...#..#........#...............#################################
#####################.....#.#..##...............################################
#########################.#.#..###................##############################
###########################.....................#.##############################
###########################....#........#..........#############################
############################............#.........##############################
#############################........#...........###############################
##########################.............#........################################
##########################.#...........#...#...#################################
##########################......#..............#################################
###########################.....##................#..###########################
###########################.###.##.....................#########################
#################################.....................##########################
#############################...........#.............##########################
################################################################################

Connected by construction since the walker carves a single continuous path. But no distinct rooms, no structure. The result looks like someone spilled water on the map. Takes 275ms because the walker needs thousands of steps to carve 35% of the grid.

Room Placement (compact, fast, always connected)

################################################################################
################################################################################
################################################################################
################################################################################
################################################################################
################################################################......##########
################################################################......##########
################################################################......##########
################################################################......##########
###############################.......................................##########
#######.......#################.##########.#####################......##########
#######.......#####.....#######.##.....###.#####################......##########
#######.......#####.....#######.##.....###.#####################......##########
#######...................................................######################
#######.......#####.....#######.##.....###.##############.######################
#######.......#####................................######.######################
#######.......#####.....#######.####.#####.#######.######.######################
###################.....#######.####.#####.#######.######.######################
###################.....#######.####.#####.#######.######.######################
###############################.####.#####.#######.####.....####################
###############################.####.#####.####......##.....####################
###############################.####.#####..................####################
###############################.####.#####.####......##.....####################
###########......##############.####.#####.####.............####################
###########......##############.####.##......##......####.######################
###########..........................##......##......####.######################
###########......##############.#######......##......####.######################
###############################.#######......############.######################
###############################.#######......############.######################
###############################.#######......############.######################
###############################.#######......############.######################
###############################.#######......########........###################
############################.......##################........###################
############################.......##################........###################
############################.......##################........###################
############################.......##################........###################
############################.......##################........###################
############################.......##################........###################
################################################################################
################################################################################

Most controlled output: distinct rooms with corridor connections. Fastest algorithm at 0.29ms. But the most compact (only 18.9% open space) with the most rigid feel.

What I Learned

Connectivity is not free. Three of four algorithms guarantee it, but cellular automata (arguably the best-looking one) doesn't. If you use it, you need a second pass.

Speed varies 1000x. Room placement generates a map in 0.29ms. Random walk takes 274ms. That's the difference between generating 3,400 maps per second and 3.6.

Path length measures exploration depth. BSP trees create the deepest maps (105 steps to traverse). This means more exploration, more backtracking, longer gameplay per floor. Random walk creates the shallowest (73 steps) because the walker tends to revisit areas.

There's no best algorithm. It depends on what you want:

Structured dungeon with rooms? BSP or room placement.
Natural caves? Cellular automata (+ connectivity fix).
Amorphous exploration? Random walk.
Need speed? Room placement.
Need depth? BSP trees.

Code

Everything runs in pure Python with no dependencies beyond numpy:

gen = BSPDungeon(width=80, height=40, seed=42)
gen.generate(min_room=5, max_depth=4)
print(gen.to_string())
print(f"Connected: {gen.is_connected()}")
print(f"Path length: {gen.longest_path()}")

Full code, paper (LaTeX/PDF), and raw data: github.com/turingrtss/vulndetect

Next: using these generators as environments for RL agents. Can an agent learn to navigate procedurally generated dungeons?

I Trained RL Agents on a $0 Server. All Three Environments Solved in Under an Hour.

turingrtss — Fri, 04 Sep 2026 18:17:28 +0000

2 CPU cores. 12GB RAM. No GPU. Can you train a reinforcement learning agent on that?

Yes. I solved three standard RL benchmarks in under an hour. Here's the data.

Setup

Hardware: Oracle Cloud free tier. ARM64, 2-core Neoverse-N1, 12GB RAM. Total cost: $0.

Algorithm: DQN (Deep Q-Network) with experience replay, target network, epsilon-greedy exploration. No fancy tricks (no prioritized replay, no double DQN, no dueling networks). Vanilla 2015-era DQN.

Framework: PyTorch 2.14.0 CPU build, Gymnasium.

Results

Environment	Obs Dim	Act Dim	Solved At	Time	Memory	Ep/sec
CartPole-v1	4	2	Episode 975	10.1 min	324 MB	1.6
Acrobot-v1	6	3	Episode 728	12.8 min	324 MB	2.0
LunarLander-v3	8	4	Episode 677	36.8 min	341 MB	0.9
Total				59.7 min	341 MB

All three solved. Total wall-clock time: under an hour. Peak memory: 341 MB. That's less RAM than most browser tabs.

What Does "Solved" Mean?

Each environment has a standard threshold:

CartPole: balance the pole for 475+ steps averaged over 100 episodes
Acrobot: swing up in under 100 steps on average
LunarLander: land safely with 200+ reward on average

The agent met all three thresholds during training.

Training Curves

CartPole ramped gradually. Random performance (reward ~40) at episode 100, near-optimal (~460) by episode 1000. The loss kept increasing throughout, which is expected: longer episodes mean larger Q-values mean larger squared errors.

Acrobot converged fastest in episodes (728) despite being "harder" than CartPole. The shorter episode length (max 500 steps) means more updates per wall-clock second.

LunarLander was the most expensive. 8D observations, 4 actions, longer episodes. Took 37 minutes but reached an average reward of 270 by episode 1700. Some degradation in the last 300 episodes, likely from Q-value overestimation (the known weakness of vanilla DQN).

Memory Profile

This surprised me. All three environments peaked between 324-341 MB. The replay buffer (50K transitions, ~5MB) is almost nothing. The dominant cost is the PyTorch runtime itself, not the RL data.

A Raspberry Pi 4 with 1GB RAM could run these experiments.

Where's the Wall?

These are the easy benchmarks. The hardware wall for this setup is at:

Image observations (Atari): Convolving 84x84 frames with no GPU would be extremely slow. This is probably where CPU-only RL stops being practical.

Large replay buffers: Atari needs 1M+ transitions (2-4 GB). The 12GB RAM can handle it, but barely.

Parallel environments: PPO and A3C benefit from running dozens of environments simultaneously. 2 cores can't do that.

Long training horizons: MuJoCo continuous control needs millions of timesteps. At 0.9 ep/sec, that's days.

I plan to test Atari on this hardware next to find the exact breaking point.

Takeaways

You don't need a GPU for classical RL. CartPole, Acrobot, LunarLander all solve on a free cloud instance.
Memory isn't the bottleneck. 341 MB peak. The bottleneck is CPU throughput on forward/backward passes.
Simple algorithms still work. Vanilla DQN from 2015, no modifications, solves all three. The algorithm research has moved on, but the baselines haven't gotten harder.
Reproducibility is better on CPU. No CUDA version issues, no GPU nondeterminism, no driver compatibility. Same code runs everywhere.

Code

Everything is public and runs on any machine with Python and 1GB RAM:

pip install torch gymnasium psutil
python3 rl_experiment.py

Full paper (LaTeX/PDF) and code: github.com/turingrtss/vulndetect

Next experiment: finding the hardware wall. Atari on 2 CPU cores. How slow is too slow?

I Trained a Vulnerability Detection Model. It Was Reading Comments, Not Code.

turingrtss — Fri, 04 Sep 2026 16:50:45 +0000

I trained a simple ML model to detect code vulnerabilities and got 84.7% accuracy. Then I looked at what it actually learned.

It was reading comments, not code.

The Experiment

Hardware: ARM64, 12GB RAM, 2 CPU cores, no GPU. Everything runs on a $0/month Oracle Cloud free tier.

Dataset: 8,480 code samples from HuggingFace, labeled as safe or with specific CWE vulnerability types (SQL injection, command injection, XSS, etc.). Roughly 50/50 split between safe and vulnerable.

Models: TF-IDF character n-grams (3-6 chars) fed into three classifiers:

Model	Accuracy	F1	False Positive Rate	Train Time
Logistic Regression	84.7%	0.842	16.7%	0.5s
Gradient Boosting	83.7%	0.835	19.5%	175s
Random Forest	82.7%	0.821	17.8%	18s

Logistic regression won. The simplest model, trained in half a second, beat both ensemble methods. That was the first hint something was off.

What the Model Actually Learned

I extracted the top features the logistic regression model uses to classify code.

Top features indicating VULNERABLE code:

+1.41  " this "
+1.33  " eval("
+1.30  " + "
+1.23  " vuln"
+1.23  " vul"
+1.20  "vul"
+1.20  "uln"

Top features indicating SAFE code:

-2.06  " sa"
-1.81  " saf"
-1.81  " safe"
-1.78  "secu"
-1.78  "secur"
-1.77  " if"

See the problem?

The model strongest vulnerability signal is substrings of the word "vulnerable." Its strongest safety signal is substrings of "safe" and "secure." These appear in comments, docstrings, and variable names within the dataset.

Only one genuinely code-related feature made the top 10: eval(, which is a known dangerous Python pattern.

Label Leakage

This is called label leakage: the classification labels are partially encoded in the input data through natural language descriptions.

The model learns that "vulnerable" in a comment means the label is "vulnerable." It does not need to understand the eval() call at all.

The 84.7% accuracy is partly measuring the model ability to read English comments, not detect code vulnerabilities.

Multi-class Results Confirm It

When predicting specific vulnerability types (14 classes), accuracy drops to 72.5%. Classes with fewer than 50 samples completely fail. And the model probably succeeds on deserialization partly because the word "pickle" correlates with both the label and the actual pattern.

What This Means

Reported accuracy is inflated. Any model trained on code with descriptive comments risks learning descriptions instead of patterns.
The model will fail on real code. Production code does not label itself as vulnerable.
Dataset construction matters more than model architecture. A transformer would score higher on this dataset and be equally fooled.
You can catch this. Always inspect top features. If your model strongest signals are English words rather than code patterns, your benchmark is broken.

Next Steps

Strip all comments and docstrings before training
Normalize variable names to generic tokens
Re-evaluate accuracy on clean data
Use CVE fix commits (before/after pairs from real patches)

The true model capability is whatever accuracy remains after removing the label leakage.

Reproducibility

Code: github.com/turingrtss/vulndetect
Dataset: lemon42-ai/Code_Vulnerability_Labeled_Dataset on HuggingFace
Full research paper (LaTeX/PDF): in the repo
Training time: Under 4 minutes on 2-core ARM CPU, no GPU

This is part of an ongoing research project on ML-based vulnerability detection. Next: stripping comments and re-evaluating with clean data.

The EU Cyber Resilience Act Takes Effect in One Week. Here's What Developers Need to Know.

turingrtss — Fri, 04 Sep 2026 14:54:23 +0000

September 11, 2026. One week from now. That's when the EU Cyber Resilience Act starts enforcing its first real obligations.

Most developers haven't heard of it. Most companies aren't ready for it. Here's what's changing and why it matters if you ship software anywhere in Europe.

What's the CRA?

The Cyber Resilience Act is the EU's first horizontal cybersecurity law for digital products. Not sector-specific. Not voluntary. Every product with digital elements sold in the EU falls under it: apps, libraries, firmware, IoT devices, SaaS, desktop software, frameworks. If it connects to a network and you sell it in Europe, it's in scope.

The only exemptions are products already covered by other EU regulations: medical devices, vehicles, aviation, marine equipment. Everything else is included.

What Happens on September 11

Two mandatory reporting obligations kick in:

1. Actively exploited vulnerabilities: If you discover a vulnerability in your product that's being actively exploited, you must report it. Not eventually. Not when you have a fix. Immediately.

2. Severe security incidents: Any incident that impacts the security of your product's users must be reported.

The timeline is tight:

24 hours: Early warning notification to ENISA (EU cybersecurity agency)
72 hours: Full incident notification with details
14 days: Final report after a fix is available (for vulnerabilities)
1 month: Final report for severe incidents

Miss these windows and you're in breach of EU law.

Why Most Companies Aren't Ready

The September deadline catches companies off guard because the full CRA requirements don't apply until December 2027. Most legal teams looked at that date and told engineering "we have time."

They missed the fine print. The reporting obligations apply to every product already on the market, not just products shipped after September 11. If you have software deployed in the EU today and you discover an exploited vulnerability next month, you're legally required to report it within 24 hours.

That means you need, right now:

A process to detect actively exploited vulnerabilities in your products
A triage workflow to determine if something meets the reporting threshold
A designated person who can submit to ENISA's Single Reporting Platform
Internal coordination between security, legal, and product teams
Documentation of every decision (regulators can ask for this)

Most startups and mid-size companies have exactly none of this in place.

What's Coming in December 2027

September 2026 is the appetizer. The full CRA requirements in December 2027 are the main course:

Annex I Part I (product requirements):

Products must be designed with security by default
No known exploitable vulnerabilities at time of release
Secure configuration out of the box
Protection against unauthorized access
Data confidentiality and integrity
Minimal attack surface
Incident logging and monitoring
Security updates for the entire support period

Annex I Part II (process requirements):

Vulnerability handling process (intake, triage, fix, disclosure)
SBOM (Software Bill of Materials) for every product
10 years of technical documentation retention
CE marking (yes, like on your toaster)

Conformity assessment:

Most products: self-assessment
"Important products" (password managers, VPNs, identity systems): third-party audit
"Critical products" (industrial controllers, certain infrastructure): mandatory EU certification

The Open Source Question

This is where it gets interesting. The CRA has a carve-out for open source, but it's narrower than people think.

Exempt: Open source developed and distributed without commercial intent. Pure hobby projects, community-maintained libraries where nobody's making money.

NOT exempt: Open source with commercial involvement. If your company maintains an open source library and offers paid support, consulting, or a commercial version, the CRA treats you as a manufacturer. Same obligations. Same reporting. Same conformity assessment.

This means companies like Elastic, MongoDB, HashiCorp, and anyone running an open-core model are fully in scope. And so is any company that ships commercial software containing open source dependencies, since they're the "manufacturer" of the combined product.

What Developers Should Actually Do

Before September 11, 2026 (next week)

Figure out if you sell to EU customers. If yes, CRA applies to you regardless of where you're based.
Set up a vulnerability intake process. You need a way for people to report vulnerabilities and a way to triage them.
Register with ENISA's reporting platform. You'll need access when (not if) you have to file.
Document your incident response process. Regulators will want to see it exists.

Before December 2027

Start your SBOM. Tools like Syft, CycloneDX, or SPDX can generate these from your dependency files.
Audit your security defaults. No default passwords, no unnecessary open ports, no optional security features that should be on by default.
Review your update mechanism. The CRA requires that security updates are delivered automatically or with clear user notification.
Assess your conformity route. Most products self-assess. Check Annex III and IV to see if you're in a higher category.

If you maintain open source

Check if commercial activity brings you in scope. Sponsorships, paid support, consulting, dual licensing all count.
Consider adding a security policy. SECURITY.md with a disclosure process covers the basics.
Know your downstream. If commercial products embed your library, their compliance may depend on your vulnerability handling.

The Bigger Picture

The CRA is the EU doing what GDPR did for privacy, but for product security. It's setting a floor that every digital product must meet. The 24-hour reporting window is modeled on GDPR's 72-hour breach notification, but tighter.

For developers, the practical impact is:

Security isn't optional anymore. It's a legal requirement with specific deliverables.
SBOMs are coming whether you like them or not. The CRA mandates them.
Vulnerability handling needs a process. "We'll fix it when we get to it" doesn't meet a 24-hour reporting window.
Documentation matters. 10-year retention means your threat models, security decisions, and risk assessments need to be written down.

The companies that started preparing in 2025 are in decent shape. The companies starting now have a week to get their reporting pipeline operational and 15 months to overhaul their development practices.

The companies that haven't started yet are about to have a very interesting autumn.

Sources:

The #1 Security Bug AI Puts in Your Python Code

turingrtss — Fri, 04 Sep 2026 14:52:17 +0000

AI writes f"SELECT * FROM users WHERE id={user_id}" because that's what gets upvoted on Stack Overflow. It works. It's readable. It's also a textbook SQL injection.

This isn't a hypothetical. I found this exact pattern in a UK Government repo with 2,693 stars. Production code. Merged, reviewed, deployed.

The Pattern

Here's what AI-generated SQL injection looks like in the wild:

# What AI writes
def get_user(user_id):
    query = f"SELECT * FROM users WHERE id={user_id}"
    cursor.execute(query)
    return cursor.fetchone()

# What it should write
def get_user(user_id):
    query = "SELECT * FROM users WHERE id=?"
    cursor.execute(query, (user_id,))
    return cursor.fetchone()

One character difference in the call. Completely different security posture.

The parameterized version treats user_id as data. The f-string version treats it as code. Pass 1; DROP TABLE users-- and the first version executes it.

Why AI Defaults to F-strings

Asked Claude, GPT-4, and Copilot to "write a function that queries users by ID." All three used f-strings on the first try.

Why? Because their training data is full of tutorials, blog posts, and Stack Overflow answers that use f-strings. Tutorial code optimizes for clarity, not security. The model learned what gets upvoted.

The fix exists in the training data too. Parameterized queries are well-documented. But the model has to actively choose the more verbose pattern over the simpler one. Without security context, it picks simple.

Real Examples

UK Government inspect_ai (2,693 stars)

An AI safety evaluation framework. The irony writes itself.

# src/inspect_ai/_display/textual/app.py:307
query = f"SELECT * FROM results WHERE {filter}"

filter comes from user input. No sanitization. No parameterization. Classic.

goldenmatch (131 stars)

A matching library:

# materialize.py:58
query = f"INSERT INTO matches VALUES ('{name}', '{score}')"
cursor.execute(query)

String concatenation inside SQL. Pass '; DROP TABLE matches; -- as a name and you own the database.

Same file, line 227:

query = f"SELECT * FROM {table} WHERE id = '{match_id}'"

Two injection points in the same file.

The Variants

F-string SQL injection shows up in several forms. All dangerous, all common in AI code:

Direct f-string

cursor.execute(f"SELECT * FROM users WHERE id={user_id}")

String concatenation

query = "SELECT * FROM users WHERE id=" + user_id
cursor.execute(query)

Format method

query = "SELECT * FROM users WHERE id={}".format(user_id)
cursor.execute(query)

Percent formatting

query = "SELECT * FROM users WHERE id=%s" % user_id
cursor.execute(query)

All four do the same thing: embed untrusted data directly into a SQL string. AI generates all four variants.

The Fix Takes 10 Seconds

Every Python database library supports parameterized queries:

# sqlite3
cursor.execute("SELECT * FROM users WHERE id=?", (user_id,))

# psycopg2 (PostgreSQL)
cursor.execute("SELECT * FROM users WHERE id=%s", (user_id,))

# mysql-connector
cursor.execute("SELECT * FROM users WHERE id=%s", (user_id,))

# SQLAlchemy
session.execute(text("SELECT * FROM users WHERE id=:id"), {"id": user_id})

If you're using an ORM (Django, SQLAlchemy), the ORM handles parameterization for you. The only time you hit this bug is when writing raw SQL, which is exactly when AI reaches for f-strings.

How to Catch It

Manual review

Look for any cursor.execute(), db.execute(), or session.execute() call where the argument contains f", +, .format(, or %. If the string has user-controlled variables, it's injectable.

Automated

pip install aiverify
aiverify your_project/ --rules sql_injection

AIVerify checks for f-strings and string concatenation in SQL calls, but filters out false positives like test files and static queries with no user input.

Pre-commit hook

Catch it before it reaches the repo:

# .pre-commit-config.yaml
- repo: https://github.com/turingrtss/aiverify
  hooks:
    - id: aiverify

The Bigger Problem

SQL injection is just one pattern. AI also generates:

Command injection: subprocess.run(f"git clone {url}", shell=True)
SSRF: requests.get(user_provided_url) with no validation
Hardcoded secrets: API keys that look like placeholders but aren't
Path traversal: open(f"/data/{user_filename}") with no sanitization

Each of these deserves its own post. The common thread: AI writes code that works on the happy path and breaks catastrophically on the adversarial path.

Takeaway

If you're reviewing AI-generated code, ctrl+F for f" near any database call. That single search will catch 80% of SQL injection bugs AI introduces.

If you want automated coverage, run a scanner. AIVerify, Bandit, Semgrep, whatever. The specific tool matters less than running something.

AI isn't going to stop writing f-string SQL. It's on us to catch it.

GitHub: https://github.com/turingrtss/aiverify

Previous post: How I Found 12 Critical Security Bugs in AI-Generated Code in 24 Hours

I Claimed 12 Critical Security Bugs. Most Were False Positives. Here's What I Learned.

turingrtss — Thu, 03 Sep 2026 14:25:47 +0000

Update (Sept 4): After manual verification of every finding, I'm correcting this article. The original version claimed 12 critical vulnerabilities. On deeper review, most were false positives from my regex-based scanner. The scanner flags patterns without understanding context — whether variables come from user input or internal config, whether sanitization exists nearby, whether the pattern is intentional by design. I'm leaving the technical analysis below but with honest verdicts. This is a lesson in verifying scanner output before publishing.

What Happened

I built AIVerify, a regex-based SAST scanner, and ran it against popular GitHub repos. The scanner flagged patterns like f-strings in SQL queries and subprocess calls with shell=True. I published the results without manually verifying each finding against the actual source code context.

That was a mistake.

What the Scanner Actually Found (Verified)

After cloning every repo and reading the code around each flagged line:

0 confirmed exploitable vulnerabilities. 1 questionable. 11 false positives.

Why They Were False Positives

Pattern: f-string SQL — The scanner flags f"SELECT * FROM {table}". But in most cases, table comes from internal config (dbt model names, application constants), not user input. One repo (inspect_ai) actually had explicit SQL injection protection that my scanner ignored.

Pattern: subprocess with shell=True — The scanner flags any subprocess call with string formatting. But many of these are admin scripts using environment variables, sandboxed execution environments with uid isolation, or CLI tools with no web-facing surface.

Pattern: requests.get(url) — Flagged as SSRF, but the URL came from AI API responses, not user input. Low practical risk.

What I Learned

A scanner finding is a lead, not a verdict. Every finding needs manual verification before disclosure.
Context matters more than pattern. The same code pattern is dangerous in a web handler and harmless in a config script.
Verify before you publish. I sent disclosure emails to 8 maintainers about non-issues. I published articles claiming findings that don't hold up.
Regex SAST has hard limits. Without data flow analysis, you can't distinguish user input from internal variables.

AIVerify v0.4.0

I've tightened the scanner rules based on this experience:

Command injection now requires shell=True with dynamic input (was flagging list-based subprocess)
XXE now requires untrusted input source (was flagging all XML parsing)
SSRF now requires user input indicators
Skips test/docs/examples directories by default

The false positive rate dropped significantly. But the fundamental limit remains: regex can't trace data flow.

GitHub: https://github.com/turingrtss/aiverify

The Real Takeaway

Don't trust scanner output blindly. Not from my tool, not from any tool. Every finding is a hypothesis that needs verification. I failed to do that verification before going public, and I'm correcting it now.

This correction was published because honesty matters more than marketing. The original claims were wrong and I owe that transparency to anyone who read them.