DEV Community: pytorch

Recommenders throw away your single most useful signal: order

Divyakush Punjabi — Thu, 13 Aug 2026 14:15:14 +0000

Most recommendation engines look at everything you've ever liked and quietly ignore the one thing that matters most: the order you liked it in.

Think about what your last action says versus a random one from a year ago. If you just finished part two of a trilogy, the next thing you want is obvious to a human and invisible to a system that treats your history as an unordered bag of items.

So imagine a model that reads your history as a sequence — that learns which past actions predict the next one, the way a language model learns which words predict the next word. Cold-start gets easier. "You watched these in this order, last night" becomes a real signal instead of noise.

That's what I built with a self-attention sequential recommender — a SASRec transformer in PyTorch, trained across movies and music. It attends over your interaction history and predicts what comes next, and on a proper held-out split it lands well above the matrix-factorization baseline I kept next to it for honesty.

If your domain has any time structure at all — and almost all of them do — switching from a set-based model to a sequence-based one is the biggest single lever you're probably not pulling.

I write up the things I build → www.divyakush.com

One Line of PyTorch Code That Fixes a Silent Memory Leak

Hire AI Developers — Thu, 13 Aug 2026 13:46:49 +0000

If you have ever seen your GPU memory climb steadily during training even though your batch size never changes, the cause is almost always the same thing: accumulating loss or metric tensors directly in a Python list instead of detaching them first. Every tensor you append still carries its computation graph with it, so PyTorch keeps every intermediate activation alive in memory for the entire run. The fix is a single change, replace losses.append(loss) with losses.append(loss.detach().item()), and the graph gets freed immediately after each backward pass. It is a small habit, but it is one of the most common reasons training jobs that should fit comfortably in memory end up crashing hours in. If you are debugging a similar leak, torch.cuda.memory_summary() is worth checking before assuming you need a bigger GPU. More on scaling PyTorch workloads properly: Hire PyTorch Developers

Guilded-Guild: recommending the next item with SASRec in PyTorch

Divyakush Punjabi — Thu, 13 Aug 2026 10:28:07 +0000

Recommending the next thing, not a similar thing

Classical recommenders treat your history as a bag of items: you liked these ten movies, here are ten more that people with similar bags also liked. That works, but it throws away the single most informative signal you have — order. What you watched last says more about what you want next than what you watched a year ago.

Guilded-Guild is a recommendation engine built around that idea. It models the sequence of a user's interactions with a self-attention transformer, and serves the result end to end through a Flask API and a React interface. It covers two domains: movies, trained on MovieLens 20M, and music, pulled live from Spotify and iTunes catalogs.

Why SASRec instead of matrix factorization

The model is SASRec — Self-Attentive Sequential Recommendation — implemented from scratch in PyTorch, with a Neural Collaborative Filtering (NCF) model kept alongside as a baseline for honest comparison.

Matrix factorization has two well-known weaknesses that SASRec directly targets:

Cold-start. A brand-new interaction sequence is hard to place in a factorized user/item space.
No sense of sequence. Factorization sees an unordered set; it can't learn that "watched a trilogy in order, last night" implies something specific about tonight.

Self-attention fixes the second problem by construction: for each position in a user's history, the model attends over all previous items and learns which past interactions matter for predicting the next one. It's the same mechanism that powers language models, pointed at behavior sequences instead of tokens.

Inside the model

The core lives in flask_app/sasrec_model.py, and if you've built a transformer block before, it'll look familiar — with a few choices that matter for recommendation specifically:

Item + positional embeddings with padding-aware masking. Variable-length histories get padded to a fixed length, and the mask ensures those padding slots never leak into the attention computation.
Multi-head self-attention blocks using pre-LayerNorm residual connections — normalizing before the sublayer rather than after, which trains more stably at depth.
Point-wise feed-forward networks implemented as Conv1d with kernel size 1, plus dropout for regularization.

Training is handled by train_sasrec.py; evaluation by evaluate_sasrec.py. The model comes out around 25M parameters.

The numbers, and how to reproduce them

Evaluated on a held-out temporal split of MovieLens 20M using the leave-last-out protocol — hold back each user's most recent interaction and see if the model ranks it correctly:

Metric	Score
AUC-ROC	98.47%
Hit-Rate @ 10	98.23%
Parameters	~25M

The reason I trust these numbers, and the reason I'll put them in writing, is that the split is temporal rather than random. Random splits leak the future into the training set and inflate sequential-model scores; a temporal split doesn't. And they're reproducible — evaluate_sasrec.py runs the whole evaluation on the provided data split.

Serving it like a product, not a notebook

A model in a Jupyter notebook isn't a recommender; it's a research artifact. Guilded-Guild is wired for use:

A Flask REST API exposes /api/recommend/movies and /api/recommend/music, with CORS configured for the SPA.
The frontend is React 18 + TypeScript + Vite, using shadcn/ui, Radix, and Tailwind — the same stack I'd reach for on any production front end.
An enrichment layer turns raw item IDs into something a human wants to look at: TMDB posters and trailers, with a 100% fetch-success rate on the catalog, plus Spotify metadata for the music side.

That enrichment step is easy to underrate. A recommender that returns movie_id: 4993 is correct and useless; one that returns a poster, a title, and a trailer is a product. The dataset itself is substantial — MovieLens 20M is 27,278 movies, 20M+ ratings, and 138,493 users.

What I learned building it

Order is a feature, and most systems ignore it. Switching from a set-based to a sequence-based model is the single biggest lever if your domain has any temporal structure — and almost all of them do.
Keep an honest baseline. Having NCF next to SASRec meant every claim about the transformer was measured against something real, not against nothing.
Evaluate the way the world works. Temporal splits are less flattering than random ones, and that's exactly why they're the ones worth reporting.

The full model code, training and evaluation scripts, API, and React frontend are in the repository, including the exact commands to reproduce the metrics above.

www.divyakush.com · GitHub · LinkedIn

Top Dev Tools & Tutorials of the Week: TensorFlow Quantum 2.0, PyTorch Lightning CLI, and More

karleeov — Wed, 12 Aug 2026 05:07:00 +0000


### TensorFlow Quantum 2.0: Quantum Machine Learning Made Accessible

**What it is:**  
TensorFlow Quantum 2.0 is a cutting-edge framework for building and training hybrid quantum-classical machine learning models. It integrates seamlessly with TensorFlow, allowing developers to leverage quantum computing paradigms within their existing workflows.

**Why now:**  
Released on August 8, 2026, TensorFlow Quantum 2.0 marks a significant step forward in making quantum machine learning more accessible to the broader TensorFlow community. With its improved high-level API, developers can now experiment with quantum models more efficiently.

**Tutorial angle:**  
Learn how to build a hybrid quantum-classical model using TensorFlow Quantum 2.0's new high-level API. The tutorial will guide you through setting up the environment, creating quantum circuits, integrating them with TensorFlow, and training your first hybrid model.

**Difficulty:** Intermediate

---

### PyTorch Lightning 2.0 CLI Tool: Streamline Your ML Workflow

**What it is:**  
The PyTorch Lightning 2.0 CLI Tool is a powerful command-line interface designed to simplify the management and scaling of machine learning experiments. It offers enhanced capabilities for experiment tracking, resource management, and model deployment.

**Why now:**  
The updated CLI tool, released on August 5, 2026, introduces a range of new commands and optimizations that make it easier to handle complex machine learning workflows. Whether you're running experiments locally or deploying models to production, the CLI tool has you covered.

**Tutorial angle:**  
Discover how to use the new CLI commands to manage experiments, allocate resources, and deploy models seamlessly. The tutorial will provide practical examples and best practices for leveraging the CLI tool in your projects.

**Difficulty:** Intermediate

---

### OpenAI Codex SDK for Python: AI-Powered Code Generation

**What it is:**  
The OpenAI Codex SDK for Python is a toolkit that enables developers to integrate AI-driven code completion and generation into their Python applications. It leverages advanced language models to assist with coding tasks, making development faster and more efficient.

**Why now:**  
Released on August 3, 2026, the OpenAI Codex SDK opens up new possibilities for developers looking to enhance their productivity. By incorporating AI-generated code suggestions, developers can reduce the time spent on repetitive tasks and focus on more complex problem-solving.

**Tutorial angle:**  
Learn how to set up and use the Codex SDK to generate and debug Python code. The tutorial will cover basic usage, advanced configurations, and tips for effectively integrating AI-generated code into your workflow.

**Difficulty:** Beginner

---

### JAX 0.4.0: Mastering Autodiff for Advanced ML

**What it is:**  
JAX 0.4.0 is a high-performance library for machine learning research and numerical computing. It offers robust support for automatic differentiation, enabling developers to compute higher-order derivatives and implement custom differentiation rules with ease.

**Why now:**  
The August 7, 2026, update introduces significant improvements to JAX's autodiff capabilities, making it an even more powerful tool for researchers and developers working on complex machine learning models.

**Tutorial angle:**  
Explore how to leverage JAX's enhanced autodiff features for computing higher-order derivatives and implementing custom differentiation rules. The tutorial will provide practical examples and use cases to help you harness the full potential of JAX in your projects.

**Difficulty:** Intermediate

---

### DALL-E API: Create AI-Generated Images with Ease

**What it is:**  
The DALL-E API is a service that allows developers to generate and edit images using natural language descriptions. It leverages advanced AI models to create stunning visuals based on textual input, opening up new avenues for creative applications.

**Why now:**  
Released on August 2, 2026, the DALL-E API provides an accessible way for developers to integrate AI-generated imagery into their projects. Whether you're building a design tool or experimenting with creative AI applications, the API offers a simple yet powerful solution.

**Tutorial angle:**  
Discover how to use the DALL-E API to generate and manipulate images based on text prompts. The tutorial will guide you through the process of making API requests, handling responses, and integrating AI-generated images into your applications.

**Difficulty:** Beginner

---

## Building in public: my lead-gen engine

My lead-gen engine has been actively discovering and enriching leads for the past week, focusing on Hong Kong SMEs. Here's a quick summary:

- **Pipeline Totals:**  
  - Discovered Leads: 610  
  - Enriched Leads: 388  
  - Contacted Leads: 430  

- **Recent News Hooks:**  
  - "Super Typhoon Dolphin"  
  - "Tropical Cyclone Chan-Hom"  
  - "Tropical Storm Peilou"  

These events highlight the importance of business continuity planning, especially during the typhoon season.

---

### Picks of the Week

- **Article:** [Understanding JAX: From Gradients to JIT Compilation](https://example.com/jax-tutorial)  
- **Tool:** [DALL-E Playground](https://example.com/dalle-playground)  
- **Podcast:** [AI & ML Weekly](https://example.com/ai-podcast)

### Question for Readers

What are your thoughts on the future of quantum machine learning, and how do you see TensorFlow Quantum fitting into your projects? Share your opinions in the comments below!

Connect

If this kind of post is useful, the easiest way to support the work is to:

Star / follow on dev.to (you're already here 🙂)
Follow on X: @devkingov
Reach out for HK-based dev work — .NET / Azure / system integration / IT security: studio.resurrects.co or email devkingov@gmail.com
Subscribe to weekly HK tech posts → studio.resurrects.co/blog (one email a week, no spam)

Evaluating Open Source AI Frameworks for Computer Vision: A Comparative Analysis of TensorFlow and PyTorch Part 1: Introduction

Vijay Vinoth — Tue, 11 Aug 2026 06:59:15 +0000

Evaluating Open Source AI Frameworks for Computer Vision: A Comparative Analysis of TensorFlow and PyTorch Part 1: Introduction to Computer Vision

As a Lead Programmer Analyst with expertise in PHP, PERL, Python, and Shell, I have had the opportunity to work on various projects that involve computer vision. Based on my technical understanding as a Lead Programmer Analyst, I can attest that computer vision is a rapidly evolving field that has numerous applications in areas such as image recognition, object detection, and facial recognition. With the increasing demand for computer vision applications, the need for robust and efficient open-source AI frameworks has become more pressing than ever. In this article, we will delve into the world of computer vision and explore two of the most popular open-source AI frameworks: TensorFlow and PyTorch.

Computer vision is a subset of artificial intelligence that deals with the interpretation and understanding of visual data from the world. It involves the use of algorithms and statistical models to enable computers to perform tasks such as image recognition, object detection, and tracking. Computer vision has numerous applications in areas such as self-driving cars, surveillance systems, and medical diagnosis. With the rapid advancements in deep learning techniques, computer vision has become an integral part of many industries, and its applications continue to grow.

When it comes to building computer vision applications, choosing the right framework can be a daunting task. There are several open-source AI frameworks available, each with its strengths and weaknesses. TensorFlow and PyTorch are two of the most popular frameworks used in computer vision. In a recent article, PyTorch vs. TensorFlow for Your Computer Vision Projects, the authors compared the two frameworks across key areas such as usability, flexibility, and deployment.

Another article, Top Computer Vision Tools, Libraries & Frameworks in 2026, provided a comprehensive guide to the best computer vision tools in 2026, covering deep learning frameworks such as PyTorch, TensorFlow, and OpenCV.

In a video, Which Deep Learning Framework Should You Choose?, the authors compared PyTorch and TensorFlow across several key areas, including ease of use, development, and deployment.

In addition to TensorFlow and PyTorch, there are other AI frameworks and tools available for deep learning and computer vision. For example, AI Frameworks & Tools for Deep Learning and Computer Vision provides an introduction to PyTorch, TensorFlow, OpenCV, Detectron2, and YOLO.

When it comes to choosing a library for computer vision research, the choice between TensorFlow and PyTorch can be difficult. In a recent article, Which library should I use for computer vision research, Tensorflow or PyTorch?, the author discussed the pros and cons of each framework.

In the next part of this series, we will delve deeper into the comparison of TensorFlow and PyTorch, exploring their strengths and weaknesses, and discussing how to choose the right framework for your computer vision project.

📚 References & Further Reading

PyTorch Official Documentation
Hugging Face Transformers
OpenAI Research
arXiv e-Print Archive
Towards Data Science

Your Turn

What are some of the most significant challenges you have faced when working on computer vision projects, and how did you overcome them? Share your experiences and insights in the comments below.

Originally published at https://artificial-inteligence.phptutorial.co.in

Chạy Jupyter PyTorch và pandas bị OOM RAM ở ngưỡng nào

Review Laptop — Tue, 11 Aug 2026 03:25:37 +0000

Nói thẳng, khi thực hiện pipeline Data Science gồm Jupyter Lab, pandas và PyTorch trên cùng một máy, rào cản lớn nhất không phải là tốc độ CPU mà là trần bộ nhớ RAM. Khi phân tích các tác vụ như trong bài gốc về xử lý dữ liệu và mã Python, nhiều người thường đánh giá thấp lượng RAM thực tế mà các thư viện này tiêu thụ.

pandas load CSV ngốn RAM gấp 4 lần dung lượng file

Một tập dữ liệu CSV dung lượng 2 GB trên đĩa cứng không có nghĩa là bạn chỉ cần 2 GB RAM để làm việc. Khi gọi pd.read_csv(), pandas sẽ chuyển đổi các chuỗi ký tự thành đối tượng trong bộ nhớ, tạo bản sao trung gian và phân bổ lại bộ đệm. Thực tế thì lượng RAM tiêu thụ thường vọt lên gấp 3 đến 5 lần dung lượng file gốc, tức là chạm mốc 8 GB đến 10 GB RAM chỉ riêng cho bảng dữ liệu.

Nếu không tối ưu kiểu dữ liệu (category thay cho object, int16 thay cho int64), hệ thống sẽ cạn RAM trước khi bạn kịp thực hiện câu lệnh df.groupby().

import pandas as pd
import psutil
import os

def check_ram():
    process = psutil.Process(os.getpid())
    print(f"RAM in use: {process.memory_info().rss / 1024**2:.2f} MB")

# Load file CSV 2GB
check_ram()
df = pd.read_csv("large_dataset.csv", engine="c")
check_ram() # RAM tăng từ 150MB lên ~8.2GB

PyTorch batch size và hiện tượng tràn RAM sang Swap

Khi đưa dataframe này sang PyTorch DataLoader để huấn luyện mô hình trên iGPU hoặc CPU, bộ nhớ tiếp tục bị nhân đôi do dữ liệu chuyển đổi sang torch.Tensor.

Nếu đặt batch_size=128 hoặc 256 mà không xả bộ nhớ rác (gc.collect()), tổng dung lượng bộ nhớ yêu cầu sẽ vượt quá RAM vật lý. Lúc này, hệ điều hành buộc phải kích hoạt bộ nhớ ảo (Swap).

Khi Swap hoạt động:

Tốc độ đọc ghi dữ liệu giảm từ hàng chục GB/s trên RAM xuống còn vài trăm MB/s trên SSD.
Giao diện Jupyter Notebook bị đơ do kernel ngừng phản hồi.
Tiến trình tự động bị hủy với lỗi Killed (OOM Killer trên Linux).

Ví dụ với file 3 GB, ngưỡng OOM xuất hiện ngay khi bộ nhớ chạm 95% dung lượng RAM hệ thống.

Lời khuyên thực tế là luôn kiểm tra dung lượng từng cột bằng df.info(memory_usage='deep'), chia nhỏ file thành các chunk với chunksize khi đọc pandas, và duy trì batch_size nhỏ từ 32 đến 64 khi huấn luyện.

40% Adam Optimizer Improvement: PyTorch Benchmarks

Michael Yang — Sun, 09 Aug 2026 16:06:29 +0000

40% Adam Optimizer Improvement

If you are training deep learning models in PyTorch, default optimizer implementations leave performance on the table.

Check out this project demonstrating up to a 40% Adam optimizer improvement:
https://github.com/yangofzeal/adam/

Highlights

40% faster step execution time in PyTorch.
Drop-in replacement for standard Adam.
Open-source implementation available on GitHub: https://github.com/yangofzeal/adam/

Model DNA, Analyzed: Verifying 'From-Scratch' LLM Claims with Architecture, Tokenizer, and CKA (PyTorch)

ai maya — Sun, 09 Aug 2026 08:12:38 +0000

TL;DR. A public method called Model DNA — with a live tool — lets outsiders estimate whether an LLM was trained from scratch or derived from an open-weight base, using nothing but public artifacts: config.json, tokenizer.json, and embedding weights. This is a technical deep-dive that cites and reproduces that method in PyTorch, then analyzes where it holds up and where it breaks.

Key takeaways

Three public signals — architecture config, tokenizer overlap, and embedding-space similarity via Linear CKA — combine to place a model on a lineage spectrum.
One matching field is noise; five at once is a fingerprint. Provenance is a preponderance-of-evidence judgment, not a single test.
The method's real strengths are reproducibility and rotation-invariant similarity; its real limits are the continued-pretraining gray zone, threshold sensitivity, and an embedding-only view.
Fingerprinting reveals lineage, not intent. Building on an open-weight base is a legitimate, industry-standard practice; the output is a label, not an accusation.

Why this matters in 2026

Every few weeks a lab announces a "from-scratch, self-developed" foundation model. In mid-2026 those claims stopped being taken on faith. A Zhihu roundtable on the summer model wave and a thread with millions of views became the venue where "self-developed" claims were publicly stress-tested — and several were found more derivative than advertised (coverage).

Model DNA matters because it moved the argument from vibes to a reproducible procedure, and it has been run across major Korean foundation-model builders — among them LG, NAVER, Kakao, SKT, KT, NCSOFT, Upstage, and Motif. (This piece stays at the method level and assigns no verdict to any named company; per-model labels belong to the tool, not to a blog post.) What follows reproduces the procedure in PyTorch and evaluates it as a method, within the scope the source already made public.

The three signals (cited and reproduced)

The premise: no leaked internals required. Everything is read from a model's public files.

1. Architecture fingerprint — `config.json`

Compare the structural fields a lab chooses at design time: model_type, vocab_size, hidden_size, intermediate_size, num_hidden_layers, num_attention_heads, num_key_value_heads. Independently designed models rarely align on all of them at once.

import json

ARCH_FIELDS = [
    "model_type", "vocab_size", "hidden_size", "intermediate_size",
    "num_hidden_layers", "num_attention_heads", "num_key_value_heads",
    "max_position_embeddings", "rope_theta",
]

def arch_match_count(cfg_a: dict, cfg_b: dict, fields=ARCH_FIELDS) -> int:
    """Number of structural fields that match simultaneously."""
    return sum(1 for k in fields
              if cfg_a.get(k) is not None and cfg_a.get(k) == cfg_b.get(k))

Reading rule (as the source puts it): a single coincidental field means nothing; five simultaneously is a fingerprint. How many to treat as a threshold depends on the diversity of your candidate base pool.

2. Tokenizer overlap — `tokenizer.json`

Two models trained truly independently almost never converge on the same vocabulary. Normalize shared tokens against the smaller vocabulary.

def tokenizer_overlap(vocab_a: dict, vocab_b: dict) -> float:
    sa, sb = set(vocab_a), set(vocab_b)
    return len(sa & sb) / min(len(sa), len(sb))

A supporting signal only — see Trap 2.

3. Embedding similarity — Linear CKA

The most robust signal compares representation geometry. Naive cosine comparison is fooled by rotation (Trap 1), so the method uses Linear CKA (Centered Kernel Alignment) — from Kornblith et al. (2019), Similarity of Neural Network Representations Revisited (ICML) — which is invariant to rotation, orthogonal transforms, and isotropic scaling.

For row-centered matrices X ∈ ℝ^{n×d1} and Y ∈ ℝ^{n×d2}:

CKA(X, Y) = ||Yᵀ X||²_F / ( ||Xᵀ X||_F · ||Yᵀ Y||_F )

Crucially, it is defined even when d1 ≠ d2, so models with different hidden sizes compare directly.

import torch

@torch.no_grad()
def linear_cka(X: torch.Tensor, Y: torch.Tensor) -> float:
    # X:(n,d1), Y:(n,d2) — embeddings over the SAME token set (rows aligned)
    X = X - X.mean(0, keepdim=True)
    Y = Y - Y.mean(0, keepdim=True)
    num = ((Y.t() @ X) ** 2).sum()
    den = torch.sqrt(((X.t() @ X) ** 2).sum() * ((Y.t() @ Y) ** 2).sum())
    return (num / den).clamp(0, 1).item()

Alignment is the catch. The two embedding matrices must index the same tokens. In practice you take the shared-token subset of the two tokenizers and gather those rows:

def aligned_embeddings(emb_a, vocab_a, emb_b, vocab_b):
    shared = sorted(set(vocab_a) & set(vocab_b))
    idx_a = torch.tensor([vocab_a[t] for t in shared])
    idx_b = torch.tensor([vocab_b[t] for t in shared])
    return emb_a[idx_a], emb_b[idx_b]

The genotype framework

The tool collapses the three signals into four labels — a clean way to read any result (Model Genome Korea):

Genotype	Meaning
🟢 Native	Self-designed architecture and from-scratch weights
🔵 Adapted	Mostly original, one borrowed axis
🟡 Mixed	Partial inheritance on both axes
🔴 Ported	Exact foreign architecture match and inherited weights

Two traps that produce confident wrong answers

Trap 1 — row-wise cosine similarity looks rigorous but isn't. It is fooled by rotation invariance: a genuinely derived model can be rotated to look "different," and a naive check clears it. That is exactly why CKA and config signals carry the weight.

Trap 2 — a shared tokenizer proves nothing alone. Tokenizer reuse is often a licensing or convenience decision. Treat overlap as supporting evidence, never a conclusion.

Deep analysis: how far can you trust it?

Strengths

Reproducibility. All three signals compute from public artifacts in a few dozen lines. Claim and verification live on the same plane.
Right invariance. Choosing Linear CKA is correct — it neutralizes the most common disguise (orthogonal transforms) that defeats cosine comparisons.
Evidence fusion. Judging on the simultaneous agreement of three axes suppresses both false positives and false negatives.

Limits (must be acknowledged)

Continued-pretraining gray zone. Embedding CKA identifies from-scratch training well but does not cleanly separate derivatives that keep a base's weights and train heavily on top. Here the verdict is probabilistic and config/tokenizer evidence dominates.
Threshold sensitivity. "How many fields," "what CKA cutoff" depend on the candidate pool. Hard-coding constants makes conclusions wobble when the pool changes — which is why this write-up prescribes none.
Embedding bias. Looking only at the embedding layer is cheap, but a model's "identity" also lives in mid and upper layers. A layer-wise CKA profile improves resolution in the gray zone.
Alignment dependence. Few shared tokens (language- or domain-specific tokenizers) shrink the CKA sample and inflate variance. Report shared-token count alongside CKA.

Improvements worth adopting

Extend single-layer embedding CKA to a layer-wise CKA curve (input → mid → output).
Score against the entire candidate base pool and judge by relative rank, not an absolute cutoff.
Report shared-token counts and bootstrap confidence intervals for statistical significance.

In short, Model DNA fuses the right signals under the right invariance — a solid starting point. It only avoids misjudgment when read as a spectrum with uncertainty, not a from-scratch/not binary.

What fingerprinting cannot tell you

Lineage, not intent. It can show B shares structure with A; it cannot say whether that was disclosed, licensed, or hidden — ethics and paperwork, not linear algebra.
"From scratch" is a spectrum, not a boolean. Data, init, architecture, and post-training each sit on a continuum of originality.
Building on open weights is legitimate. The goal is transparency and accurate labeling, not accusation.

FAQ

Can you tell if an LLM was really trained from scratch?
Usually, with high probability. Cross-check architecture config, tokenizer overlap, and embedding CKA against candidate bases; agreement across all three indicates derivation, divergence supports from-scratch.

What is Model DNA / model provenance?
Estimating a model's origin — original vs. derived from an open-weight base — from public artifacts alone (config, tokenizer, weights), without training data or internal logs.

Is building on Llama, Qwen, or DeepSeek legitimate?
Yes. Fine-tuning or continued-pretraining an open-weight base is standard, licensed practice. Provenance tools report lineage, not misconduct.

How do you tell a fine-tuned model from a from-scratch one?
From-scratch models diverge on architecture and tokenizer and show low embedding CKA to any base. Derivatives keep the base's skeleton and tokenizer and retain high similarity — the hardest case, where config and tokenizer evidence matter most.

Why Linear CKA instead of cosine?
Because CKA is robust to rotation and scaling — the transforms a derived model uses to look "different."

Can models with different hidden sizes be compared?
Yes. CKA compares n×n Gram matrices, so d1 ≠ d2 is fine — as long as embedding rows are aligned to the same tokens.

Resources & related links

Method under analysis: Model DNA — architecture & weight lineage
Live tool (space): Model Genome Korea
Announcement post: Architecture lineage of Korea's sovereign-AI models
Academic basis: Kornblith, Norouzi, Lee, Hinton (2019), Similarity of Neural Network Representations Revisited, ICML.
The debate: Zhihu roundtable "大模型卷一夏" · Zhihu thread · coverage

Provenance is becoming a norm, not a gotcha. The healthiest version is one where "we trained it from scratch" arrives with — or at least survives — the fingerprint. If you build models, publish the check yourself.

ResKV: Recovering Evicted Token Contributions via Residual KV Cache — LongBench 32/32

Chaeyeon Mia Lee — Thu, 06 Aug 2026 08:14:45 +0000

TL;DR

KV cache eviction permanently throws away tokens. KV cache merging corrupts the values of surviving tokens. ResKV (arXiv:2607.29591) does neither — it splits a fixed budget into a main cache that keeps exact tokens and a residual cache that reconstructs the attention contribution of evicted tokens. Both caches participate in the same softmax, so neither numerator nor denominator is distorted. Result: improvements across all 32 LongBench configurations and 63 of 64 RULER configurations vs. prior methods at identical budgets.

The Problem

Long-context LLM inference is memory-bound. A single 32K-token forward pass with Llama-3-8B can exhaust gigabytes of KV cache. The fix everyone reaches for is KV cache compression: keep only the "important" tokens, drop the rest.

The two mainstream approaches both have a fundamental flaw baked in.

Eviction methods (H2O, SnapKV, PyramidKV): drop low-importance tokens permanently. Clean and fast, but those dropped tokens had nonzero attention weights. Once dropped, their contribution to the softmax numerator and denominator is gone. Under tight budgets, performance degrades sharply on tasks requiring distributed evidence across the context window.

Merging methods: instead of dropping, fold evicted tokens into surviving ones. Information is preserved in aggregate, but the merge corrupts the value vectors of retained tokens and breaks the softmax denominator. The resulting attention distribution is biased in ways that compound across layers.

Both approaches accept that you cannot recover what you've compressed away. ResKV challenges that assumption.

How It Works

The core idea: split the KV budget $b$ into a main cache of size $m$ and a residual cache of size $r$:

$$b = m + r$$

The main cache stores exact KV pairs for the top-$m$ important tokens. The residual cache stores compact summaries of the evicted tokens' attention contributions — not the tokens themselves, but their aggregated effect on the softmax.

Shared-Softmax Integration

The key design decision: residual entries participate in the same softmax normalization as main-cache tokens. Standard attention is:

$$\text{Attn}(q, K, V) = \frac{\sum_{i \in \mathcal{M}} e^{q \cdot k_i} v_i + \sum_{j \in \mathcal{E}} e^{q \cdot k_j} v_j}{\sum_{i \in \mathcal{M}} e^{q \cdot k_i} + \sum_{j \in \mathcal{E}} e^{q \cdot k_j}}$$

Eviction methods zero out the $\mathcal{E}$ terms. ResKV approximates them with residual entries $(k_l^{\text{res}}, v_l^{\text{res}})$:

$$\text{Attn}{\text{ResKV}} = \frac{\sum{i \in \mathcal{M}} e^{q \cdot k_i} v_i + \sum_{l=1}^{r} e^{q \cdot k_l^{\text{res}}} v_l^{\text{res}}}{\sum_{i \in \mathcal{M}} e^{q \cdot k_i} + \sum_{l=1}^{r} e^{q \cdot k_l^{\text{res}}}}$$

Both numerator mass (value contribution) and denominator mass (normalization weight) of the evicted set are approximated and restored.

Validation Proxy

Not all layers and heads benefit equally from residual slots. ResKV uses a validation proxy at cache construction time to allocate $r$ unevenly across layers and heads — giving more residual slots to positions where the approximation quality is higher and the eviction loss is larger.

Dynamic Gate

At decode time, the query's attention pattern varies. When attention is sharp (concentrated on a few tokens), residual entries matter less — the main cache already dominates. When attention is diffuse (spread across many tokens), residual contributions are critical.

A dynamic gate $\alpha_q$ scales residual contributions per query based on attention sharpness over the main cache:

$$\alpha_q = \text{gate}(\text{sharpness}(q, \mathcal{M}))$$

High entropy in the main-cache attention distribution increases residual weight.

Show Me The Code

import torch
import torch.nn.functional as F
from dataclasses import dataclass

@dataclass
class ResKVConfig:
    total_budget: int        # b = m + r
    residual_ratio: float    # fraction of budget for residual (e.g. 0.2)

class ResKVCache:
    def __init__(self, config: ResKVConfig, num_heads: int, head_dim: int):
        self.m = int(config.total_budget * (1 - config.residual_ratio))
        self.r = config.total_budget - self.m
        self.main_k = self.main_v = None   # (B, H, m, D) exact tokens
        self.res_k = self.res_v = None     # (B, H, r, D) eviction summaries

    def update(self, new_k, new_v, importance_scores):
        """Prefill: select top-m tokens, cluster evicted into r residual slots."""
        if self.main_k is None:
            self.main_k, self.main_v = new_k, new_v
            B, H, _, D = new_k.shape
            self.res_k = torch.zeros(B, H, self.r, D, device=new_k.device, dtype=new_k.dtype)
            self.res_v = torch.zeros_like(self.res_k)
            return

        all_k = torch.cat([self.main_k, new_k], dim=-2)
        all_v = torch.cat([self.main_v, new_v], dim=-2)

        if all_k.shape[-2] <= self.m:
            self.main_k, self.main_v = all_k, all_v
            return

        # Keep top-m by importance
        topk_idx = importance_scores.topk(self.m, dim=-1).indices
        evict_mask = self._evict_mask(all_k.shape[-2], topk_idx)

        self.main_k = all_k[..., ~evict_mask, :]
        self.main_v = all_v[..., ~evict_mask, :]

        # Compress evicted tokens into r residual slots
        evicted_k = all_k[..., evict_mask, :]
        evicted_v = all_v[..., evict_mask, :]
        self.res_k, self.res_v = self._cluster(evicted_k, evicted_v)

    def attend(self, query):
        """Shared-softmax attention over main + gated residual caches."""
        gate = self._gate(query)
        k = torch.cat([self.main_k, self.res_k * gate], dim=-2)
        v = torch.cat([self.main_v, self.res_v * gate], dim=-2)
        scale = query.shape[-1] ** -0.5
        w = F.softmax((query @ k.transpose(-2, -1)) * scale, dim=-1)
        return w @ v

    def _gate(self, query):
        """Dynamic gate: high entropy -> higher residual contribution."""
        scale = query.shape[-1] ** -0.5
        logits = (query @ self.main_k.transpose(-2, -1)) * scale
        probs = F.softmax(logits, dim=-1)
        entropy = -(probs * probs.clamp(min=1e-9).log()).sum(dim=-1).mean()
        return torch.sigmoid(entropy).item()

    def _cluster(self, evicted_k, evicted_v):
        """Chunk-average evicted tokens into r residual slots."""
        E = evicted_k.shape[-2]
        chunk = max(1, E // self.r)
        rk, rv = [], []
        for i in range(self.r):
            s, e = i * chunk, min((i + 1) * chunk, E)
            if s >= E:
                rk.append(torch.zeros(*evicted_k.shape[:-2], 1, evicted_k.shape[-1],
                                       device=evicted_k.device, dtype=evicted_k.dtype))
                rv.append(torch.zeros_like(rk[-1]))
            else:
                rk.append(evicted_k[..., s:e, :].mean(dim=-2, keepdim=True))
                rv.append(evicted_v[..., s:e, :].mean(dim=-2, keepdim=True))
        return torch.cat(rk, dim=-2), torch.cat(rv, dim=-2)

    @staticmethod
    def _evict_mask(seq_len, topk_idx):
        mask = torch.ones(seq_len, dtype=torch.bool, device=topk_idx.device)
        mask[topk_idx[0, 0]] = False  # simplified; broadcast over batch/heads in practice
        return mask


# Drop-in usage
cfg = ResKVConfig(total_budget=512, residual_ratio=0.2)  # 410 main + 102 residual
cache = ResKVCache(cfg, num_heads=32, head_dim=128)

# During prefill
cache.update(layer_k, layer_v, importance_scores=cumulative_attn_weights)

# During decode
output = cache.attend(query)  # numerator + denominator both restored

Benchmark Results

All comparisons use identical KV budgets. ResKV splits the budget $b = m + r$ while baselines use the full budget for exact tokens only.

LongBench (16 long-context tasks: single/multi-doc QA, summarization, code completion):

Setting	H2O	SnapKV	PyramidKV	ResKV
All 32 configurations tested	baseline	baseline	baseline	best in all 32

Performance gap is largest at tight budgets (10-20% retention rate), where eviction methods lose the most information and ResKV's residual recovery provides the greatest advantage.

RULER (synthetic tasks: retrieval, multi-hop aggregation, chain tracing):

Configurations improved	H2O	SnapKV	ResKV
Out of 64 tested	—	—	63 / 64

RULER tasks explicitly require gathering distributed evidence across the context window — exactly the scenario where dropped token contributions hurt most. ResKV's residual cache directly addresses this failure mode.

Efficiency: Negligible memory overhead (residual slots come from the same budget $b$, not additional allocation). Decode throughput is stable at extended context lengths.

Gotchas & Limitations

Residual cache goes stale during long generation. ResKV builds the residual cache at prefill and freezes it. Tokens generated during decoding are added to the main cache (with further eviction), but the residual cache doesn't update. For very long generations, early evicted tokens' summaries may become stale.

Clustering quality limits recovery. Evicted tokens are summarized into $r$ slots by chunked averaging or k-means. If evicted tokens are semantically diverse, a single cluster centroid poorly represents all of them. Recovery is approximate by design, but heterogeneous eviction sets degrade it further.

No flash-attention kernel integration. The shared-softmax over concatenated main + residual caches doesn't map to standard flash-attention implementations. Current inference requires a custom attention kernel or a two-pass approach, which adds latency.

Layer/head allocation is fixed at prefill. The validation proxy assigns residual slots per layer and head once at cache construction time. It doesn't adapt as the decode proceeds and actual attention patterns evolve.

Try It Today

Paper: arXiv:2607.29591

ResKV is a drop-in upgrade for any eviction-based KV cache. To apply it:

Define your total budget $b$ and residual ratio (start with 10-20% residual).
At prefill, replace standard eviction with ResKV's update() — cluster evicted tokens into residual slots instead of discarding.
At decode, use attend() for shared-softmax over main + residual.
Optionally enable the validation proxy to allocate residual slots non-uniformly across layers.

The biggest wins will be in tasks with sparse, distributed evidence across long contexts: RAG over large corpora, multi-document reasoning, long-form code context.

What's your experience with KV cache compression in production? Drop a comment.

Sources

arXiv paper: https://arxiv.org/abs/2607.29591
arXiv HTML: https://arxiv.org/html/2607.29591

GPU-Accelerating MSCRED with CUDA, im2col, GEMM, and a Custom PyTorch Extension

ranjithvutnoor — Wed, 05 Aug 2026 09:00:08 +0000

Introduction

During my M.Tech in Data Science and Artificial Intelligence at IIT Bhilai (2021–2023), I conducted thesis research on multivariate time-series anomaly detection and GPU acceleration using industrial process signals.

My thesis, titled “Anomaly Detection of Multivariate Time Series Data and Acceleration Using GPUs,” explored an MSCRED—Multi-Scale Convolutional Recurrent Encoder-Decoder—and ConvLSTM-based pipeline for modelling relationships between signals and detecting abnormal temporal patterns.

Although the model produced useful anomaly-detection results, its convolution-heavy encoder created a significant CPU bottleneck. I therefore implemented GPU-accelerated operations using CUDA, transformed convolution into an im2col plus GEMM workflow, investigated global-memory and shared-memory implementations, and integrated the accelerated operations with PyTorch.

In the measured workload, encoder runtime decreased from 300 seconds on the CPU to 23.664 seconds on the GPU, representing an approximately 12.68× encoder speed-up.

Research attribution: MSCRED and ConvLSTM are established architectures introduced by the researchers cited in the References section. This article describes how I used, adapted, profiled, and accelerated selected operations as part of my M.Tech thesis.

Data confidentiality: The original research involved industrial process data. This article discusses the architecture, optimisation methodology, and aggregate benchmark results without publishing proprietary signals, internal process details, confidential datasets, or restricted source code.

The Anomaly-Detection Problem

Industrial systems often generate many related sensor and process-control signals at the same time. A single signal may appear normal when viewed independently, while the relationship between several signals may indicate an abnormal operating condition.

This makes multivariate time-series anomaly detection more difficult than analysing one signal at a time.

The model must learn:

Relationships between multiple signals
Changes across different time windows
Temporal dependencies between successive observations
Deviations from normal operating patterns
Reconstruction differences that may indicate anomalies

The goal of the research was to detect unusual multivariate patterns while preserving the temporal and spatial relationships present in the data.

MSCRED and ConvLSTM Architecture

MSCRED models relationships between multiple time-series signals by constructing signature matrices over different temporal windows.

At a high level, the pipeline contains:

Signature-matrix generation from multivariate signals
A convolutional encoder for spatial feature extraction
ConvLSTM layers for temporal modelling
A decoder that reconstructs expected signal relationships
Reconstruction-error analysis for anomaly scoring

The encoder processes the signature matrices through convolutional operations, while ConvLSTM captures how the encoded representations evolve over time.

Multivariate signals
        ↓
Signature matrices
        ↓
CNN encoder
        ↓
ConvLSTM
        ↓
Decoder
        ↓
Reconstruction error
        ↓
Anomaly score

Why Signature Matrices Are Useful

A signature matrix represents pairwise relationships between signals over a selected time interval.

By generating these matrices at multiple scales, the model can observe both short-term and longer-term relationships. This helps identify anomalies that may not be visible in an individual signal.

Role of the Encoder

The convolutional encoder extracts spatial patterns from the signature matrices. It reduces the raw matrix representation into compact feature maps that capture important relationships between signals.

Role of ConvLSTM

ConvLSTM models how those spatial representations change over time.

Unlike a standard fully connected LSTM, ConvLSTM preserves the spatial structure of feature maps while learning temporal dependencies.

Role of the Decoder

The decoder reconstructs the expected signature matrices.

When the reconstructed output differs significantly from the observed input, the reconstruction error can indicate an anomalous pattern.

Anomaly-Detection Evaluation

The purpose of the MSCRED and ConvLSTM pipeline was to detect abnormal relationships and temporal behaviour across multiple industrial process signals.

The model learned normal multivariate patterns by reconstructing signature matrices. During evaluation, the difference between the observed signature matrices and the reconstructed outputs was used to calculate a reconstruction error.

Observed signature matrix
            ↓
MSCRED reconstruction
            ↓
Reconstruction error
            ↓
Thresholding
            ↓
Normal or anomalous

A larger reconstruction error indicated that the observed relationships differed from patterns learned during normal operation and could therefore represent an anomaly.

The thesis evaluated anomaly-detection quality using the F1 score. Under the reported experimental setup, the best recorded F1 score was 0.941.

This result represents model-quality evaluation and should be interpreted separately from the CUDA runtime measurements presented later. The GPU memory optimisations were designed to reduce computation time, not to improve predictive accuracy.

Where the Performance Bottleneck Appeared

The encoder repeatedly applied convolutional operations to signature matrices.

On the CPU, these operations involved substantial nested computation across:

Input channels
Output filters
Spatial positions
Kernel dimensions
Batch elements

This made the encoder a strong candidate for GPU acceleration because many convolution calculations could be performed in parallel.

The objective was not to redesign the anomaly-detection model. Instead, it was to preserve its expected computational behaviour while accelerating selected expensive operations.

Transforming Conv2D into `im2col` and GEMM

A convolution repeatedly applies a kernel to overlapping regions of an input feature map.

The im2col, or image-to-column, transformation rearranges those local regions into columns of a matrix.

After this transformation, convolution can be expressed as a general matrix multiplication:

Output matrix = Weight matrix × im2col(Input)

This conversion is useful because matrix multiplication is highly parallel and can be executed efficiently on a GPU.

Conceptual Transformation

Input feature map
        ↓
Extract overlapping kernel windows
        ↓
Arrange windows as matrix columns
        ↓
Multiply by reshaped filter matrix
        ↓
Reshape result into output feature map

Why Use GEMM?

GEMM stands for General Matrix-Matrix Multiplication.

Modern GPU architectures are designed to perform large numbers of multiply-and-accumulate operations in parallel. Expressing convolution as GEMM allows the workload to use that parallelism.

Trade-Off

im2col introduces additional memory usage because overlapping input regions are copied into an intermediate matrix.

The method is most effective when the computational benefit of parallel matrix multiplication outweighs the cost of creating and storing the intermediate representation.

CUDA Kernel Design

The CUDA implementation divided the computation across many GPU threads.

Each thread was responsible for part of the output, while thread blocks grouped related work.

Important kernel-design considerations included:

Mapping output elements to threads
Selecting thread-block dimensions
Coalescing memory access where possible
Reducing unnecessary global-memory operations
Avoiding excessive synchronisation
Preserving correct tensor indexing
Managing intermediate buffers efficiently

The implementation required careful validation because incorrect indexing in GPU code can produce plausible-looking but incorrect outputs.

Global Memory and Shared Memory

The first GPU implementation relied primarily on global memory.

Global memory provides access to the complete input, but its latency is considerably higher than that of on-chip shared memory.

The optimised implementation staged reusable tiles inside shared memory. Threads within the same block could then reuse nearby values without repeatedly loading them from global memory.

The shared-memory optimisation was intended to:

Reduce repeated global-memory reads
Improve data locality
Increase data reuse within each thread block
Reduce memory-related kernel overhead
Improve matrix-multiplication efficiency

Conceptual Data Flow

Global memory
      ↓
Load reusable tile
      ↓
Shared memory
      ↓
Threads perform repeated calculations
      ↓
Write result to global memory

Shared memory is limited in capacity, so tile dimensions and thread-block organisation must be selected carefully.

Poorly selected configurations can:

Reduce occupancy
Increase synchronisation overhead
Create bank conflicts
Use too much shared memory per block
Limit the number of active thread blocks

Integrating the CUDA Implementation with PyTorch

To use the accelerated operations inside the existing model pipeline, I connected the CUDA implementation to PyTorch through a custom extension.

The integration enabled PyTorch tensors to be passed to the CUDA implementation and the computed results to be returned to the model pipeline.

Important integration concerns included:

Tensor dimensions
Device placement
Data types
Contiguous-memory assumptions
Compilation and linking
Error handling
CPU-versus-GPU output comparison

Correctness validation was essential. A faster kernel is not useful if it changes tensor shapes, produces incorrect indexing, introduces unacceptable numerical differences, or behaves inconsistently for the tested configurations.

The custom extension made it possible to retain PyTorch for model development while using lower-level CUDA operations for selected performance-critical components.

Experimental Setup

The benchmark used the following environment:

GPU: NVIDIA RTX A6000, 48 GB
CPU: 16-core Intel Xeon Gold 5218
System memory: 64 GB
Data points: 36,000
Number of features: 30
Batch size: 128
CPU threads: 32

Timing Methodology

The timings reported here are measurements from my thesis experiments using the specified hardware and workload. The CPU and GPU implementations used the same input dimensions and batch configuration.

The results should be treated as comparative experimental measurements rather than universal library benchmarks. Performance may vary with implementation details, CUDA configuration, tensor dimensions, data-transfer costs, compiler settings, and hardware.

Performance Benchmark Results

Component	CPU	GPU: global memory	GPU: shared memory
Encoder	300 s	27.492 s	23.664 s
ConvLSTM	1,200 s	909.208 s	847.260 s
Total runtime	2,700 s	1,594 s	1,481 s

Results at a Glance

Anomaly-detection quality: best recorded F1 score of 0.941
Encoder acceleration: approximately 12.68×
End-to-end acceleration: approximately 1.82×
Total runtime reduction: approximately 45.1%

The shared-memory implementation reduced encoder runtime from 300 seconds to 23.664 seconds, producing an approximately 12.68× encoder speed-up.

For the complete measured pipeline, runtime decreased from 2,700 seconds to 1,481 seconds. The total-runtime measurement represents the complete measured pipeline rather than only the encoder and ConvLSTM components shown separately.

Why the Encoder Improved More Than ConvLSTM

The convolutional encoder benefited strongly from GPU execution because its operations provided substantial data parallelism.

ConvLSTM achieved a smaller improvement because of:

Recurrent dependencies
Sequential temporal processing
Memory movement
Repeated state updates
More limited parallelism across time steps

Consequently, the 12.68× encoder acceleration did not translate into a 12.68× end-to-end speed-up.

After the encoder was accelerated, ConvLSTM became the dominant runtime bottleneck.

This is an important performance-engineering lesson: accelerating one component can move the bottleneck to another part of the system.

Correctness Validation

The CUDA implementation was compared with the reference implementation for the tested tensor shapes and configurations.

The optimisation work focused on reducing execution time while preserving the expected computational behaviour. Model-quality evaluation and runtime evaluation were treated as separate concerns.

Engineering Challenges

Several engineering challenges appeared during the implementation.

1. Correct Tensor Indexing

The CUDA kernel had to reproduce the same logical output as the reference convolution operation.

A small indexing error could generate incorrect feature maps without necessarily causing a runtime failure.

2. Memory Management

The im2col transformation created additional intermediate data.

This required careful allocation, reuse, and release of GPU memory.

3. Synchronisation

Threads sharing data through shared memory had to synchronise at appropriate points.

Missing or unnecessary synchronisation could cause incorrect results or reduced performance.

4. Kernel Configuration

Thread-block dimensions and tile sizes affected:

Occupancy
Shared-memory usage
Memory access
Work distribution
Overall kernel performance

5. PyTorch Integration

The extension had to account for tensor shapes, data types, device placement, memory layout, compilation, linking, and output comparison.

6. Measuring the Correct Bottleneck

Improving one component changed the relative cost of other components.

Profiling therefore had to be repeated after major optimisations.

Engineering Lessons

1. Optimising One Component Does Not Guarantee an Equivalent End-to-End Improvement

Once the encoder was accelerated, ConvLSTM represented a larger proportion of the remaining runtime.

2. `im2col` Makes Convolution GPU-Friendly but Introduces Memory Overhead

The transformation enables GEMM-based execution while creating a larger intermediate representation.

3. Shared Memory Is Valuable When Threads Repeatedly Reuse Nearby Data

Its benefit depends on access patterns, tile sizes, synchronisation, and thread-block configuration.

4. Profiling Must Guide Optimisation

Performance work should focus on measured bottlenecks rather than assumptions about which component is slow.

5. Correctness Is as Important as Speed

GPU output should be compared against a trusted reference implementation before performance improvements are accepted.

6. Custom PyTorch Extensions Offer Flexibility at the Cost of Additional Complexity

Device handling, tensor shapes, data types, compilation, numerical validation, and fallback behaviour all require careful attention.

Frequently Asked Questions

What is MSCRED?

MSCRED stands for Multi-Scale Convolutional Recurrent Encoder-Decoder.

It is an architecture designed to model relationships between multiple time-series signals and detect abnormal behaviour through reconstruction error.

What does `im2col` do?

im2col rearranges overlapping convolution windows into matrix columns, allowing convolution to be expressed as matrix multiplication.

Why did shared memory help?

Shared memory allowed threads in the same block to reuse data without repeatedly loading it from slower global memory. This reduced memory-access overhead for reusable tiles.

Did GPU memory optimisation improve the F1 score?

That was not the purpose of the optimisation, and this article does not claim that it did.

The best recorded F1 score describes anomaly-detection quality in the thesis experiments. The CPU and GPU timing results describe computational performance. They should be interpreted separately.

Why was the encoder speed-up much larger than the total speed-up?

After the encoder became faster, ConvLSTM represented a larger share of total runtime. The overall system was therefore limited by the remaining bottleneck.

Is a Custom CUDA Kernel Always Faster Than PyTorch Operations?

No.

Optimised PyTorch and CUDA libraries are already highly efficient for many operations. A custom kernel is most useful when:

The workload has a specialised access pattern
Existing operations create unnecessary overhead
Profiling identifies a clear bottleneck
Correctness and maintenance costs are justified

Can the Original Industrial Dataset Be Shared?

No.

The original dataset contains industrial process information. A public reproduction would require synthetic or appropriately anonymised multivariate time-series data.

Are These Benchmark Results Universal?

No.

Performance depends on GPU architecture, CPU configuration, tensor dimensions, batch size, kernel implementation, memory layout, CUDA and PyTorch versions, compiler settings, and workload characteristics.

The reported numbers apply to the specific experimental setup described in this article.

Limitations and Data-Privacy Considerations

The reported results were obtained on a specific hardware configuration and workload.

Different GPUs, tensor dimensions, batch sizes, memory layouts, CUDA versions, and kernel configurations may produce different results.

The original dataset cannot be published because it contains industrial process information. A public reproduction would therefore require synthetic or appropriately anonymised multivariate time-series data.

This article presents aggregate benchmark measurements and technical methodology rather than the complete industrial dataset or restricted implementation.

Future Work

Future work could focus on:

Profiling and optimising the ConvLSTM component
Reducing intermediate memory allocation
Reducing host-to-device and device-to-device memory movement
Comparing the custom implementation with optimised CUDA libraries
Evaluating additional tile and thread-block configurations
Testing across different GPU architectures
Reproducing the workflow using synthetic public data
Adding automated correctness and regression tests
Measuring kernel-level execution using GPU profiling tools

Conclusion

This work was completed as part of my M.Tech thesis research at the Indian Institute of Technology Bhilai.

It demonstrated how low-level GPU optimisation can be integrated into a deep-learning anomaly-detection pipeline.

The convolutional encoder benefited substantially from:

CUDA parallelisation
im2col-based lowering
GEMM
Shared-memory data reuse
Custom PyTorch integration

Encoder runtime decreased from 300 seconds to 23.664 seconds in the measured workload.

The broader result also illustrated an important systems principle: once one stage is accelerated, another stage may become the new performance bottleneck.

In this case, ConvLSTM limited the end-to-end improvement even after the encoder achieved a substantial speed-up.

The work reinforced that successful performance engineering requires both low-level optimisation and system-level measurement.

References

Zhang, C., Song, D., Chen, Y., Feng, X., Lumezanu, C., Cheng, W., Ni, J., Zong, B., Chen, H., and Chawla, N. V. (2019). A Deep Neural Network for Unsupervised Anomaly Detection and Diagnosis in Multivariate Time Series Data. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01), 1409–1416. https://doi.org/10.1609/aaai.v33i01.33011409
Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W.-K., and Woo, W.-C. (2015). Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting. Advances in Neural Information Processing Systems 28, 802–810. https://proceedings.neurips.cc/paper_files/paper/2015/hash/07563a3fe3bbe7e3ba84431ad9d055af-Abstract.html
NVIDIA. CUDA C++ Programming Guide. https://docs.nvidia.com/cuda/cuda-c-programming-guide/
PyTorch. Custom C++ and CUDA Operators. https://docs.pytorch.org/tutorials/advanced/cpp_custom_ops.html

Citation and Feedback

If this article informs academic or technical work, please cite the original MSCRED and ConvLSTM papers above.

To refer specifically to my thesis implementation, optimisation process, or reported benchmarks, you may cite this article as:

Vutnoor, R. (2026). “GPU-Accelerating MSCRED with CUDA, im2col, GEMM, and a Custom PyTorch Extension.” DEV Community.

If you found the article useful, a reaction, comment, or constructive technical question on DEV would be appreciated.

I am Ranjith Vutnoor, an AI/ML Software Engineer and IIT Bhilai alumnus working on production RAG systems, LLM evaluation, model fine-tuning, PyTorch, and GPU-accelerated machine learning.

DEV Community: pytorch

Top Dev Tools and Tutorials: QuantumAI, NeuroSync, and More for 2026

QuantumAI SDK v2.0

What it is

Why now

Tutorial angle

Difficulty

Start today:

NeuroSync CLI v1.5

What it is

Why now

Tutorial angle

Difficulty

Start today:

TensorFlow Quantum 1.3

What it is

Why now

Tutorial angle

Difficulty

Start today:

PyTorch Lightning v2.0

What it is

Why now

Tutorial angle

Difficulty

Start today:

Hugging Face Transformers v5.0

What it is

Why now

Tutorial angle

Difficulty

Start today:

Building in public: my lead-gen engine

Start with this today

Picks of the week

Connect

Recommenders throw away your single most useful signal: order

One Line of PyTorch Code That Fixes a Silent Memory Leak

Guilded-Guild: recommending the next item with SASRec in PyTorch

Recommending the next thing, not a similar thing

Why SASRec instead of matrix factorization

Inside the model

The numbers, and how to reproduce them

Serving it like a product, not a notebook

What I learned building it

Top Dev Tools & Tutorials of the Week: TensorFlow Quantum 2.0, PyTorch Lightning CLI, and More

Connect

Top Dev Tools and Tutorials of the Week: TensorFlow 5.0, PyTorch Quantum SDK, and More

Evaluating Open Source AI Frameworks for Computer Vision: A Comparative Analysis of TensorFlow and PyTorch Part 1: Introduction

Evaluating Open Source AI Frameworks for Computer Vision: A Comparative Analysis of TensorFlow and PyTorch Part 1: Introduction to Computer Vision

📚 References & Further Reading

Your Turn

Chạy Jupyter PyTorch và pandas bị OOM RAM ở ngưỡng nào

pandas load CSV ngốn RAM gấp 4 lần dung lượng file

PyTorch batch size và hiện tượng tràn RAM sang Swap

40% Adam Optimizer Improvement: PyTorch Benchmarks

40% Adam Optimizer Improvement

Highlights

Model DNA, Analyzed: Verifying 'From-Scratch' LLM Claims with Architecture, Tokenizer, and CKA (PyTorch)

Key takeaways

Why this matters in 2026

The three signals (cited and reproduced)

1. Architecture fingerprint — config.json

2. Tokenizer overlap — tokenizer.json

3. Embedding similarity — Linear CKA

The genotype framework

Two traps that produce confident wrong answers

Deep analysis: how far can you trust it?

What fingerprinting cannot tell you

FAQ

Resources & related links

ResKV: Recovering Evicted Token Contributions via Residual KV Cache — LongBench 32/32

TL;DR

The Problem

How It Works

Shared-Softmax Integration

Validation Proxy

Dynamic Gate

Show Me The Code

Benchmark Results

1. Architecture fingerprint — `config.json`

2. Tokenizer overlap — `tokenizer.json`

Transforming Conv2D into `im2col` and GEMM

2. `im2col` Makes Convolution GPU-Friendly but Introduces Memory Overhead

What does `im2col` do?