When fine-tuning language models like Qwen2-0.5B or Llama-3 on a constrained heap, the biggest memory and compute bottleneck often isn't self-attention. On models with a small hidden dimension and a large vocabulary, the bottleneck is the output projection head (lmHead) and cross-entropy loss.
I ran into this while working on v1.1.0 of Kotensor (io.github.tspstudio:kotensor), an open-source, pure-Kotlin Transformer and reverse-mode autograd engine for JVM 17+ and Android with no native C++ or JNI dependencies. Inference was already reasonably fast and ran within tight mobile memory limits, but running a LoRA fine-tuning step on the full 24-layer Qwen2-0.5B architecture inside a 6 GB JVM heap consistently failed with OutOfMemoryError: Java heap space.
At first I looked for a reference leak in the autograd graph, assuming intermediate activations weren't being freed between steps. Once I actually calculated the tensor sizes at the output layer, though, it became obvious that the problem was just raw array allocation in the output head.
This post goes over the math behind that bottleneck and the changes I made in pure Kotlin to get the full 24-layer model training inside 2.84 GB of heap: prompt loss masking before projection, chunked fused linear cross-entropy, in-place transposed matrix multiplication, and SIMD acceleration via the JVM Vector API (jdk.incubator.vector).
1. Looking at the Numbers on a 151,936-Token Vocabulary
Qwen2-0.5B has a hidden size of D = 896 and a vocabulary size of V = 151,936. That means the final linear projection matrix W_head ([896 x 151,936]) holds 136.1 million parameters, which is roughly 544 MB in FP32.
In a standard autograd training step with a batch of N = 64 tokens, three things happen at once:
-
Logit and Probability Materialization: Multiplying the
[64 x 896]hidden states byW_headproduces a[64 x 151,936]logit tensor (9.72 million floats, or ~38.9 MB). Standard cross-entropy then allocates an equally large array for softmax probabilities and another for the logit gradients during the backward pass (~116 MB total just for the output activations). -
Transposed Weight Copy in Backward: In Kotensor v1.0.0, the general matrix multiplication backward pass (
dH = dLogits * W^T) transposed the right-hand matrix into a temporary array before multiplying. ForW_head, that meant allocating a fresh 544 MBFloatArrayon every step while all 24 layers of forward activations were still live in memory. -
Unmasked Prompt Tokens: In instruction fine-tuning, roughly 70% of a batch consists of system prompt and user query tokens that shouldn't contribute to the loss. Projecting all 64 tokens across 151,936 columns costs 52 billion FLOPs per step, even though most of those rows get masked out right after.
Combined with the activations of 24 Transformer blocks, that 544 MB temporary transpose and 116 MB of vocabulary buffers were enough to push the JVM past a 6 GB heap limit.
---
## 2. Filtering Prompt Tokens Before Projection
Usually,
crossEntropyLoss(logits, targets, ignoreIndex = -100)takes the full[batch, seqLen, vocabSize]logit tensor and skips the masked indices during the softmax loop. That works fine when memory isn't tight, but it means you already paid the cost of multiplying the masked rows throughW_head. To avoid doing that work in the first place, I moved theIGNORE_INDEX = -100check ahead of the linear projection so only the active target positions are gathered:
const val IGNORE_INDEX = -100
private fun activeRows(targets: IntArray, ignoreIndex: Int, vocabularySize: Int): IntArray {
var active = 0
for (target in targets) {
if (target == ignoreIndex) continue
require(target in 0 until vocabularySize) {
"target $target out of bounds $vocabularySize"
}
active++
}
val rows = IntArray(active)
var next = 0
for (i in targets.indices) {
if (targets[i] != ignoreIndex) rows[next++] = i
}
return rows
}
In our benchmark batch of 64 tokens where only 18 tokens belong to the assistant's response, filtering the indices first drops the number of projected rows from 64 to 18. That cuts the output-layer FLOPs by about 72% before changing anything else.
3. Chunked Fused Linear + Cross-Entropy
Filtering out the prompt tokens helps, but on longer responses we still don't want to allocate an [activeTokens x 151,936] tensor or hold onto a full probability matrix until backward() runs.
To fix this, I added fusedCrossEntropyLoss in Loss.kt. Instead of calling lmHead.forward(hidden) and passing the result to the loss function, fusedCrossEntropyLoss takes hidden and head directly and processes the active tokens in fixed chunks of at most FUSED_CHUNK_ELEMENTS = 1 shl 22 floats (4,194,304 elements, or 16 MB).
For each chunk of active rows:
- It copies only
chunkRowshidden states into a smallinputsbuffer and projects them into a reusablelogitsbuffer. - It computes the row-wise
maxLogitandsumExpfor the log-sum-exp loss:
loss_i = maxLogit_i + ln(sum_v(exp(z_{i,v} - maxLogit_i))) - z_{i, target_i}
- While the chunk's logits are still in cache, it overwrites the
logitsbuffer in-place with the analytical softmax gradient(prob - indicator) / activeTokens, and immediately multiplies that gradient back intohiddenGradusingMatMul.multiplyTransposedB(which readsW_headtransposed in-place rather than copying the 544 MB array):
private const val FUSED_CHUNK_ELEMENTS = 1 shl 22
@JvmOverloads
fun fusedCrossEntropyLoss(
hidden: Tensor,
head: Projection,
targets: IntArray,
ignoreIndex: Int = IGNORE_INDEX
): Tensor {
val dimension = head.inSize
val vocabularySize = head.outSize
val weight = (head as? Linear)?.weight
val quantized = (head as? QuantizedLinear)?.weight
val bias = head.bias
val rows = activeRows(targets, ignoreIndex, vocabularySize)
val active = rows.size
if (active == 0) return emptyLoss()
val chunkRows = (FUSED_CHUNK_ELEMENTS / vocabularySize).coerceIn(1, active)
val inputs = FloatArray(chunkRows * dimension)
val logits = FloatArray(chunkRows * vocabularySize)
val rowMax = FloatArray(active)
val rowSum = FloatArray(active)
val scale = 1.0f / active.toFloat()
val hiddenGrad = if (hidden.requiresGrad) FloatArray(active * dimension) else null
val chunkGrad = if (hiddenGrad != null) FloatArray(chunkRows * dimension) else null
var totalLoss = 0.0f
var start = 0
while (start < active) {
val count = minOf(chunkRows, active - start)
projectRows(hidden.data, rows, start, count, dimension, vocabularySize, weight, quantized, bias, inputs, logits)
for (r in 0 until count) {
val offset = r * vocabularySize
var maxLogit = -1e9f
for (v in 0 until vocabularySize) {
val logit = logits[offset + v]
if (logit > maxLogit) maxLogit = logit
}
var sumExp = 0.0f
for (v in 0 until vocabularySize) {
sumExp += kotlin.math.exp(logits[offset + v] - maxLogit)
}
rowMax[start + r] = maxLogit
rowSum[start + r] = sumExp
totalLoss += maxLogit + kotlin.math.ln(sumExp) - logits[offset + targets[rows[start + r]]]
}
if (hiddenGrad != null && chunkGrad != null) {
logitGradients(logits, count, vocabularySize, targets, rows, start, rowMax, rowSum, scale)
if (weight != null) {
MatMul.multiplyTransposedB(logits, weight.data, chunkGrad, count, vocabularySize, dimension, accumulate = false)
} else if (quantized != null) {
MatMul.multiplyByQuantizedTransposed(logits, quantized, chunkGrad, count, accumulate = false)
}
System.arraycopy(chunkGrad, 0, hiddenGrad, start * dimension, count * dimension)
}
start += count
}
val meanLoss = totalLoss / active.toFloat()
lateinit var out: Tensor
out = Tensor(floatArrayOf(meanLoss), intArrayOf(1, 1), listOfNotNull(hidden, weight, bias)) {
val upstreamGrad = out.grad[0]
if (hiddenGrad != null && hidden.requiresGrad) {
val target = hidden.grad
for (r in 0 until active) {
val destination = rows[r] * dimension
val source = r * dimension
for (d in 0 until dimension) {
target[destination + d] += upstreamGrad * hiddenGrad[source + d]
}
}
}
}
return out
}
One side benefit here is how clean the backward closure becomes during LoRA or QLoRA training. Since lmHead is frozen (weight.requiresGrad == false), we don't need to keep the vocabulary logits around at all. When loss.backward() runs, it just scales the [active x 896] hiddenGrad buffer and adds it into hidden.grad.
(Note: if weight.requiresGrad is true for full fine-tuning, the actual implementation in Loss.kt recomputes the chunked logits during the backward closure to accumulate weight.grad via MatMul.multiplyTransposedA, trading a second forward projection for the same 16 MB memory cap).
4. Using the JVM Vector API (jdk.incubator.vector)
For the scalar matrix multiplication kernels in MatMul.kt, I originally structured the inner loops as y[j] += a * x[j] (axpy) over 32 x 256 cache tiles. HotSpot's C2 JIT compiler auto-vectorizes that pattern well as long as both arrays are indexed by a plain loop variable j with no offset arithmetic inside the brackets.
The problem is that multiplyTransposedB naturally wants to compute dot products across rows (sum += a[j] * b[j]), and C2 refuses to auto-vectorize floating-point dot-product reductions because floating-point addition isn't associative.
To get around C2's auto-vectorization limits on desktop and server JVMs, I added an optional SimdKernels implementation using Java's Incubator Vector API (jdk.incubator.vector). It uses FloatVector.SPECIES_PREFERRED and fused multiply-add (fma) across four rows at a time (axpy4 and dot4):
import jdk.incubator.vector.FloatVector
import jdk.incubator.vector.VectorSpecies
@JvmField
internal val FLOAT_SPECIES: VectorSpecies<Float> = FloatVector.SPECIES_PREFERRED
fun axpy4(
y0: FloatArray, y1: FloatArray, y2: FloatArray, y3: FloatArray, x: FloatArray,
a0: Float, a1: Float, a2: Float, a3: Float, len: Int
) {
val species = FLOAT_SPECIES
val step = species.length()
val bound = species.loopBound(len)
val va0 = FloatVector.broadcast(species, a0)
val va1 = FloatVector.broadcast(species, a1)
val va2 = FloatVector.broadcast(species, a2)
val va3 = FloatVector.broadcast(species, a3)
var j = 0
while (j < bound) {
val xv = FloatVector.fromArray(species, x, j)
xv.fma(va0, FloatVector.fromArray(species, y0, j)).intoArray(y0, j)
xv.fma(va1, FloatVector.fromArray(species, y1, j)).intoArray(y1, j)
xv.fma(va2, FloatVector.fromArray(species, y2, j)).intoArray(y2, j)
xv.fma(va3, FloatVector.fromArray(species, y3, j)).intoArray(y3, j)
j += step
}
while (j < len) {
val xv = x[j]
y0[j] += a0 * xv
y1[j] += a1 * xv
y2[j] += a2 * xv
y3[j] += a3 * xv
j++
}
}
When the JVM starts with --add-modules jdk.incubator.vector, MatMul picks up SimdKernels automatically. On Android ART, where the Vector API module doesn't exist, it falls back to the tiled scalar kernels.
5. Benchmarks on Qwen2-0.5B
I measured a rank-8 LoRA training step on the Qwen2-0.5B shape (24 layers, hidden dimension 896, vocabulary 151,936, LoRA attached to wQ and wV, 64-token batch with 18 active response tokens) on a 2-core Intel Xeon container with AVX-512, running JDK 21 with a 6 GB heap limit. Numbers below are the mean of three steps after one warm-up step:
| Metric | v1.0.0 (Naive Head + Unmasked) | v1.1.0 (Fused Head + Masked, Scalar) | v1.1.0 (+ JVM Vector API) |
|---|---|---|---|
| Active Tokens Projected | 64 / 64 | 18 / 64 | 18 / 64 |
| Output-Head FLOPs / Step | 52.0 Billion | 9.8 Billion | 9.8 Billion |
| Output-Head Latency | 2.70 s | 0.70 s | 0.33 s |
| 4-Layer Step Latency | 3.87 s | 1.46 s | 0.81 s |
| Full 24-Layer Step Latency | OutOfMemoryError (> 6 GB) | 4.62 s | 3.04 s |
| Heap Used After Step (24 Layers) | > 6.00 GB (Failed) | 2.84 GB | 2.84 GB |
Output-layer time dropped from 2.70 seconds to 0.33 seconds, and peak memory stayed under 2.85 GB for the full 24-layer step.
With the output head fixed, most of the remaining 3.04 seconds per step is spent propagating input gradients backward through the 24 frozen Transformer blocks (~0.11s per block) so they reach the adapters in the early layers. Because v1.1.0 also added early backward termination via requiresGrad, restricting LoRA to the upper blocks (LoRAAdapter(model, blockIndices = 20 until 24)) lets the autograd engine stop backpropagation at layer 20 and skip layers 0 through 19 entirely.
If you want to inspect the full implementation or test the engine yourself, the code is on GitHub at https://github.com/TSPstudio/kotensor and published on Maven Central as io.github.tspstudio:kotensor:1.2.0.
Top comments (0)