Vincent Tran

Posted on Jul 26 • Originally published at 0xgosu.dev on Jul 23

SIMD for Everyday Programmers: The Parallel Loop Hiding in Plain Sight

#ai #productivity #software

Most performance work starts with subtraction: remove an allocation, avoid a database round trip, choose a better data structure, or stop doing work that nobody needs. Those changes often beat clever instruction-level tuning.

But eventually a profiler points at a small loop that really does need to inspect every value. It scans bytes, compares pixels, counts matches, applies a threshold, sums samples, or searches for a delimiter. The algorithm may already be linear and the data may already be contiguous. At that point the remaining question is not how to avoid the loop. It is how much useful work the processor can do on each trip through it.

That is the approachable part of SIMD: single instruction, multiple data. Instead of loading and comparing one value, a vector instruction applies the same operation across several lanes at once. A 256-bit vector can hold eight 32-bit integers or thirty-two bytes. The exact throughput is not simply the lane count—loads, reductions, branches, cache misses, and surrounding work still cost time—but the processor can retire much more useful work per instruction.

SIMD is associated with dense intrinsic names, architecture manuals, codecs, and parsing libraries such as simdutf and simdjson. Those projects deserve their reputation for sophistication. They are also a misleading introduction. The common form of vectorization is much less exotic: prepare a vector, load a chunk, operate on all lanes, turn the result into something scalar code can use, then clean up the remainder.

Once that shape is familiar, SIMD stops looking like a separate discipline. It becomes another way to write a loop.

The Mental Model: One Operation, Several Lanes

Imagine a scalar loop that counts values above a threshold:

var count: usize = 0;
for (values) |value| {
    if (value > threshold) count += 1;
}

Every iteration loads one value, compares it, and updates state. A vector version groups adjacent values:

values: [12, 91, 34, 88, 7, 63, 55, 2]
threshold: [50, 50, 50, 50, 50, 50, 50, 50]
result: [F, T, F, T, F, T, T, F]

The comparison is element-wise. Lane zero compares with lane zero, lane one with lane one, and so on. There is no inner loop in the source. A vector-capable compiler lowers the operation to the instructions available for the chosen target.

The vector width depends on both hardware and element size. A 128-bit Arm Neon register can carry four u32 values or sixteen bytes. A 256-bit AVX2 register can carry eight u32 values or thirty-two bytes. Wider elements mean fewer lanes; narrower elements mean more. This is why data representation matters. If an algorithm only needs bytes but expands everything to 32-bit integers, it gives up three quarters of its potential lanes before doing any work.

SIMD is not the same as threads. Threads run independent instruction streams and involve scheduling and coordination. SIMD keeps one instruction stream and applies an operation across a small fixed group of values. It is local parallelism inside a core, often with no locks, tasks, or worker pools.

The Five-Part Vector Loop

Everyday SIMD usually follows five steps.

1. Prepare vectors and broadcast constants

A vector-to-vector comparison needs compatible values on both sides. If every input is compared with 0x0F, the scalar threshold must be copied into every lane. This is called a broadcast or splat:

const V = @Vector(8, u32);
const threshold: V = @splat(0x0F);

The conceptual value is:

[0x0F, 0x0F, 0x0F, 0x0F, 0x0F, 0x0F, 0x0F, 0x0F]

This setup belongs outside the hot loop. A sum or checksum may also initialize one or more vector accumulators here. Rebuilding invariant vectors for every chunk wastes the advantage we are trying to create.

2. Load one complete vector at a time

Advance by lanes instead of elements:

while (index + lanes <= input.len) : (index += lanes) {
    const chunk: V = input[index..][0..lanes].*;
    // Work on the full vector.
}

The bound matters. A normal full-width load cannot read beyond the slice merely because the algorithm plans to ignore those lanes. Some instruction sets provide masked loads, and carefully padded buffers can support other strategies, but the simple portable approach processes only complete chunks here.

Contiguous input is the friendly case. Pointer chasing, scattered fields, unpredictable lookups, and per-element allocations can cost more than the arithmetic. This is why data-oriented design often comes before explicit SIMD. Arrays of homogeneous values are easy for the cache, easy for prefetchers, and easy to load as vectors. A linked structure may technically be vectorizable with gather operations, but it is rarely the first target to choose.

3. Apply the lane-wise operation

This is the part people imagine when they hear SIMD:

const matches = chunk > threshold;

The result is a vector of booleans. Arithmetic, minimum, maximum, bitwise operations, and many conversions work the same way. The best candidates perform one regular operation over many independent values.

Branches need more thought. A vector cannot cheaply send lane two through one complex path and lane five through an unrelated path while preserving the full benefit. Simple conditions can often become masks and selects: compute candidate values, then choose per lane. Deep, data-dependent control flow is a warning that the scalar implementation may remain clearer or faster.

Dependencies between iterations are another warning. If element i requires the result of element i - 1, the lanes are not independent. Some reductions and prefix-style operations have vector algorithms, but they no longer fit the easy pattern.

4. Reduce, mask, or store the result

Vector work must eventually answer the original program’s question. There are three common endings:

A transform stores the whole output vector.
An aggregate keeps a vector accumulator and reduces its lanes near the end.
A scan converts comparison lanes into a bit mask and locates a match.

This step varies more than the first three. It also determines whether an apparent vector speedup survives contact with the rest of the algorithm. Comparing eight values in parallel is not useful if code immediately loops over all eight booleans one by one on every iteration.

Good scan loops make the common path cheap. If all lanes are ordinary data, reduce the boolean vector with and and immediately continue to the next chunk. Only when a chunk contains an exceptional value does the code build a mask and inspect its position. The expensive path runs rarely, while long ordinary runs move a vector at a time.

For sums, several independent vector accumulators can help the processor overlap work, followed by one horizontal reduction. Floating-point reductions require extra care: changing the grouping changes rounding, because floating-point addition is not associative. LLVM documents this explicitly and usually requires relaxed floating-point rules for the fastest reordered reduction.

5. Finish with the scalar tail

If the input length is not divisible by the lane count, the vector loop leaves a remainder:

while (index < input.len) : (index += 1) {
    // Original scalar operation.
}

This is the scalar tail. It is deliberately boring. For an eight-lane vector it handles zero to seven values, so a straightforward loop is usually better than elaborate masking.

Keeping the original scalar implementation has another benefit: it can be the fallback for targets where the program has no selected vector path. One piece of simple code becomes both the correctness reference and the cleanup path.

A Real Scan from a Terminal

Mitchell Hashimoto’s Ghostty example makes the pattern concrete. A terminal receives decoded code points and wants to batch a run of ordinary printable content. It needs to stop when it reaches a low-valued control character relevant to that parser path.

The scalar form is almost perfect documentation:

while (end < codepoints.len and codepoints[end] > 0x0F) {
    end += 1;
}

The vector version chooses a lane count supported by its target, creates a vector of u32, and splats the cutoff. It then loads one full chunk and compares every lane with the threshold.

If every comparison is true, an and reduction says the entire chunk is ordinary text. The loop advances by the vector width. If any lane fails, the boolean vector is bit-cast into an integer mask. The code inverts the bits so failures become ones, then counts trailing zeros to obtain the first failing lane. It adds that lane offset to end and exits. The scalar tail handles whatever remains after the last full vector or handles the whole input when no vector path is selected.

The important optimization is not merely “eight comparisons at once.” It is the shape of the data. Terminal text commonly contains long stretches that take the all-true fast path. The reduction rejects an entire vector with one decision. Detailed work happens only in the chunk containing the boundary.

Hashimoto reports theoretical lane-count improvements of four times for 128-bit Neon with u32, eight times for AVX2, and sixteen times for 512-bit vectors. More meaningfully, the measured end-to-end terminal pipeline on an AVX2 desktop improved by roughly five times. That gap between theoretical width and observed application speed is normal. The terminal still has other parsing and state work; vector loads, reductions, mask conversion, and branches are not free; and the CPU can already overlap some scalar instructions.

This is why benchmark claims need a scope. “The comparison handles eight lanes” describes an instruction. “The scan loop is faster” describes a microbenchmark. “Terminal throughput improved fivefold” describes the surrounding application on one machine and workload. All three can be true without being interchangeable.

Why the Compiler Does Not Always Do It

Modern compilers already vectorize many scalar loops. LLVM has a loop vectorizer that widens consecutive iterations and an SLP vectorizer that combines similar neighboring scalar operations. Both are enabled by default. Before writing explicit vector code, build the scalar implementation with real release settings and inspect what happened.

For Clang, optimization remarks make this visible:

clang -O3 \
  -Rpass=loop-vectorize \
  -Rpass-missed=loop-vectorize \
  -Rpass-analysis=loop-vectorize \
  source.c

These reports tell you which loops were vectorized, which were missed, and which statement blocked the transformation. Compiler Explorer or a local disassembler can show the generated instructions, but optimization remarks are often the faster first answer.

Auto-vectorization is especially effective for regular arithmetic with clear bounds and independent memory. It becomes harder when the compiler cannot prove that pointers do not overlap, when calls have unknown side effects, when control flow is complicated, or when an early exit must identify an exact element. Compilers can add runtime alias checks, recognize reductions, and vectorize some early exits, but every transformation must preserve language semantics and pass a profitability model.

Explicit vectors provide intent. They say that operating by lanes is part of the algorithm, not an optional optimization the compiler may rediscover. That predictability is valuable in a proven hot path, but it creates responsibility too: maintainers must preserve the scalar fallback, test multiple architectures, and check that a supposedly generic vector operation did not lower into a slow sequence of scalar calls.

The right order is therefore:

Measure the release build.
Fix needless work and hostile data layout.
Check whether the compiler vectorized the loop.
Write explicit vectors when the hot path still matters and the result is maintainable.
Measure again on representative hardware and data.

Portable Source Is Not Runtime Dispatch

Generic vector syntax avoids writing separate intrinsic dialects for every instruction set. Zig’s @Vector, @splat, comparisons, bitwise operators, and @reduce express the algorithm without naming AVX or Neon intrinsics. The compiler still generates code for the selected target.

That distinction matters for distributed binaries. A binary compiled for a conservative baseline cannot assume every user’s newer CPU supports AVX2 or AVX-512. Projects solve this with multiple compiled implementations and runtime dispatch: detect CPU features during startup, then choose the best compatible function. Google Highway is one library designed around portable SIMD and runtime target selection. Ghostty uses that heavier machinery for its hottest specialized paths while simpler generic vectors remain useful elsewhere.

Runtime dispatch adds binary size, build complexity, and test combinations. It is worthwhile for a widely distributed parser, image library, database, or terminal with heavily exercised kernels. It may be unnecessary for software compiled specifically for one deployment target. “Portable SIMD” can mean portable source, portable baseline behavior, or one binary that dynamically exploits several instruction sets; those are different promises.

Where SIMD Wins—and Where It Does Not

Good first candidates have several traits:

The profiler identifies the loop as meaningful.
Input is large enough to amortize setup and tail handling.
Values are contiguous and use compact element types.
Each lane performs the same comparison, arithmetic, or transformation.
Iterations are mostly independent.
The result can be stored, reduced, or represented as a mask cheaply.
The scalar fallback is simple and testable.

Byte scanning, ASCII classification, delimiter search, pixel transforms, audio mixing, checksums, filters, and numeric kernels often fit. Tiny collections, pointer-heavy graphs, allocation-dominated code, unpredictable per-element branches, and I/O-bound request handlers usually do not.

Memory bandwidth also places a ceiling on improvement. If a loop only copies or adds a trivial value to a huge buffer, the memory subsystem may already be the bottleneck. A wider arithmetic instruction cannot make DRAM deliver bytes indefinitely faster. SIMD shines when it reduces instruction and branch work without merely moving an already bandwidth-limited stream.

Alignment is less frightening than it once was on mainstream CPUs, which commonly support unaligned vector loads, but “supported” does not mean every address and cache-line crossing has identical cost. Use the language or library’s safe load APIs, avoid reading past allocations, and measure the actual target rather than relying on folklore.

Testing Vector Code as an Algorithm

Performance code still needs ordinary correctness tests. The scalar implementation is an excellent oracle: run scalar and vector versions over the same inputs and require identical results.

Exercise lengths around every boundary: zero, one, lanes - 1, exactly lanes, lanes + 1, several full vectors, and a large irregular length. For a scan, place the exceptional value in every possible lane, at the first element, at the last element, in the scalar tail, and nowhere at all. Include repeated exceptional values and values exactly on both sides of the threshold.

Then add property-based or randomized comparisons across many inputs. Run on every supported instruction-set path in CI when possible, not just the developer’s laptop. A runtime dispatcher should also be testable with target selection forced, so an AVX2 machine can still exercise the scalar implementation.

Benchmarks need the same discipline. Prevent dead-code elimination, warm up JITs where relevant, separate setup from the measured kernel, use realistic distributions, and record CPU, compiler, flags, input size, and selected instruction set. Test both best-case long runs and match-heavy data. A scan optimized for rare delimiters may lose its advantage when almost every chunk exits early.

The Skill Is Recognition

Most developers do not need to memorize AVX instruction tables. They do benefit from recognizing the vector-shaped loop hiding in ordinary code.

When a profile reveals a hot scan or transform over contiguous data, ask five questions: What constants should be broadcast? How many values can be loaded safely? Which operation applies independently to every lane? How can the vector result be stored or reduced without serializing all the work? What scalar code handles the remainder and fallback?

That checklist turns SIMD from a mysterious bag of instructions into a repeatable program structure. Sometimes the compiler has already generated it. Sometimes the data layout must change first. Sometimes the workload is too small or branchy and the correct decision is to keep the clear scalar loop.

And sometimes twelve careful lines let a terminal advance through ordinary text several code points at a time. That is not a stunt reserved for optimization specialists. It is a practical loop, expressed at the width the processor was built to run.

DEV Community