Debugging Mandelbrot Rendering in x86‑16 Assembly

Developers new to low‑level graphics often launch a Mandelbrot renderer written in x86‑16 assembly only to see an image that looks nothing like the instructor’s reference. The picture may be garbled, shifted, or badly colored, and the cause is usually a mismatch in how the complex plane is mapped to screen coordinates, an incorrect escape‑time iteration count, or a palette lookup error. These discrepancies are not just cosmetic; they shake confidence when learning assembly graphics and debugging because the abstract algorithm feels detached from the concrete pixels on the screen. This guide explains the most common pitfalls that generate color and scaling errors, shows how to trace them with simple step‑by‑step debugging, and prepares the reader for the next sections on environment setup, testing techniques, and performance optimizations. By the end of the introduction the reader should understand why the output diverges, what to look for in the code, and what tools will be used to bring the visual result into line with the expected Mandelbrot picture.

Fundamentals of Mandelbrot Set Rendering

The Mandelbrot set is defined by iterating the complex recurrence z_{n+1}=z_n^2 + c starting from z_0 = 0. For each pixel the screen coordinates (x,y) are converted to a point c = re + i*im in the complex plane. The typical view maps real values from -2.5 to 1 and imaginary values from -1.5 to 1.5, which scales to a 640×200 VGA screen where pixel (0,0) corresponds to the top‑left corner. The iteration continues until the magnitude of z exceeds an escape radius of 2.0; if this never happens within a maximum iteration count (e.g., 256) the point is treated as belonging to the set. In 16‑bit MASM code the loop uses integer fixed‑point arithmetic, so overflow in multiplication can corrupt the escape test and produce distorted colors. Coloring is performed by mapping the iteration count to a palette; common choices include smooth gradients that highlight the boundary. Because 16‑bit registers can store only up to 32767, any intermediate product must be clamped or split to avoid overflow, which otherwise leads to incorrect iteration counts and visual artifacts.

Setting Up the Development Environment

To develop Mandelbrot set renderers in x86-16 assembly, you must configure a low-level toolchain and emulator environment. Begin by installing MASM (Microsoft Macro Assembler) version 6.11 or later, which supports 16-bit x86 targets. Download it from Microsoft's official archive or use a modern fork like MASM32 for improved compatibility. Pair this with a linker such as LanLx86, which converts MASM object files into executable binary formats suitable for DOSBox emulation.

Next, configure DOSBox to emulate VGA mode 13h, the standard 320x200 resolution mode used for fractal visualization. Create a config.cfg file with GAME=auto and V=w to auto-detect graphics settings, then add settings to enforce the desired resolution. Ensure DOSBox's BIOS is patched to support real-mode interrupts, as graphics operations in mode 13h require direct VGA memory access at segment 0xA000:0x0000.

When building your 16-bit project, use MASM's model tiny directive to ensure code and data reside in contiguous memory segments, simplifying VGA memory mapping. Define segment registers in assembly code—for example, set vesa_seg: dw 0xA000 to target video memory. Link your assembly source (e.g., mandel.asm) with LanLx86 using the command lanlxx86 -t mandel.exe mandel.obj, producing a binary executable.

To debug and test, launch DOSBox and execute the binary via INT 10h interrupts or direct COM port invocation. Enable debug output by writing diagnostic messages to the display or using a DOS debugger like DEBUG.EXE. For interactive debugging, integrate tools like OllyDbg within DOSBox by piping debug symbols to the console. This setup allows you to isolate issues in the escape-time algorithm or color calculations before VGA rendering begins.

Critical attention must be paid to memory model limitations. x86-16 code runs in real mode, where segment registers dictate physical addresses. To draw pixels in mode 13h, rays must calculate offset addresses like A000:0000 + (y*320 + x) and store results via OUTSB instructions. Failure to manage segment boundaries or memory layout will produce distorted fractals. Test your setup with a simple "Hello VGA" program first to verify hardware access before tackling the Mandelbrot algorithm.

This foundation enables precise control over pixel operations while mitigating common pitfalls like overflow in 16-bit fixed-point math or incorrect palette indexing. Proceed with confidence once basic VGA output is validated.

Common Pitfalls causing Image Discrepancies

When translating the Mandelbrot set from mathematics to x86‑16 assembly, several subtle bugs routinely produce distorted or oddly colored images.

Incorrect scaling of complex coordinates to screen space – The mapping from pixel (x, y) to a point on the complex plane must use the exact width and height of the video mode (e.g., 320 × 200 for mode 13h). A common mistake is to divide by the pixel count instead of the coordinate range, which stretches or compresses the fractal horizontally or vertically.

Off‑by‑one pixel errors in the mapping loop – Loop bounds that run from 0 to 319 inclusive for a 320‑column screen generate 321 iterations, shifting the whole image by one column. Using cx as a counter and testing cx <= MAX_X instead of cx < MAX_X is a typical culprit.

Too few or too many iteration limits – An iteration ceiling of 16 yields a blocky silhouette, while 256 may overflow an 8‑bit counter and wrap to zero, producing noisy bands. Choose a limit that fits the counter width (e.g., 100 for an 8‑bit register) and matches the desired detail level.

Palette mismatches between calculation and display – The escape‑time value is often used directly as a color index. If the VGA palette is programmed with a gradient that does not correspond to the iteration range, the fractal appears with unexpected hues or banding.

Integer overflow in iteration counters or color values – In 16‑bit arithmetic, ax can overflow when squaring large fixed‑point components. Guard each multiplication with dx:ax checks or use a 32‑bit accumulator (dx:ax pair) to avoid wrap‑around that corrupts the escape test.

By auditing each of these areas — scaling formulas, loop termini, iteration caps, palette setup, and arithmetic width — developers can eliminate the most common sources of visual discrepancy in a MASM‑based Mandelbrot renderer.

Debugging and Testing Techniques

When you finally get a Mandelbrot renderer to run in MASM, the next hurdle is confirming that every pixel represents the correct iteration count. In 16‑bit assembly the usual high‑level tools are unavailable, so a mix of DOSBox’s built‑in debugger, tiny test harnesses, and offline comparison scripts becomes essential.

1. Breakpoints and Step‑Through in DOSBox Debugger

DOSBox ships with a simple debugger that can be invoked with debug. Load your compiled .exe, then:

; example entry point
start:
    mov ax, @data
    mov ds, ax
    call RenderMandelbrot
    int 20h

Set a breakpoint at the beginning of the inner iteration loop:

debug> break RenderMandelbrot+45   ; address of the `cmp cx, maxIter` instruction
debug> go

When the breakpoint hits, use step to watch how ax (real part) and dx (imaginary part) evolve. Pay close attention to the registers that hold the squared magnitude; a single overflow will flip the color of an entire region.

2. Small Test Patterns

Instead of rendering the whole 320×200 screen, write a tiny loop that draws a single pixel repeatedly while varying one parameter. For instance, a “single‑pixel tracer” can validate the mapping from screen coordinates to the complex plane:

    mov cx, 0          ; column = 0
    mov dx, 0          ; row = 0
    call PlotPixel     ; should plot the top‑left corner
    inc cx
    call PlotPixel     ; next column, same row
    ; repeat for a few steps and then `int 20h`

If the pixel appears where expected, your scaling constants are correct. If not, adjust the fixed‑point factors defined in Section 4.

3. Logging Iteration Counts to a File

DOSBox can mount a host folder as a virtual drive. Use simple DOS file‑IO to dump the iteration count for each pixel into a text file:

    mov ah, 3Dh        ; open/create file
    mov al, 2          ; write‑only
    mov dx, offset filename
    int 21h
    mov bx, ax         ; file handle in BX
    ; inside the pixel loop
    mov al, cl         ; low byte of iteration count
    mov ah, 40h        ; write
    mov cx, 1
    mov dx, offset buffer
    int 21h

After the program finishes, compare the generated iter.txt with a reference file produced by a known‑good C implementation. A diff will instantly highlight rows where the iteration count diverges.

4. Side‑by‑Side Image Comparison

Convert the raw VGA frame buffer (e.g., a 0xA000 segment dump) to a BMP using a small Python script. Render the reference image with the same palette, then place the two BMPs side‑by‑side in an image viewer. Human eyes are excellent at spotting systematic color shifts—often a sign that the escape‑time loop off‑by‑one or that the palette index calculation wraps at 255 instead of 256.

5. Automated Pixel Verification Script

For repeatable testing, automate the diff process. Below is a minimalist Bash‑compatible script that runs the emulator, extracts the frame buffer, and compares it to a golden file stored in ref/:

#!/usr/bin/env bash
EMULATOR="dosbox -conf dosbox_debug.conf"
$EMULATOR -c "mount c ./ && c: && mandel.exe && exit"
# Assume mandel.exe writes a 64000‑byte raw screen dump to SCREEN.RAW
python3 compare.py ref/screen_ref.raw SCREEN.RAW || echo "Pixel mismatch!"

compare.py reads both raw dumps, iterates over each byte, and reports the first coordinate where the values differ. Integrate this script into a make test target so every code change is automatically validated.

Putting It All Together

1. Start with the breakpoint method to locate any overflow in the iteration loop.
2. Validate the coordinate mapping using the single‑pixel test harness.
3. Export iteration counts to a file for numerical verification.
4. Visually compare the rendered BMPs to catch palette anomalies.
5. Lock the process with an automated script that runs on every build.

By chaining these techniques, you turn the mysterious “wrong colors” symptom into a concrete, reproducible bug that can be fixed in MASM itself. The same workflow scales nicely when you later embed the renderer in a larger educational product built on Paradane’s low‑level graphics modules.

---

Optimizing Performance and Visual Quality

In a 16‑bit real‑mode environment every cycle counts, so the Mandelbrot renderer benefits from a handful of low‑level tricks that keep the inner loop tight while improving the final image.

Pre‑computed tables. The escape‑time algorithm only needs squares and a few constants, but smooth coloring (continuous iteration count) requires a logarithm. A 256‑entry log₂ table stored in the data segment lets you replace the expensive fyl2x or a software log routine with a single XLAT lookup. If you implement a rotating zoom, a small sine/cosine table (e.g., 64 entries) avoids the FSIN/FCOS instructions that are unavailable on 8086/8088 and slow on 286+.

Division via shifts. Mapping pixel coordinates to the complex plane involves dividing by the screen width or height. When the dimensions are powers of two (320×200 → 320 = 2⁵×10, not a pure power, but you can scale to 256×256 for a quick prototype) you can replace DIV with SHR/SAR. For arbitrary resolutions, pre‑compute the reciprocal in fixed‑point (Q16.16) and use IMUL followed by a 16‑bit shift, which is far faster than a 32‑bit DIV.

Simple anti‑aliasing. Render the fractal at double the target resolution (e.g., 640×400) and average each 2×2 block into a single 320×200 pixel. The averaging can be done with ADD/SHR on the palette indices, or you can accumulate RGB values in a small buffer before the final palette mapping. This costs roughly 4× the pixel work but yields noticeably smoother edges without floating‑point overhead.

Balancing iteration depth. Deep zooms demand higher MAX_ITER values, which linearly increase runtime. Implement an adaptive limit: start with 64 iterations, and if the average iteration count per frame exceeds a threshold (say 80% of MAX_ITER), raise the limit by 16 for the next frame. Conversely, drop the limit when the view is shallow. This keeps the frame rate playable (≥10 fps on a 286) while preserving detail where it matters.

Measuring cycle counts. Use the BIOS timer interrupt (INT 1Ah, AH=00h) to read the tick count before and after a full frame render. For finer granularity, DOSBox’s built‑in debugger (debug command) can set breakpoints at the loop entry and exit, then report the elapsed cycles. Record the numbers in a log file and plot them after each optimization pass to verify that table lookups, shift‑based scaling, and adaptive iteration actually reduce the cycle count.

By combining these techniques — table‑driven logarithms, shift‑based arithmetic, 2×2 supersampling, adaptive iteration limits, and cycle‑accurate profiling — you can push a Mandelbrot renderer on x86‑16 from a sluggish slideshow to a responsive, visually pleasing demo that still fits comfortably in a 64 KB code segment.

Applying the Concepts to Real Projects

The same iteration engine that drives a DOSBox rasterizer can be packaged as a reusable library. In a web‑based fractal viewer the core routine is called from JavaScript via WebAssembly, passing the current canvas offset and scale. This preserves the original integer‑based fixed‑point calculation while allowing floating‑point coordinates for smoother zooming.

In an educational STEAM kit the algorithm is packaged as a small executable that students run on a Raspberry Pi or an Arduino‑compatible board. By exposing the iteration count as a color index the kit lets learners experiment with parameters such as maxIterations and bailOut without touching graphics APIs.

When targeting modern SDKs like Unity or Vulkan, developers often keep the 16‑bit core for hardware‑limited platforms but compile it into a static library that other modules can link against. This preserves the deterministic behavior needed for reproducible demos while benefiting from contemporary shading and UI frameworks.

Performance profiling using BIOS timer hooks shows that the optimized shift‑based divide replacement reduces per‑pixel latency by roughly twenty percent on 8086‑class CPUs. Pre‑computed logarithm tables further accelerate the color‑mapping step.

Finally, the knowledge gained can be leveraged to prototype a commercial graphics library that offers both a low‑level raster mode for embedded devices and a high‑level API for desktop applications. If integration challenges arise—such as synchronizing palette updates across multiple windows—consulting Paradane provides access to experts familiar with 16‑bit graphics pipelines.