DEV Community: Javad

Advanced GPU Optimization: How can I tech an LLM with CUDA and ROCm? - Part 3

Javad — Mon, 10 Aug 2026 23:47:54 +0000

Welcome back to the final (for now) chapter of our GPU optimization saga! In Part 1, we mastered single-GPU matrix multiplication and built the transformer block. In Part 2, we implemented backpropagation, the AdamW optimizer, and mixed-precision training.

But here is the harsh reality: a single GPU—even an H100 or an MI300X—can barely hold a 70B parameter model in VRAM, let alone train it within a human lifetime. To train state-of-the-art LLMs, we must span hundreds or thousands of GPUs.

In this third part, we will tear down the walls of a single device and venture into the world of distributed training. We will implement All-Reduce for data parallelism, explore Ring-AllReduce algorithms, dive into ZeRO (Zero Redundancy Optimizer) sharding, and write real HIP/C++ code using NCCL (NVIDIA) and RCCL (AMD) to synchronize gradients across nodes.

By the end of this part, you will understand:

How collective communication (All-Reduce) works under the hood.
How to shard optimizer states and gradients (ZeRO Stage 1 & 2).
How to write a fully distributed training loop that scales linearly.
The difference between Data Parallelism, Model Parallelism, and Pipeline Parallelism.

Prerequisites: Completion of Parts 1 & 2, a cluster with at least 2-4 GPUs (they don't have to be same vendor, but we will write portable HIP code), and nccl/rccl libraries installed.

1. The Communication Backbone – NCCL vs. RCCL

CUDA uses NCCL (NVIDIA Collective Communications Library), while ROCm uses RCCL (ROCm Collective Communications Library). Fortunately, they share the exact same API signatures with a prefix change (nccl vs rccl). We can unify them using preprocessor macros.

// Unified header selection
#ifdef __HIP_PLATFORM_AMD__
    #include <rccl/rccl.h>
    #define COMM_ID rcclUniqueId
    #define COMM_INIT rcclCommInitRank
    #define COMM_ALL_REDUCE rcclAllReduce
    #define COMM_GET_ERROR rcclGetErrorString
#else
    #include <nccl.h>
    #define COMM_ID ncclUniqueId
    #define COMM_INIT ncclCommInitRank
    #define COMM_ALL_REDUCE ncclAllReduce
    #define COMM_GET_ERROR ncclGetErrorString
#endif

Initializing the communicator requires a unique ID that is broadcasted to all ranks (typically via MPI, but we can use simple environment variables for a single node):

COMM_ID id;
if (rank == 0) { // rank 0 generates the ID
    COMM_GET_UNIQUE_ID(&id);
}
// In a real cluster, you broadcast this via MPI_Bcast.
// For single-node, we just pass it directly.
COMM_COMM_T comm;
COMM_INIT(&comm, world_size, id, rank);

2. Data Parallelism – The All-Reduce Operation

In Data Parallelism (DP) , each GPU holds a full copy of the model. We feed different micro-batches to each GPU, compute local gradients (dW_local), and then average them across all GPUs.

The mathematical operation is:

dW_global = (1 / world_size) * Σ dW_local_i

This is exactly an All-Reduce operation with the SUM operator (we just divide by world_size separately or use AVG if supported).

2.1 Naïve All-Reduce (Parameter Server)

A naive approach uses a central "server" GPU to receive all gradients, average them, and send them back. This creates a bottleneck. Instead, we use the Ring-AllReduce algorithm, which is bandwidth-optimal.

2.2 Implementing Ring-AllReduce in your Kernel

The Ring algorithm works in two phases over N GPUs connected in a logical ring:

Scatter-Reduce (N-1 steps): Each GPU sends a unique chunk of its gradient to the next GPU while receiving another chunk, accumulating the sum.
All-Gather (N-1 steps): GPUs forward the now-aggregated chunks around the ring until every GPU has the full sum.

Instead of writing this from scratch, we use the highly optimized COMM_ALL_REDUCE:

void sync_gradients(float* d_gradient, int num_elements, 
                    ncclComm_t comm, hipStream_t stream) {
    // In-place All-Reduce: sums gradients across all GPUs
    // Result is stored in d_gradient on every rank.
    COMM_ALL_REDUCE((const void*)d_gradient,  // sendbuff
                    (void*)d_gradient,         // recvbuff (in-place)
                    num_elements,
                    COMM_FLOAT,                // data type
                    COMM_SUM,                  // operation
                    comm, stream);

    // Average the sum to get the mean gradient
    int world_size;
    COMM_COMM_COUNT(comm, &world_size);
    float inv_world = 1.0f / world_size;
    scale_kernel<<<(num_elements+255)/256, 256, 0, stream>>>(d_gradient, inv_world, num_elements);
}

Note on AMD: RCCL uses the exact same function signatures. Just replace nccl with rccl in your linker flags (-lrccl vs -lnccl).

3. ZeRO (Zero Redundancy Optimizer) – Sharding the Memory

Data Parallelism is great, but each GPU still stores the entire model, optimizer states, and gradients. For 175B parameters, this requires ~3.2TB just for weights (in FP32). ZeRO, introduced by Microsoft, shards these components across GPUs to reduce memory footprint.

3.1 ZeRO Stage 1 – Sharding the Optimizer States

In Part 2, we stored m (momentum) and v (variance) for AdamW. These are the same size as the model weights. In Stage 1, we partition these optimizer states: GPU 0 holds the optimizer states for parameters 0 to N/2, and GPU 1 holds the rest.

Implementation modification to our AdamW kernel:
Instead of launching the kernel over all parameters, we launch it only over the local partition.

__global__ void adamw_update_sharded_kernel(float* W, float* dW, float* m, float* v,
                                            int total_params, int rank, int world_size,
                                            ...) {
    int idx = blockIdx.x * blockDim.x + threadIdx.x + rank * (total_params / world_size);
    if (idx >= total_params) return;
    // ... (same update logic as Part 2)
}

During the backward pass, we must All-Gather the updated weights so that every GPU has the full updated model before the next forward pass. This adds a communication step after the optimizer.

3.2 ZeRO Stage 2 – Sharding Gradients

Stage 2 goes further: it shards the gradients (dW) as well. Before the All-Reduce, we only reduce gradients that belong to the local GPU's partition. This reduces the communication volume by up to half.

Pseudocode for the training step with ZeRO-2:

// Forward: All GPUs have full weights (via All-Gather after previous step)
forward_pass();

// Backward: Compute local gradients (full size)
backward_pass();

// Reduce-Scatter: Each GPU sums only its assigned partition of the gradients
reduce_scatter_gradients(dW, local_partition, ...);

// Update: Update only local weights and local optimizer states
adamw_update_local(W_local, dW_local, m_local, v_local, ...);

// All-Gather: Sync the updated weights so everyone has the full model
all_gather_weights(W_full, W_local, ...);

This is the core of modern frameworks like DeepSpeed and FairScale.

4. Hybrid Parallelism – Model + Data Parallelism

For truly massive models (e.g., > 1 Trillion parameters), even ZeRO-3 isn't enough. We need Model Parallelism (MP) , where the linear layers themselves are split across GPUs.

Tensor Parallelism (TP): Splitting the weight matrices row-wise or column-wise. For example, Y = X * W where W is split column-wise. This requires an All-Reduce after every linear layer.
Pipeline Parallelism (PP): Splitting the layers across GPUs (e.g., GPU 0 holds layers 1-10, GPU 1 holds layers 11-20). This requires sending activations (forward) and gradients (backward) between GPUs via P2P (Peer-to-Peer) communication.

Here is how we launch a P2P send/receive in NCCL/RCCL:

// GPU 0 sends activations to GPU 1
if (rank == 0) {
    COMM_SEND(d_activation, size, comm, 1, stream);
} else if (rank == 1) {
    COMM_RECV(d_activation, size, comm, 0, stream);
}

The 3D Parallelism Paradigm (DP + TP + PP) is the secret sauce behind training GPT-4 and Llama 3. You apply TP within a node (using NVLink/Infinity Fabric), PP across nodes, and DP across groups of nodes.

5. Optimizing the Communication Overlap

Communication is expensive. The best way to hide it is Overlap. While the GPU is computing the backward pass for layer 10, we can start the All-Reduce for the gradients of layer 1 in the background.

We achieve this using Streams and Event-based Synchronization:

hipStream_t compute_stream, comm_stream;
hipStreamCreate(&compute_stream);
hipStreamCreate(&comm_stream);

// During backward pass:
for (int layer = 0; layer < num_layers; ++layer) {
    // Compute gradient for this layer on compute_stream
    backward_layer<<<..., compute_stream>>>(...);

    // Record an event when gradient is ready
    hipEventRecord(grad_ready_event, compute_stream);
    // Make comm_stream wait for the event
    hipStreamWaitEvent(comm_stream, grad_ready_event, 0);

    // Launch All-Reduce on comm_stream (async)
    COMM_ALL_REDUCE(..., comm_stream);
}

This way, the GPU spends almost zero idle time waiting for network packets; the computation fills the gaps.

6. The Fully Distributed Training Loop (Final Code)

Combining everything we've learned—ZeRO-2 sharding, overlapping communication, and mixed-precision—here is the skeleton of a single training step for a distributed environment:

void distributed_train_step(float* d_model_weights, ... , int rank, int world_size, 
                            ncclComm_t comm, hipStream_t comp_stream, hipStream_t comm_stream) {

    // 1. Forward Pass (Full model on each GPU)
    forward_pass(d_model_weights, d_activations, ..., comp_stream);

    // 2. Backward Pass (Compute full gradients)
    backward_pass(d_model_weights, d_gradients, ..., comp_stream);

    // 3. Reduce-Scatter Gradients (ZeRO-2)
    hipEventRecord(grad_ready, comp_stream);
    hipStreamWaitEvent(comm_stream, grad_ready, 0);

    int local_size = total_params / world_size;
    COMM_REDUCE_SCATTER(d_gradients, d_gradients_local, ... , comm_stream);
    hipEventRecord(comm_done, comm_stream);
    hipStreamWaitEvent(comp_stream, comm_done, 0);

    // 4. Update local weights & optimizer states (AdamW)
    adamw_update_local<<<..., comp_stream>>>(d_local_weights, d_gradients_local, 
                                              d_momentum_local, d_variance_local, ...);

    // 5. All-Gather to sync the full model
    hipEventRecord(update_done, comp_stream);
    hipStreamWaitEvent(comm_stream, update_done, 0);
    COMM_ALL_GATHER(d_local_weights, d_full_weights, ... , comm_stream);

    // 6. Synchronize the main stream at the very end
    hipStreamSynchronize(comp_stream);
}

Conclusion

We have just crossed the finish line of the GPU optimization marathon. Starting from a simple vector addition in Part 1, we built a single-GPU transformer, added a full training stack with mixed precision in Part 2, and finally shattered the single-GPU ceiling by implementing distributed data parallelism, ZeRO sharding, and overlapping communication using NCCL/RCCL in this final part.

You now possess the architectural blueprints that power every major AI company's training cluster. Whether you are running on a 4-GPU workstation at home or a 4,000-GPU supercomputer, the principles remain exactly the same: reduce communication, maximize compute, and shard everything that can be sharded.

What lies beyond?
If there is demand for a Part 4, we could explore FP8 Quantization for even faster training, Fused Multi-Head Attention (Flash Attention) kernels to optimize memory bandwidth further, or Distributed Checkpointing to save sharded models without crashing the filesystem.

Let me know in the comments below what excites you the most!

Thank you for sticking with me through this deep dive. If you ran into any errors while implementing these kernels, or if you have a specific scenario (e.g., using AMD MI300X with RCCL vs NVIDIA H100 with NCCL) that you want me to elaborate on, drop a comment—I personally respond to all of them.

Until next time, keep your warps converged and your bandwidth saturated!

See ya on the next adventure! 🚀

Advanced GPU Optimization: How can I tech an LLM with CUDA and ROCm? - Part 2

Javad — Sat, 08 Aug 2026 22:29:47 +0000

Hey Dev Community!

Welcome back! In Part 1, we built the foundation: from vector addition to tiled GEMM, and finally assembled a complete forward pass of a Transformer block using HIP (CUDA/ROCm). But a forward pass without gradients is just a very expensive random number generator.

Training an LLM requires the backward pass (backpropagation), an optimizer (like AdamW), and brutal memory management to fit billions of parameters into VRAM. In this second part, we will implement the missing pieces: gradient computation for every layer, the weight update step, mixed-precision training (FP16/BF16), and a fully functional training loop.

By the end of this part, you will understand:

· How to write gradient kernels for Linear layers, Softmax, and LayerNorm.
· How to implement a fused AdamW optimizer on the GPU.
· How to leverage activation checkpointing to trade compute for memory.
· How to run a full training iteration (forward, loss, backward, update) in pure HIP/C++.

Prerequisites: Completion of Part 1, a strong grasp of the chain rule, and a GPU with at least 8GB of VRAM to follow along locally.

The Backward Pass – Gradients of the Linear Layer

The forward pass of a linear layer is Y = X * W + b (ignoring bias for simplicity). During backpropagation, we receive dY (gradient of the loss w.r.t output) and must compute:

· dX = gradient w.r.t input (to pass to the previous layer).
· dW = gradient w.r.t weights (to update the weights).

Mathematically:
dX = dY * W^T
dW = X^T * dY

We can reuse the exact same hipblasSgemm (or rocBLAS) we used before, just transposing matrices.

// Assuming d_Y is [B, S, D] and W is [D, D]
// Compute d_X = d_Y * W^T  (Matrix multiply)
float alpha = 1.0f, beta = 0.0f;
hipblasSgemm(handle, HIPBLAS_OP_N, HIPBLAS_OP_T,
             S, D, D,           // M, N, K
             &alpha, 
             d_Y, S,            // matrix A (dY)
             W, D,              // matrix B (W) transposed internally
             &beta, 
             d_X, S);           // matrix C (dX)

// Compute d_W = X^T * d_Y
hipblasSgemm(handle, HIPBLAS_OP_T, HIPBLAS_OP_N,
             D, D, S,
             &alpha, 
             X, S,              // matrix A (X) transposed
             d_Y, S,            // matrix B (dY)
             &beta, 
             d_W, D);           // matrix C (dW)

For multi-head attention, the gradients are trickier because of the softmax and the Q*K^T multiplication, but the principle is the same: every matrix multiplication in the forward pass corresponds to two matrix multiplications in the backward pass.

Gradient of Softmax and LayerNorm (Custom Kernels)

While GEMMs handle the linear parts, we need custom kernels for the non-linearities.

2.1 Softmax Backward Kernel

Let P be the output of the forward softmax (probabilities). The backward pass computes dX given dY. The formula is:
dX_i = P_i * (dY_i - sum(P_j * dY_j)).

__global__ void softmax_backward_kernel(const float* d_Y, const float* P, 
                                        float* d_X, int rows, int cols) {
    int row = blockIdx.x * blockDim.x + threadIdx.x;
    if (row >= rows) return;

    // First, compute dot product of P and dY for this row
    float dot = 0.0f;
    for (int j = 0; j < cols; ++j) {
        dot += P[row * cols + j] * d_Y[row * cols + j];
    }

    // Second, compute dX = P * (dY - dot)
    for (int j = 0; j < cols; ++j) {
        int idx = row * cols + j;
        d_X[idx] = P[idx] * (d_Y[idx] - dot);
    }
}

2.2 LayerNorm Backward

LayerNorm requires computing gradients w.r.t input X, and the scale/bias parameters gamma and beta. We won't write the full kernel here to save space, but the pattern is:

Compute mean and variance from the forward pass (you stored them).
Compute d_gamma and d_beta by reducing over the hidden dimension.
Compute d_X using the standardized values.

Pro Tip: Store the mean and inv_std (inverse standard deviation) from the forward pass in a small buffer. This saves you from recomputing them during backward.

The AdamW Optimizer – Fused Kernel

After computing dW and db, we need to update the model weights. AdamW is the standard optimizer for LLMs. It maintains two exponential moving averages per parameter: m (momentum) and v (variance).

Instead of launching a separate kernel for each parameter update, we write a fused kernel that updates everything in one pass. This minimizes kernel launch overhead and maximizes memory bandwidth.

__global__ void adamw_update_kernel(float* W, float* dW, float* m, float* v,
                                    int num_params, float lr, float beta1, 
                                    float beta2, float eps, float weight_decay,
                                    int step) {
    int idx = blockIdx.x * blockDim.x + threadIdx.x;
    if (idx >= num_params) return;

    // Bias correction factors
    float bias_correction1 = 1.0f - powf(beta1, step);
    float bias_correction2 = 1.0f - powf(beta2, step);

    // Update biased first moment estimate
    m[idx] = beta1 * m[idx] + (1.0f - beta1) * dW[idx];
    // Update biased second raw moment estimate
    v[idx] = beta2 * v[idx] + (1.0f - beta2) * dW[idx] * dW[idx];

    // Compute bias-corrected estimates
    float m_hat = m[idx] / bias_correction1;
    float v_hat = v[idx] / bias_correction2;

    // Update weight (with weight decay)
    W[idx] = W[idx] - lr * (m_hat / (sqrtf(v_hat) + eps) + weight_decay * W[idx]);

    // Optionally reset gradient to zero for next iteration (or do it async)
    dW[idx] = 0.0f;
}

Launch this kernel over num_params (e.g., 1 billion floats). This is highly efficient and leverages the GPU's massive threading capacity.

Mixed Precision Training (FP16 / BF16)

Modern GPUs (NVIDIA Ampere+ and AMD CDNA+) have dedicated hardware for FP16/BF16 matrix multiplication, effectively doubling throughput. To train an LLM with mixed precision:

Cast weights and activations to FP16/BF16 for the forward and backward passes.
Keep a FP32 master copy of the weights for the optimizer update to avoid gradient underflow.
Loss Scaling: Since FP16 has a small dynamic range, we multiply the loss by a large scalar (e.g., 1024) before backpropagation, and divide the gradients by this scalar before updating.

Here’s how we modify our forward pass:

// Use half type in HIP
#include <hip/hip_fp16.h>

__global__ void cast_and_scale_gradients(half* dW_half, float* dW_float, 
                                         float scale, int n) {
    int idx = threadIdx.x + blockIdx.x * blockDim.x;
    if (idx < n) dW_float[idx] = (float)dW_half[idx] * scale;
}

Note: rocBLAS/cuBLAS supports hipblasHgemm for FP16. Use hipblasGemmEx to choose the compute type (FP32 for accumulation) for maximum precision.

Activation Checkpointing (Trading Compute for Memory)

The biggest bottleneck in LLM training is memory. Storing every activation for backpropagation is impossible for models with 7B+ parameters.

Activation Checkpointing (or Gradient Checkpointing) saves only the input to specific layers (e.g., every 4th transformer block) and recomputes the intermediate activations during the backward pass.

Implementation strategy:

· Before the forward pass of a block, we store the input X in a "checkpoint" buffer.
· During the backward pass, we reload X and rerun the entire forward pass of that block (without storing activations again) just to recompute the activations needed for the backward pass of that block.

This roughly halves your memory footprint but increases compute by ~30-40%. For large LLMs, it is mandatory.

Pseudo-code:

// During Forward
if (layer_idx % checkpoint_interval == 0) {
    hipMemcpy(checkpoint_buffer + offset, d_X, size, hipMemcpyDeviceToDevice);
}
// During Backward
if (layer_idx % checkpoint_interval == 0) {
    // 1. Load X from checkpoint
    // 2. Run Forward pass of this layer (discard output, keep activations)
    // 3. Run Backward pass using recomputed activations
}

The Full Training Loop (Putting It All Together)

Now, we combine everything. A single training iteration in pure HIP/C++ looks like this:

void train_step() {
    // 1. Copy batch from CPU to GPU (async)
    hipMemcpyAsync(d_input, h_input, batch_bytes, hipMemcpyHostToDevice, stream);

    // 2. Forward Pass
    forward_transformer(d_input, d_output, ...); // Uses FP16 for matmuls

    // 3. Compute Loss (Cross Entropy)
    float loss = compute_loss_kernel(d_output, d_labels); 

    // 4. Loss Scaling
    scale_loss_kernel<<<...>>>(d_loss_scaled, loss, loss_scale);

    // 5. Backward Pass (traverses graph in reverse order)
    backward_transformer(d_output, d_input, ...); // Computes FP16 gradients

    // 6. Unscale Gradients (cast to FP32)
    cast_and_scale_gradients<<<...>>>(dW_weights, dW_fp32, 1.0f/loss_scale, n);

    // 7. Apply Gradient Clipping (optional, to prevent exploding gradients)
    float norm = compute_l2_norm_kernel(dW_fp32, n);
    if (norm > max_norm) scale_gradients(dW_fp32, max_norm / norm, n);

    // 8. Optimizer Step (AdamW on FP32 master weights)
    adamw_update_kernel<<<(n+255)/256, 256>>>(W_fp32, dW_fp32, m, v, ...);

    // 9. Copy updated FP32 weights back to FP16 for next forward pass
    cast_fp32_to_fp16_kernel<<<...>>>(W_fp16, W_fp32, n);

    // 10. Synchronize Stream
    hipStreamSynchronize(stream);
}

Performance Profiling – Where is the bottleneck?

You can't optimize what you can't measure. Use these tools:

· NVIDIA: nvprof or Nsight Systems (nsys profile).
· AMD: rocprof or OmniTrace.

Look for these metrics:

Occupancy: Are your warps/wavefronts idle? (Check achieved_occupancy).
Memory Bandwidth: GEMMs should be Compute-bound, but custom kernels (Softmax, LayerNorm) are often Memory-bound. Use shared memory to reduce global reads.
Kernel Launch Overhead: If you have 1,000 tiny kernels, fuse them! (e.g., fuse bias add, activation, and dropout into one kernel).

Conclusion

If you have reached this far, congratulations! You have just built the architectural skeleton of a modern LLM trainer from the ground up—covering GEMM optimization on CUDA/ROCm, custom backward kernels, the AdamW optimizer, mixed precision, and memory-saving checkpointing.

Obviously, frameworks like PyTorch and JAX handle all of this transparently and add distributed training (FSDP, ZeRO, and all-reduce), which we haven't touched. But understanding these low-level primitives makes you a master of GPU computing. You now know exactly what loss.backward() and optimizer.step() do under the hood, regardless of whether you are running on an NVIDIA H100 or an AMD MI300X.

What's next for Part 3?
We will dive into Multi-GPU Distributed Training – implementing All-Reduce, Ring-AllReduce, and sharding strategies (ZeRO stages) using NCCL/RCCL.

Until then, happy kernel coding, and may your occupancy be high and your warp divergence be low!

Have questions about the backward pass, loss scaling, or checkpointing? Drop them in the comments below! I read every single one and will respond in detail.

See you in the next part!

Advanced GPU Optimization: How can I teach an LLM with CUDA and ROCm?

Javad — Mon, 15 Jun 2026 18:48:06 +0000

Hey Dev Community!
I'm glad that I'm back after months! Reason is because of my Country!
But I'm here again, and I want to teach you some new interesting things about AI and GPU!

Part 1: From Zero to Matrix Multiplication – Building the Foundation

Introduction

Training a Large Language Model (LLM) is not magic. At its core, it’s massive matrix multiplication (GEMM), attention mechanisms, and gradient descent – all accelerated by GPUs. But how do you actually teach an LLM using CUDA (NVIDIA) or ROCm (AMD)? This two-part guide will take you from absolute zero to a fully functional training loop, with code examples for both platforms.

By the end, you will understand:

How to write custom CUDA/HIP kernels for LLM building blocks.
How to implement a transformer layer from scratch on GPU.
How to optimize memory and compute for large-scale training.

Prerequisites: Basic C++, some Python, and a GPU (NVIDIA or AMD) with CUDA 11+ or ROCm 5+.

1. Setting Up the Environment

For CUDA (NVIDIA)

nvcc --version   # should show 11.x or 12.x

For ROCm (AMD)

hipcc --version

We'll write portable HIP code that runs on both NVIDIA and AMD. HIP is essentially CUDA with a different prefix (hip instead of cuda). We'll use macros to keep the same codebase.

Example kernel launch:

// HIP (works on both)
#include <hip/hip_runtime.h>
#define CHECK_HIP(cmd) { hipError_t err = cmd; if(err != hipSuccess) { printf("HIP error: %s\n", hipGetErrorString(err)); exit(1); } }

2. The Absolute Basic: Vector Addition (Warm-up)

Before an LLM, let's ensure GPU communication works.

__global__ void vec_add(const float* a, const float* b, float* c, int n) {
    int idx = threadIdx.x + blockIdx.x * blockDim.x;
    if(idx < n) c[idx] = a[idx] + b[idx];
}

int main() {
    int n = 1 << 20;
    float *a, *b, *c; // device pointers
    CHECK_HIP(hipMalloc(&a, n*sizeof(float)));
    CHECK_HIP(hipMalloc(&b, n*sizeof(float)));
    CHECK_HIP(hipMalloc(&c, n*sizeof(float)));
    // ... copy data, launch kernel, copy back
}

This is trivial, but an LLM is just billions of such operations.

3. Matrix Multiplication (GEMM) – The Heart of LLMs

Every transformer block is 90% matrix multiplications: Q·Kᵀ, attention·V, FFN projections.

We'll implement a naïve matmul and then tiled with shared memory.

3.1 Naïve matmul kernel

__global__ void matmul_naive(const float* A, const float* B, float* C, int M, int N, int K) {
    int row = blockIdx.y * blockDim.y + threadIdx.y;
    int col = blockIdx.x * blockDim.x + threadIdx.x;
    if(row < M && col < N) {
        float sum = 0.0f;
        for(int k = 0; k < K; ++k)
            sum += A[row * K + k] * B[k * N + col];
        C[row * N + col] = sum;
    }
}

Launch configuration:

dim3 block(16,16);
dim3 grid((N+15)/16, (M+15)/16);
matmul_naive<<<grid, block>>>(d_A, d_B, d_C, M, N, K);

This is terribly slow for large matrices (1000+). We need tiling.

3.2 Tiled matmul with shared memory

#define TILE_SIZE 16

__global__ void matmul_tiled(const float* A, const float* B, float* C, int M, int N, int K) {
    __shared__ float As[TILE_SIZE][TILE_SIZE];
    __shared__ float Bs[TILE_SIZE][TILE_SIZE];

    int bx = blockIdx.x, by = blockIdx.y;
    int tx = threadIdx.x, ty = threadIdx.y;

    int row = by * TILE_SIZE + ty;
    int col = bx * TILE_SIZE + tx;

    float sum = 0.0f;

    for(int k_tile = 0; k_tile < (K + TILE_SIZE - 1) / TILE_SIZE; ++k_tile) {
        // Load A and B tiles into shared memory with bounds checking
        if(row < M && (k_tile * TILE_SIZE + tx) < K)
            As[ty][tx] = A[row * K + k_tile * TILE_SIZE + tx];
        else As[ty][tx] = 0.0f;

        if(col < N && (k_tile * TILE_SIZE + ty) < K)
            Bs[ty][tx] = B[(k_tile * TILE_SIZE + ty) * N + col];
        else Bs[ty][tx] = 0.0f;

        __syncthreads();

        for(int k = 0; k < TILE_SIZE; ++k)
            sum += As[ty][k] * Bs[k][tx];

        __syncthreads();
    }

    if(row < M && col < N)
        C[row * N + col] = sum;
}

This tiled kernel is ~10x faster than naïve for large matrices. For LLMs, we use highly optimized libraries like cuBLAS/rocBLAS, but writing your own teaches you memory hierarchy.

4. Building a Single Transformer Block on GPU

A transformer block consists of:

Multi-head self-attention (softmax(QKᵀ/√d) V)
Feed-forward network (two linear layers with ReLU)
LayerNorm + residual connections

We'll implement a simplified version in mixed HIP/C++ that can be called from Python via pybind11 or directly used in a C++ training loop.

4.1 Attention kernel (simplified, single head)

We need:

Q, K, V matrices of shape [seq_len, d_head]
Output = softmax(Q·Kᵀ/√d) · V

Let's write a kernel that computes the attention output for one head.

__global__ void attention_kernel(const float* Q, const float* K, const float* V,
                                 float* O, int seq_len, int d_head) {
    extern __shared__ float shared_mem[]; // dynamic shared memory for partial softmax
    int row = threadIdx.y + blockIdx.y * blockDim.y; // query position
    int col = threadIdx.x + blockIdx.x * blockDim.x; // output feature

    if(row >= seq_len || col >= d_head) return;

    // Step 1: compute scores for this query row: scores[j] = Q[row,:] dot K[j,:] / sqrt(d_head)
    // We'll compute in registers and store in shared memory for softmax.
    // For simplicity, we compute the whole row of scores sequentially.
    float* scores = shared_mem; // size = seq_len floats
    float scale = rsqrtf((float)d_head);

    float max_score = -INFINITY;
    for(int j = 0; j < seq_len; ++j) {
        float dot = 0.0f;
        for(int k = 0; k < d_head; ++k) {
            dot += Q[row * d_head + k] * K[j * d_head + k];
        }
        scores[j] = dot * scale;
        if(scores[j] > max_score) max_score = scores[j];
    }

    // Step 2: exp and sum (numerically stable)
    float sum_exp = 0.0f;
    for(int j = 0; j < seq_len; ++j) {
        scores[j] = expf(scores[j] - max_score);
        sum_exp += scores[j];
    }

    // Step 3: weighted sum of V
    float out_val = 0.0f;
    for(int j = 0; j < seq_len; ++j) {
        float attn = scores[j] / sum_exp;
        out_val += attn * V[j * d_head + col];
    }
    O[row * d_head + col] = out_val;
}

This is a single-head, no masking version. Real LLMs have multiple heads, causal masking, and flash attention. But this kernel already teaches you the entire attention mechanism on GPU.

Launch with:

dim3 threads(16, 16); // each block handles 16x16 output elements
dim3 blocks((d_head+15)/16, (seq_len+15)/16);
size_t smem = seq_len * sizeof(float); // shared memory for scores
attention_kernel<<<blocks, threads, smem>>>(d_Q, d_K, d_V, d_O, seq_len, d_head);

5. Multi-Head Attention with Batching

In reality, we have batch_size, num_heads, seq_len, d_head. The total dimension d_model = num_heads * d_head.

We can reshape Q,K,V to [batch, num_heads, seq_len, d_head] and launch the above kernel per head. Or better, use cuBLAS/rocBLAS for matmuls and custom softmax kernels.

Here's a batched multi-head attention using rocBLAS (identical to cuBLAS):

#include <hipblas/hipblas.h>
hipblasHandle_t handle;
hipblasCreate(&handle);

// Assume Q, K, V are [batch, seq_len, num_heads, d_head] in row-major.
// We'll use hipblasGemmStridedBatched.
float alpha = 1.0f / sqrtf(d_head), beta = 0.0f;
hipblasSgemmStridedBatched(handle, HIPBLAS_OP_T, HIPBLAS_OP_N,
                           seq_len, seq_len, d_head,
                           &alpha, K, d_head, seq_len*d_head,
                           Q, d_head, seq_len*d_head,
                           &beta, scores, seq_len, seq_len*seq_len,
                           batch_size * num_heads);
// Then softmax (custom kernel), then gemm with V.

This is much faster than custom kernels because it uses tensor cores/matrix cores.

6. The Forward Pass of a Transformer Layer (Full Code)

We'll now combine all pieces into a single HIP/C++ function that runs a transformer block. This is the core of training an LLM.

void transformer_block_forward(
    float* d_X,        // input [batch, seq_len, d_model]
    float* d_Wq, float* d_Wk, float* d_Wv, float* d_Wo, // weights
    float* d_ffn1, float* d_ffn2,
    float* d_Y,        // output
    int B, int S, int D, int H, int D_head) // dimensions
{
    // D = H * D_head
    // Q = X * Wq  etc.
    float alpha = 1.0f, beta = 0.0f;
    // Gemm for Q, K, V
    hipblasSgemm(handle, HIPBLAS_OP_N, HIPBLAS_OP_N,
                 S, D, D, &alpha, d_X, S, d_Wq, D, &beta, d_Q, S);
    // similarly for K, V
    // ... then reshape to [B, H, S, D_head] (using kernel or view)

    // Attention: compute scores = Q*K^T / sqrt(D_head)
    // Use batched gemm as shown above.

    // Softmax (custom kernel)
    softmax_kernel<<<...>>>(d_scores, B*H*S, S);

    // Output = scores * V
    hipblasSgemmStridedBatched(...);

    // Then Wo projection
    // Residual add
    // LayerNorm
    // FFN: two linear layers with ReLU
}

This is the exact blueprint of a transformer block on GPU.

Do you have any question? Leave them here as comments and we will respond to them.

See ya on Part 2 of this interesting blog! Have a nice time!

Advanced GPU Optimization: Metal & Vulkan Compute from zero to hero - Part 2

Javad — Sat, 14 Feb 2026 16:41:09 +0000

Welcome back, GPU programming experts! You've mastered the fundamentals of Metal and Vulkan compute. Now it's time to explore the advanced features that truly unlock the power of modern GPUs. In this comprehensive guide, we'll dive deep into:

Metal Performance Shaders (MPS) – Apple's library of highly optimized compute primitives.
Metal Ray Tracing – Hardware‑accelerated ray tracing on Apple Silicon.
Vulkan Subgroup Operations – Efficient intra‑workgroup communication.
Vulkan Ray Tracing – Cross‑platform ray tracing extensions.
Vulkan Video – Hardware‑accelerated video encoding/decoding.
Cross‑Platform Alternatives – How SYCL and OpenCL fit into the picture.

By the end, you'll be equipped to build cutting‑edge applications that leverage the full capabilities of modern GPUs.

Part 1: Metal Performance Shaders (MPS)

Metal Performance Shaders is a framework that provides a collection of highly optimized compute and graphics functions for image processing, machine learning, linear algebra, and more. MPS is tightly integrated with Metal and takes advantage of the specific hardware capabilities of Apple GPUs.

1.1 Overview and Setup

MPS is available on all Apple platforms (iOS, macOS, tvOS) with Metal support. To use MPS, import the framework:

#import <MetalPerformanceShaders/MetalPerformanceShaders.h>

MPS kernels are objects that you create with a specific device and then encode into a Metal command buffer. They manage their own internal state and resources, so you don't have to worry about shader compilation or pipeline state.

1.2 Key MPS Features

Image Processing: Convolution, morphology, histogram, resizing, etc.
Matrix Multiplication: MPSMatrixMultiplication for fast linear algebra.
Neural Network: MPSCNNConvolution, MPSCNNFullyConnected, MPSCNNPooling – building blocks for deep learning.
Ray Tracing: MPSRayIntersector for ray‑triangle intersection and acceleration structure building.
Random Number Generation: MPSParallelRandom for GPU‑side random numbers.

1.3 Example: Matrix Multiplication with MPS

Let's see how to perform matrix multiplication using MPSMatrixMultiplication. This is far simpler than writing your own kernel and often faster.

#import <MetalPerformanceShaders/MetalPerformanceShaders.h>

id<MTLDevice> device = MTLCreateSystemDefaultDevice();
id<MTLCommandQueue> commandQueue = [device newCommandQueue];

// Create matrices as MPSMatrix objects (wrappers around MTLBuffer)
MPSMatrixDescriptor *desc = [MPSMatrixDescriptor matrixDescriptorWithRows:1024 columns:1024 rowBytes:1024 * sizeof(float) dataType:MPSDataTypeFloat32];
id<MTLBuffer> bufferA = [device newBufferWithLength:desc.rowBytes * desc.rows options:MTLResourceStorageModeShared];
id<MTLBuffer> bufferB = [device newBufferWithLength:desc.rowBytes * desc.rows options:MTLResourceStorageModeShared];
id<MTLBuffer> bufferC = [device newBufferWithLength:desc.rowBytes * desc.rows options:MTLResourceStorageModeShared];

MPSMatrix *matrixA = [[MPSMatrix alloc] initWithBuffer:bufferA descriptor:desc];
MPSMatrix *matrixB = [[MPSMatrix alloc] initWithBuffer:bufferB descriptor:desc];
MPSMatrix *matrixC = [[MPSMatrix alloc] initWithBuffer:bufferC descriptor:desc];

// Create the multiplication kernel
MPSMatrixMultiplication *matMul = [[MPSMatrixMultiplication alloc] initWithDevice:device transposeLeft:NO transposeRight:NO resultRows:1024 resultColumns:1024 interiorColumns:1024 alpha:1.0 beta:0.0];

// Encode the kernel
id<MTLCommandBuffer> commandBuffer = [commandQueue commandBuffer];
[matMul encodeToCommandBuffer:commandBuffer leftMatrix:matrixA rightMatrix:matrixB resultMatrix:matrixC];
[commandBuffer commit];
[commandBuffer waitUntilCompleted];

// Result in bufferC

That's it! The kernel automatically chooses the optimal algorithm for the device.

1.4 Example: Convolution for Deep Learning

MPS provides a full suite of neural network layers. Here's a simple convolution layer:

MPSCNNConvolutionDescriptor *convDesc = [MPSCNNConvolutionDescriptor cnnConvolutionDescriptorWithKernelWidth:3 kernelHeight:3 inputFeatureChannels:3 outputFeatureChannels:64];
convDesc.strideInPixelsX = 1; convDesc.strideInPixelsY = 1;

MPSCNNConvolution *conv = [[MPSCNNConvolution alloc] initWithDevice:device convolutionDescriptor:convDesc kernelWeights:weightsData biasTerms:biasData flags:MPSCNNConvolutionFlagsNone];

// Assume you have an MPSImage *srcImage
[conv encodeToCommandBuffer:commandBuffer sourceImage:srcImage destinationImage:dstImage];

MPS automatically handles weight layout and uses hardware‑accelerated matrix multiplication or Winograd algorithms.

1.5 MPSRayIntersector for Ray Tracing

MPSRayIntersector provides a fast, high‑level interface for ray tracing on Apple GPUs. It builds acceleration structures (BVHs) and performs ray‑primitive intersection tests.

Basic usage:

Create MPSRayIntersector with a device.
Build an acceleration structure from vertex and index data.
Encode intersection tests with rays.

MPSRayIntersector *intersector = [[MPSRayIntersector alloc] initWithDevice:device];

// Create a vertex buffer and index buffer
id<MTLBuffer> vertexBuffer = [device newBufferWithBytes:vertices length:vertexDataSize options:MTLResourceStorageModeManaged];
id<MTLBuffer> indexBuffer = [device newBufferWithBytes:indices length:indexDataSize options:MTLResourceStorageModeManaged];

// Build acceleration structure
MPSPolygonBuffer *polygonBuffer = [MPSPolygonBuffer polygonBufferWithDevice:device];
polygonBuffer.vertexBuffer = vertexBuffer;
polygonBuffer.indexBuffer = indexBuffer;
NSArray<MPSPolygonBuffer *> *buffers = @[polygonBuffer];
id<MTLBuffer> vertexCountBuffer = ...; // optional per‑polygon buffer

MPSTriangleAccelerationStructure *accel = [[MPSTriangleAccelerationStructure alloc] initWithDevice:device];
accel.polygonBuffers = buffers;
[accel rebuild];

// Prepare ray buffer (with origin, direction, etc.)
id<MTLBuffer> rayBuffer = [device newBufferWithLength:rayCount * sizeof(MPSRayOriginMaskDirection) options:MTLResourceStorageModeManaged];
MPSRayOriginMaskDirection *rays = (MPSRayOriginMaskDirection*)[rayBuffer contents];
// fill rays...

// Encode intersection
MPSIntersectionDataType intersectionDataType = MPSIntersectionDataTypeDistance;
id<MTLBuffer> intersectionBuffer = [device newBufferWithLength:rayCount * sizeof(MPSIntersectionDistance) options:MTLResourceStorageModeManaged];
[intersector encodeIntersectionToCommandBuffer:commandBuffer intersectionType:MPSIntersectionTypeNearest rayBuffer:rayBuffer rayBufferOffset:0 intersectionBuffer:intersectionBuffer intersectionBufferOffset:0 rayCount:rayCount accelerationStructure:accel];

You can then read the intersection distances from intersectionBuffer. MPSRayIntersector supports both triangle and sphere primitives, and can handle large scenes efficiently.

1.6 Performance Tips for MPS

Reuse MPS kernels and acceleration structures across frames.
Use MPSKernelOptions to tune for performance (e.g., MPSKernelOptionsAllowReducedPrecision for FP16).
For neural networks, consider using MPSNNGraph to fuse multiple layers into a single optimized graph.
Profile with Xcode's Metal System Trace to see how MPS kernels are scheduled.

Part 2: Metal Ray Tracing

Beyond MPSRayIntersector, Metal provides a more explicit ray tracing API that integrates with the Metal shading pipeline. This allows you to write custom ray generation, any‑hit, closest‑hit, and miss shaders.

2.1 Metal Ray Tracing Concepts

Acceleration Structure: A BVH built from geometry (primitives).
Intersection Function: A shader that determines if a ray hits a primitive. Metal provides a default triangle intersection, but you can write custom ones.
Ray Tracing Pipeline: Similar to a compute pipeline, but specialized for ray tracing.
Shader Stages: raygen, intersection, anyhit, closesthit, miss.

2.2 Setting Up a Basic Ray Tracer

Step 1: Create an acceleration structure
Similar to MPS, but using Metal's MTLAccelerationStructure APIs. You build a primitive acceleration structure descriptor, then encode the build into a command buffer.

Step 2: Create a ray tracing pipeline
You need a MTLRayTracingPipelineDescriptor that specifies the shader functions.

Step 3: Allocate resources

Ray buffer (origin, direction, etc.)
Intersection result buffer
Texture for output

Step 4: Encode ray tracing work
Use MTLRayTracingCommandEncoder to set the pipeline, bind resources, and dispatch rays.

2.3 Example: Simple Ray Generation Shader (MSL)

#include <metal_stdlib>
using namespace metal;

struct Ray {
    float3 origin;
    float3 direction;
};

struct Intersection {
    float distance;
    uint primitiveIndex;
    float2 barycentricCoord;
};

kernel void raygen(device Ray* rays [[buffer(0)]],
                   device Intersection* intersections [[buffer(1)]],
                   texture2d<float, access::write> output [[texture(0)]],
                   uint2 tid [[thread_position_in_grid]]) {
    // Generate ray for pixel (tid.x, tid.y)
    float2 uv = float2(tid) / float2(output.get_width(), output.get_height());
    rays[tid.y * output.get_width() + tid.x] = Ray{ float3(uv * 2.0f - 1.0f, 0.0f), float3(0,0,1) };

    // Wait for intersection results (simplified)
    Intersection hit = intersections[tid.y * output.get_width() + tid.x];
    float3 color = (hit.distance < 1e30) ? float3(1,0,0) : float3(0,0,0);
    output.write(float4(color, 1), tid);
}

You'll also need a closest‑hit shader that computes color based on material.

2.4 Building the Pipeline

On the host side, you compile the shaders and create a pipeline:

id<MTLLibrary> library = [device newDefaultLibrary];
id<MTLFunction> raygenFunc = [library newFunctionWithName:@"raygen"];
id<MTLFunction> closestHitFunc = [library newFunctionWithName:@"closesthit"];
id<MTLFunction> missFunc = [library newFunctionWithName:@"miss"];

MTLRayTracingPipelineDescriptor *desc = [MTLRayTracingPipelineDescriptor new];
desc.rayGeneratorFunction = raygenFunc;
desc.closestHitFunction = closestHitFunc;
desc.missFunction = missFunc;
desc.maxCallStackDepth = 1; // for recursion
desc.maxPayloadSize = sizeof(float3); // per‑ray data
desc.maxAttributeSize = sizeof(float2); // barycentrics

id<MTLRayTracingPipelineState> pipeline = [device newRayTracingPipelineStateWithDescriptor:desc error:nil];

2.5 Encoding and Dispatch

id<MTLRayTracingCommandEncoder> rtEncoder = [commandBuffer rayTracingCommandEncoder];
[rtEncoder setRayTracingPipelineState:pipeline];
[rtEncoder setRayBuffer:rayBuffer offset:0 atIndex:0];
[rtEncoder setIntersectionBuffer:intersectionBuffer offset:0 atIndex:1];
[rtEncoder setTexture:outputTexture atIndex:0];

// Dispatch rays (one per pixel)
[rtEncoder dispatchRays:MTLSizeMake(width, height, 1)];
[rtEncoder endEncoding];

This is a minimal example. Full ray tracing requires handling acceleration structures, intersection queries, and multiple hit groups.

2.6 Performance Considerations

Use MTLAccelerationStructure for static geometry; rebuild only when needed.
Batch rays in coherent groups for better coherence.
Use the smallest possible payload and attribute sizes.
Profile with Xcode to identify bottlenecks.

Part 3: Vulkan Subgroup Operations

Subgroup operations are a powerful feature in Vulkan (and other APIs) that allow efficient communication and computation among a set of invocations that execute together (a subgroup, roughly corresponding to a CUDA warp or AMD wavefront). They can dramatically improve performance for reductions, scans, and other data‑parallel patterns.

3.1 What are Subgroups?

A subgroup is a collection of invocations within a workgroup that can communicate and synchronize using specialized operations. The size of a subgroup is implementation‑defined but can be queried. On NVIDIA, it's 32; on AMD, 64; on Intel, maybe 8,16,32.

Subgroup operations include:

Vote: subgroupAll(), subgroupAny(), subgroupBallot().
Broadcast: subgroupBroadcast().
Shuffle: subgroupShuffle(), subgroupShuffleXor().
Arithmetic: subgroupAdd(), subgroupMin(), etc. (reductions).
Scan: subgroupInclusiveAdd(), subgroupExclusiveAdd().

3.2 Enabling Subgroup Support in Vulkan

You need to check for the VK_EXT_subgroup_size_control extension or at least the core subgroup support (since Vulkan 1.1, subgroups are core but optional). In your shader, declare:

#version 450
#extension GL_KHR_shader_subgroup_arithmetic : enable
#extension GL_KHR_shader_subgroup_ballot : enable
#extension GL_KHR_shader_subgroup_shuffle : enable

3.3 Example: Parallel Reduction with Subgroups

Suppose you want to sum an array within a workgroup. Without subgroups, you'd use shared memory and barriers. With subgroups, you can do a hierarchical reduction:

#version 450
#extension GL_KHR_shader_subgroup_arithmetic : enable

layout(local_size_x = 256) in;
layout(binding = 0) buffer In { float data[]; } inBuf;
layout(binding = 1) buffer Out { float data[]; } outBuf;

void main() {
    uint gid = gl_GlobalInvocationID.x;
    uint lid = gl_LocalInvocationID.x;
    float val = inBuf.data[gid];

    // First reduce within subgroup
    float subgroupSum = subgroupAdd(val);

    // The first invocation in each subgroup writes to shared memory
    shared float sharedSums[gl_WorkGroupSize.x / gl_SubgroupSize];
    if (subgroupElect()) {
        sharedSums[lid / gl_SubgroupSize] = subgroupSum;
    }
    barrier();

    // One subgroup reduces the shared sums
    if (gl_SubgroupID == 0) {
        float total = (lid < gl_NumSubgroups) ? sharedSums[lid] : 0;
        total = subgroupAdd(total);
        if (subgroupElect() && lid == 0) {
            outBuf.data[gl_WorkGroupID.x] = total;
        }
    }
}

This is much faster than a naive loop because it leverages hardware acceleration.

3.4 Vulkan Subgroup Query on Host

You can query subgroup properties:

VkPhysicalDeviceSubgroupProperties subgroupProps = {};
subgroupProps.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_SUBGROUP_PROPERTIES;
VkPhysicalDeviceProperties2 props2 = {};
props2.sType = VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_PROPERTIES_2;
props2.pNext = &subgroupProps;
vkGetPhysicalDeviceProperties2(physicalDevice, &props2);

uint32_t subgroupSize = subgroupProps.subgroupSize;
// Check supported operations in subgroupProps.supportedOperations

3.5 Best Practices

Use subgroups when you need intra‑workgroup communication with low latency.
Avoid branching that diverges within a subgroup (all invocations in a subgroup should take the same path).
Prefer subgroup operations over shared memory for small reductions/scans.
Fall back to shared memory if subgroup operations are not supported (query via gl_SubgroupSize == 0 or use #ifdef).

Part 4: Vulkan Ray Tracing

Vulkan ray tracing extensions (VK_KHR_ray_tracing_pipeline, VK_KHR_acceleration_structure, etc.) provide a cross‑platform API for hardware‑accelerated ray tracing. They are supported on NVIDIA RTX, AMD RDNA2+, and Intel Arc GPUs.

4.1 Core Concepts

Acceleration Structure: A BVH built from bottom‑level (geometry) and top‑level (instances) structures.
Shader Binding Table (SBT): A table of shader handles for ray generation, hit groups, and miss shaders.
Ray Tracing Pipeline: Combines shader stages (raygen, miss, closesthit, anyhit, intersection).
Ray Tracing Commands: vkCmdTraceRaysKHR to launch rays.

4.2 Building Acceleration Structures

First, create buffers with geometry data. Then use vkCmdBuildAccelerationStructuresKHR to build bottom‑level and top‑level AS.

Example: Build a bottom‑level AS from a triangle mesh:

VkAccelerationStructureGeometryKHR geometry = {};
geometry.sType = VK_STRUCTURE_TYPE_ACCELERATION_STRUCTURE_GEOMETRY_KHR;
geometry.geometryType = VK_GEOMETRY_TYPE_TRIANGLES_KHR;
geometry.flags = VK_GEOMETRY_OPAQUE_BIT_KHR;
geometry.geometry.triangles.sType = VK_STRUCTURE_TYPE_ACCELERATION_STRUCTURE_GEOMETRY_TRIANGLES_DATA_KHR;
geometry.geometry.triangles.vertexFormat = VK_FORMAT_R32G32B32_SFLOAT;
geometry.geometry.triangles.vertexData.deviceAddress = vertexBufferAddress;
geometry.geometry.triangles.vertexStride = sizeof(Vertex);
geometry.geometry.triangles.maxVertex = vertexCount - 1;
geometry.geometry.triangles.indexType = VK_INDEX_TYPE_UINT32;
geometry.geometry.triangles.indexData.deviceAddress = indexBufferAddress;

VkAccelerationStructureBuildRangeInfoKHR range = {};
range.primitiveCount = indexCount / 3;
range.primitiveOffset = 0;
range.firstVertex = 0;
range.transformOffset = 0;

// Create and build bottom AS...

You'll need to query the required memory size, allocate, and then build.

4.3 Shader Example (GLSL)

Ray generation shader:

#version 460
#extension GL_EXT_ray_tracing : require

layout(binding = 0) uniform accelerationStructureEXT topLevelAS;
layout(binding = 1, rgba8) uniform image2D image;
layout(location = 0) rayPayloadEXT vec3 hitColor;

void main() {
    ivec2 pixel = ivec2(gl_LaunchIDEXT.xy);
    vec2 dim = vec2(gl_LaunchSizeEXT.xy);
    vec2 d = (vec2(pixel) + vec2(0.5)) / dim - 0.5;
    float aspect = dim.x / dim.y;
    vec3 origin = vec3(0,0,-2);
    vec3 direction = normalize(vec3(d.x * aspect, -d.y, 1));

    traceRayEXT(topLevelAS, gl_RayFlagsOpaqueEXT, 0xff, 0, 0, 0, origin, 0.001, direction, 1000.0, 0);
    imageStore(image, pixel, vec4(hitColor, 1));
}

Closest hit shader:

#version 460
#extension GL_EXT_ray_tracing : require

layout(location = 0) rayPayloadInEXT vec3 hitColor;
hitAttributeEXT vec2 attribs;

void main() {
    hitColor = vec3(1,0,0); // red
}

Miss shader:

#version 460
#extension GL_EXT_ray_tracing : require

layout(location = 0) rayPayloadInEXT vec3 hitColor;

void main() {
    hitColor = vec3(0,0,0); // black
}

4.4 Creating a Ray Tracing Pipeline

You need to create shader modules for each stage, then a ray tracing pipeline with VkRayTracingPipelineCreateInfoKHR. This includes the shader groups (raygen, miss, hit groups).

4.5 Shader Binding Table (SBT)

The SBT is a buffer that holds shader handles and any local data for each shader group. You must fill it according to the pipeline layout. It's complex but crucial.

4.6 Dispatching Rays

After recording the SBT buffer, use:

vkCmdTraceRaysKHR(commandBuffer, &raygenSBT, &missSBT, &hitSBT, &callableSBT, width, height, 1);

4.7 Performance Tips

Use opaque geometry and avoid any‑hit shaders when possible.
Keep ray payload small.
Group rays coherently (e.g., sort by direction).
Use inline ray tracing (VK_KHR_ray_query) for simpler tasks without full pipeline overhead.

Part 5: Vulkan Video

Vulkan Video extensions (VK_KHR_video_queue, VK_KHR_video_decode_queue, VK_KHR_video_encode_queue) provide hardware‑accelerated video encoding and decoding. This is a relatively new area but powerful for applications needing real‑time video processing.

5.1 Overview

Vulkan Video exposes video codec capabilities as Vulkan queues and operations. You can decode H.264, H.265, etc., directly to Vulkan images, and encode from images. It integrates seamlessly with the rest of Vulkan for post‑processing and rendering.

5.2 Setting Up Video Queues

When creating a logical device, you need to request a video decode/encode queue:

float priority = 1.0f;
VkDeviceQueueCreateInfo queueCreateInfo = {};
queueCreateInfo.sType = VK_STRUCTURE_TYPE_DEVICE_QUEUE_CREATE_INFO;
queueCreateInfo.queueFamilyIndex = videoQueueFamilyIndex;
queueCreateInfo.queueCount = 1;
queueCreateInfo.pQueuePriorities = &priority;

You must first query queue families that support VK_QUEUE_VIDEO_DECODE_BIT_KHR or VK_QUEUE_VIDEO_ENCODE_BIT_KHR.

5.3 Decoding a Video Frame

Create a video session (VkVideoSessionKHR) with codec parameters.
Create video picture resources (reference and output images).
Submit decode operations via vkCmdDecodeVideoKHR.

Example decode command:

VkVideoDecodeInfoKHR decodeInfo = {};
decodeInfo.sType = VK_STRUCTURE_TYPE_VIDEO_DECODE_INFO_KHR;
decodeInfo.srcBuffer = bitstreamBuffer;
decodeInfo.srcBufferOffset = bitstreamOffset;
decodeInfo.srcBufferRange = bitstreamSize;
decodeInfo.dstPictureResource = pictureResource; // reference to image view
decodeInfo.pSetupReferenceSlots = ...; // for DPB

vkCmdDecodeVideoKHR(commandBuffer, &decodeInfo);

You'll need to manage the decoded picture buffer (DPB) and reference frames.

5.4 Encoding

Similarly, you can encode frames by providing raw images and receiving a bitstream.

5.5 Limitations and Considerations

Vulkan Video is still evolving; not all vendors support all codecs.
You need to handle bitstream parsing yourself (the API expects raw NAL units).
It's a low‑level interface; for simpler use, consider libraries like FFmpeg with Vulkan acceleration.

Part 6: Cross‑Platform Considerations

While Metal and Vulkan are powerful, they are platform‑specific. For applications that need to run on both Apple and non‑Apple hardware, you have several options:

6.1 SYCL

SYCL (as covered in previous tutorials) is a high‑level C++ abstraction that can target multiple backends including OpenCL, Level Zero, and CUDA. With SYCL 2020, you can write a single source that compiles to various devices. However, SYCL does not yet directly target Metal or Vulkan compute, but it can target OpenCL which runs on many platforms (including macOS via Intel GPU or with limitations). For Apple Silicon, SYCL is not a direct fit; you'd need to use OpenCL (which is deprecated on macOS) or a SYCL implementation that supports Metal (none currently).

6.2 OpenCL

OpenCL is widely supported (including on Intel GPUs, AMD, NVIDIA), but Apple has deprecated OpenCL on macOS in favor of Metal. For cross‑platform, you might write OpenCL kernels and use them on non‑Apple platforms, and rewrite for Metal on Apple. This is double the work.

6.3 Vulkan Everywhere?

Vulkan runs on Windows, Linux, Android, and macOS via MoltenVK. So you could write your compute kernels in Vulkan and use MoltenVK to run on macOS. MoltenVK translates Vulkan to Metal, and for compute, it works quite well. This gives you a single codebase for all platforms. However, you lose some Metal‑specific optimizations, but the portability is valuable.

For ray tracing, Vulkan ray tracing extensions are not yet supported through MoltenVK (Metal ray tracing is different). So if you need ray tracing on Apple, you'd need separate Metal code.

6.4 Recommendations

If you target only Apple: Use Metal and MPS for maximum performance.
If you target multiple platforms and need compute only: Use Vulkan with MoltenVK for macOS, or SYCL with appropriate backends.
If you need ray tracing across platforms: Write separate backends for Metal and Vulkan, as the APIs differ significantly.
If you need video encode/decode: Vulkan Video is cross‑vendor but not yet on Apple; use platform‑specific APIs (VideoToolbox on macOS, MediaFoundation on Windows, VAAPI on Linux).

Conclusion

You've now explored the advanced corners of Metal and Vulkan: the high‑level performance primitives of MPS, the excitement of ray tracing on both platforms, the efficiency of subgroup operations, and the emerging field of Vulkan Video. Armed with this knowledge, you can build applications that push the boundaries of what's possible on modern GPUs.

Remember, the key to mastery is practice. Implement a small path tracer in Metal, try a subgroup‑accelerated reduction in Vulkan, decode a video stream with Vulkan Video, and see how MPS can accelerate your machine learning models. The possibilities are endless.

If you have questions or want to share your own projects, drop a comment below. Happy coding!

Advanced GPU Optimization: Metal & Vulkan Compute from zero to hero

Javad — Sat, 14 Feb 2026 16:38:13 +0000

Welcome to the exciting world of GPU compute programming! In this comprehensive guide, we'll explore two powerful, low‑level APIs: Metal for Apple devices (especially Apple Silicon) and Vulkan Compute for cross‑platform high‑performance computing. By the end, you'll be able to write efficient compute kernels, manage GPU resources, and optimize for real‑world tasks like matrix multiplication and image processing.

Whether you're a macOS/iOS developer wanting to leverage the Neural Engine and GPU on Apple Silicon, or a cross‑platform developer aiming for maximum performance on Windows, Linux, or Android, this tutorial has you covered.

Introduction: Why Metal and Vulkan Compute?

Metal

Metal is Apple's graphics and compute API, available on all modern Apple devices (iPhone, iPad, Mac). With the advent of Apple Silicon (M1, M2, M3 families), the GPU is tightly integrated with the CPU, offering unified memory and exceptional performance. Metal provides a low‑overhead, explicit API that gives you fine‑grained control over the GPU.

Vulkan Compute

Vulkan is a cross‑platform graphics and compute API from the Khronos Group. Its compute capabilities are exposed through compute pipelines and shaders written in GLSL or HLSL (compiled to SPIR‑V). Vulkan is designed for high‑performance, low‑latency applications and runs on Windows, Linux, Android, macOS (via MoltenVK), and more.

Both Metal and Vulkan follow a similar explicit, command‑buffer‑based model. Understanding one makes learning the other easier. Let's dive in!

Part 1: Metal on Apple Silicon

1.1 Setting Up Metal

To develop with Metal, you'll need:

A Mac with Xcode installed (Xcode includes the Metal compiler, framework, and debugging tools).
An Apple Silicon Mac (M1 or later) for native testing; Intel Macs also work but performance characteristics differ.
For iOS/tvOS, you'll need an Apple Developer account (free for device testing).

Create a new Xcode project (e.g., a command‑line tool for macOS) and add the Metal framework. In your source file, import Metal:

#import <Metal/Metal.h>
// or in Swift: import Metal

1.2 Metal Shading Language (MSL)

Metal uses its own C++‑based shading language, MSL. For compute, you write kernels (functions with kernel keyword). Basic MSL features:

Data types: float, int, half, vectors (float4, int2), matrices (float4x4).
Address spaces: device (global memory), threadgroup (shared memory within a workgroup), constant (read‑only), thread (private).
Built‑in variables:
- uint3 [[thread_position_in_grid]] – global thread ID.
- uint3 [[thread_position_in_threadgroup]] – local thread ID within threadgroup.
- uint3 [[threadgroup_position_in_grid]] – threadgroup ID.
- uint3 [[threads_per_threadgroup]] – size of threadgroup.

Example kernel that adds two vectors:

#include <metal_stdlib>
using namespace metal;

kernel void vecAdd(device const float* a [[buffer(0)]],
                   device const float* b [[buffer(1)]],
                   device float* c [[buffer(2)]],
                   uint id [[thread_position_in_grid]]) {
    c[id] = a[id] + b[id];
}

Note the use of [[buffer(n)]] to bind resources.

1.3 Metal Compute Pipeline

The host side (Objective‑C or Swift) sets up the pipeline:

MTLDevice: The abstract representation of the GPU.
MTLCommandQueue: Queue for submitting command buffers.
MTLComputePipelineState: Compiled compute pipeline (kernel + device).
MTLCommandBuffer: Container for commands.
MTLComputeCommandEncoder: Encodes compute commands.
Dispatch: Specify threadgroup dimensions.

1.4 Your First Metal Compute Program: Vector Addition

Let's walk through the host code in Objective‑C. We'll assume you have a Metal device and have compiled the kernel source.

Step 1: Get the default device

id<MTLDevice> device = MTLCreateSystemDefaultDevice();

Step 2: Create a command queue

id<MTLCommandQueue> commandQueue = [device newCommandQueue];

Step 3: Load the kernel

// Assume you have a .metal file in your project named "kernel.metal"
id<MTLLibrary> library = [device newDefaultLibrary];
id<MTLFunction> kernelFunction = [library newFunctionWithName:@"vecAdd"];

Step 4: Create the pipeline state

NSError *error = nil;
id<MTLComputePipelineState> pipelineState = [device newComputePipelineStateWithFunction:kernelFunction error:&error];

Step 5: Create buffers

NSUInteger dataSize = VEC_SIZE * sizeof(float);

id<MTLBuffer> bufferA = [device newBufferWithBytes:h_a length:dataSize options:MTLResourceStorageModeShared];
id<MTLBuffer> bufferB = [device newBufferWithBytes:h_b length:dataSize options:MTLResourceStorageModeShared];
id<MTLBuffer> bufferC = [device newBufferWithLength:dataSize options:MTLResourceStorageModeShared];

MTLResourceStorageModeShared means the buffer is accessible by both CPU and GPU (unified memory on Apple Silicon).

Step 6: Create command buffer and encoder

id<MTLCommandBuffer> commandBuffer = [commandQueue commandBuffer];
id<MTLComputeCommandEncoder> encoder = [commandBuffer computeCommandEncoder];
[encoder setComputePipelineState:pipelineState];
[encoder setBuffer:bufferA offset:0 atIndex:0];
[encoder setBuffer:bufferB offset:0 atIndex:1];
[encoder setBuffer:bufferC offset:0 atIndex:2];

Step 7: Dispatch threads

MTLSize threadgroupSize = MTLSizeMake(256, 1, 1); // threads per threadgroup
MTLSize threadgroupCount = MTLSizeMake((VEC_SIZE + 255) / 256, 1, 1);
[encoder dispatchThreadgroups:threadgroupCount threadsPerThreadgroup:threadgroupSize];

Step 8: End encoding and commit

[encoder endEncoding];
[commandBuffer commit];
[commandBuffer waitUntilCompleted]; // or use completion handler

Step 9: Read results

float *c_ptr = (float*)[bufferC contents];
// c_ptr now contains the result

Step 10: Verify and clean up (ARC will handle objects).

That's it! You've just run your first Metal compute kernel.

1.5 Memory Management in Metal

Metal buffers are created with a storage mode:

MTLResourceStorageModeShared: CPU and GPU access same memory (ideal for Apple Silicon).
MTLResourceStorageModePrivate: Only GPU accessible; used for intermediate buffers.
MTLResourceStorageModeManaged: For discrete GPUs (Intel Macs); requires explicit synchronization.

On Apple Silicon, use Shared for simplicity and performance.

You can also create textures (id<MTLTexture>) for image data. Textures are useful for compute kernels that need spatial locality.

1.6 Performance Considerations

Threadgroup memory (shared memory)

Metal provides threadgroup memory (like CUDA shared memory) for fast data exchange within a workgroup. Declare it in the kernel:

kernel void myKernel(device float* in [[buffer(0)]],
                     device float* out [[buffer(1)]],
                     threadgroup float* shared [[threadgroup(0)]],
                     uint tid [[thread_position_in_threadgroup]],
                     uint gid [[thread_position_in_grid]]) {
    // use shared[tid] ...
}

On the host, you specify the size of threadgroup memory when dispatching.

Coalescing and bank conflicts

Threads in a threadgroup access memory. Ensure that within a warp (32 threads on Apple GPUs), accesses are coalesced. Avoid bank conflicts in threadgroup memory by accessing different banks.

Occupancy and threadgroup size

Use pipelineState.threadExecutionWidth to get the optimal threadgroup size multiple. Typical sizes are multiples of 32.

1.7 Advanced Metal Compute

Using textures

Compute kernels can read/write textures. Useful for image processing.

kernel void sobel(texture2d<float, access::read> inTexture [[texture(0)]],
                  texture2d<float, access::write> outTexture [[texture(1)]],
                  uint2 gid [[thread_position_in_grid]]) {
    float4 color = inTexture.read(gid);
    // compute...
    outTexture.write(result, gid);
}

Multiple dispatches and synchronization

You can encode multiple dispatches in one command buffer, and even use MTLComputeCommandEncoder's setThreadgroupMemoryLength to adjust per‑dispatch.

Profiling

Use Xcode's GPU Debugger and Metal System Trace to analyze performance, view occupancy, and find bottlenecks.

1.8 Example: Tiled Matrix Multiplication in Metal

Let's implement a tiled matrix multiplication kernel using threadgroup memory.

Kernel (MSL):

kernel void matMulTiled(device const float* A [[buffer(0)]],
                        device const float* B [[buffer(1)]],
                        device float* C [[buffer(2)]],
                        constant uint& N [[buffer(3)]],
                        threadgroup float* As [[threadgroup(0)]],
                        threadgroup float* Bs [[threadgroup(1)]],
                        uint2 tgid [[threadgroup_position_in_grid]],
                        uint2 lid [[thread_position_in_threadgroup]],
                        uint2 dims [[threads_per_threadgroup]]) {
    uint tileSize = dims.x; // assuming square threadgroup
    uint row = tgid.y * tileSize + lid.y;
    uint col = tgid.x * tileSize + lid.x;

    float sum = 0.0f;
    for (uint tile = 0; tile < N / tileSize; ++tile) {
        // Load tile of A and B
        As[lid.y * tileSize + lid.x] = A[row * N + (tile * tileSize + lid.x)];
        Bs[lid.y * tileSize + lid.x] = B[(tile * tileSize + lid.y) * N + col];
        threadgroup_barrier(mem_flags::mem_threadgroup);

        // Compute partial product
        for (uint k = 0; k < tileSize; ++k) {
            sum += As[lid.y * tileSize + k] * Bs[k * tileSize + lid.x];
        }
        threadgroup_barrier(mem_flags::mem_threadgroup);
    }
    C[row * N + col] = sum;
}

Host setup:

Buffer A, B, C.
Set N as a constant buffer.
Dispatch with threadgroup size (16,16) and grid size (N/16, N/16).

This kernel leverages threadgroup memory for data reuse, significantly improving performance.

Part 2: Vulkan Compute

2.1 Setting Up Vulkan

To develop Vulkan compute applications, you'll need:

Vulkan SDK from LunarG. Install and set VULKAN_SDK environment variable.
A Vulkan‑capable GPU and drivers (most modern GPUs from NVIDIA, AMD, Intel).
For macOS, you can use MoltenVK (included in the SDK) which translates Vulkan to Metal. Performance is good for compute.

Create a C++ project and include vulkan/vulkan.h. Link against the Vulkan loader (vulkan-1 on Windows, vulkan on Linux).

2.2 Vulkan Compute Pipeline

The Vulkan compute pipeline involves several steps:

Create a VkInstance (with validation layers for debugging).
Pick a physical device (VkPhysicalDevice) that supports compute.
Create a logical device (VkDevice) with a compute queue.
Create a command pool and allocate command buffers.
Create shader module from SPIR‑V.
Create descriptor set layout and pipeline layout.
Create compute pipeline.
Create buffers and allocate device memory.
Update descriptor sets to bind buffers.
Record command buffer: bind pipeline, bind descriptor sets, dispatch.
Submit to queue and synchronize.

Let's go through each step with code.

2.3 GLSL for Compute Shaders

We'll write compute shaders in GLSL (or HLSL compiled to SPIR‑V). A simple vector addition shader:

#version 450
layout(local_size_x = 256, local_size_y = 1, local_size_z = 1) in;

layout(binding = 0) buffer InputA { float data[]; } a;
layout(binding = 1) buffer InputB { float data[]; } b;
layout(binding = 2) buffer OutputC { float data[]; } c;

void main() {
    uint id = gl_GlobalInvocationID.x;
    c.data[id] = a.data[id] + b.data[id];
}

Compile this to SPIR‑V using glslangValidator or glslc. For example:

glslc -c -o vecAdd.spv vecAdd.comp

2.4 Memory and Resources

Vulkan uses VkBuffer objects backed by VkDeviceMemory. You must allocate memory, bind it, and map it for host access.

Descriptor sets are used to bind resources (buffers, images) to shaders. You need:

VkDescriptorSetLayout describing the bindings.
VkDescriptorPool to allocate descriptor sets.
VkDescriptorSet containing actual buffer references.

Push constants are small amounts of data that can be set directly in command buffers (fast).

2.5 Your First Vulkan Compute Program: Vector Addition

Let's outline the host code in C++. Error checking omitted for brevity.

Step 1: Create instance and pick device (standard Vulkan initialization). We'll assume you have a VkDevice and a compute queue.

Step 2: Create shader module

VkShaderModuleCreateInfo createInfo = {};
createInfo.sType = VK_STRUCTURE_TYPE_SHADER_MODULE_CREATE_INFO;
createInfo.codeSize = codeSize;
createInfo.pCode = (uint32_t*)spvCode;
vkCreateShaderModule(device, &createInfo, nullptr, &shaderModule);

Step 3: Create descriptor set layout (bindings for three buffers)

VkDescriptorSetLayoutBinding bindings[3] = {};
bindings[0].binding = 0; bindings[0].descriptorType = VK_DESCRIPTOR_TYPE_STORAGE_BUFFER; bindings[0].descriptorCount = 1; bindings[0].stageFlags = VK_SHADER_STAGE_COMPUTE_BIT;
// similarly for 1 and 2
VkDescriptorSetLayoutCreateInfo layoutInfo = {};
layoutInfo.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_SET_LAYOUT_CREATE_INFO;
layoutInfo.bindingCount = 3; layoutInfo.pBindings = bindings;
vkCreateDescriptorSetLayout(device, &layoutInfo, nullptr, &descriptorSetLayout);

Step 4: Create pipeline layout

VkPipelineLayoutCreateInfo pipelineLayoutInfo = {};
pipelineLayoutInfo.sType = VK_STRUCTURE_TYPE_PIPELINE_LAYOUT_CREATE_INFO;
pipelineLayoutInfo.setLayoutCount = 1; pipelineLayoutInfo.pSetLayouts = &descriptorSetLayout;
vkCreatePipelineLayout(device, &pipelineLayoutInfo, nullptr, &pipelineLayout);

Step 5: Create compute pipeline

VkComputePipelineCreateInfo pipelineInfo = {};
pipelineInfo.sType = VK_STRUCTURE_TYPE_COMPUTE_PIPELINE_CREATE_INFO;
pipelineInfo.stage.sType = VK_STRUCTURE_TYPE_PIPELINE_SHADER_STAGE_CREATE_INFO;
pipelineInfo.stage.stage = VK_SHADER_STAGE_COMPUTE_BIT;
pipelineInfo.stage.module = shaderModule;
pipelineInfo.stage.pName = "main";
pipelineInfo.layout = pipelineLayout;
vkCreateComputePipelines(device, VK_NULL_HANDLE, 1, &pipelineInfo, nullptr, &pipeline);

Step 6: Create buffers and allocate memory

VkBufferCreateInfo bufferInfo = {};
bufferInfo.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO;
bufferInfo.size = dataSize;
bufferInfo.usage = VK_BUFFER_USAGE_STORAGE_BUFFER_BIT | VK_BUFFER_USAGE_TRANSFER_DST_BIT; // for copying from host
bufferInfo.sharingMode = VK_SHARING_MODE_EXCLUSIVE;
vkCreateBuffer(device, &bufferInfo, nullptr, &bufferA);
// Similarly for B, C

// Get memory requirements, allocate memory, bind buffers
VkMemoryRequirements memReqs;
vkGetBufferMemoryRequirements(device, bufferA, &memReqs);
VkMemoryAllocateInfo allocInfo = {};
allocInfo.sType = VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO;
allocInfo.allocationSize = memReqs.size;
allocInfo.memoryTypeIndex = findMemoryType(memReqs.memoryTypeBits, VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT);
vkAllocateMemory(device, &allocInfo, nullptr, &memoryA);
vkBindBufferMemory(device, bufferA, memoryA, 0);

Step 7: Map memory and copy initial data

void* mapped;
vkMapMemory(device, memoryA, 0, dataSize, 0, &mapped);
memcpy(mapped, h_a, dataSize);
vkUnmapMemory(device, memoryA);
// Similarly for B

Step 8: Create descriptor pool and descriptor set

VkDescriptorPoolSize poolSize = {};
poolSize.type = VK_DESCRIPTOR_TYPE_STORAGE_BUFFER;
poolSize.descriptorCount = 3;
VkDescriptorPoolCreateInfo poolInfo = {};
poolInfo.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_POOL_CREATE_INFO;
poolInfo.maxSets = 1;
poolInfo.poolSizeCount = 1; poolInfo.pPoolSizes = &poolSize;
vkCreateDescriptorPool(device, &poolInfo, nullptr, &descriptorPool);

VkDescriptorSetAllocateInfo allocInfo = {};
allocInfo.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_SET_ALLOCATE_INFO;
allocInfo.descriptorPool = descriptorPool;
allocInfo.descriptorSetCount = 1; allocInfo.pSetLayouts = &descriptorSetLayout;
vkAllocateDescriptorSets(device, &allocInfo, &descriptorSet);

Step 9: Update descriptor set with buffer info

VkDescriptorBufferInfo bufferInfoA = { bufferA, 0, dataSize };
VkWriteDescriptorSet writeA = {};
writeA.sType = VK_STRUCTURE_TYPE_WRITE_DESCRIPTOR_SET;
writeA.dstSet = descriptorSet;
writeA.dstBinding = 0;
writeA.descriptorCount = 1;
writeA.descriptorType = VK_DESCRIPTOR_TYPE_STORAGE_BUFFER;
writeA.pBufferInfo = &bufferInfoA;
// Similarly for B, C (bindings 1,2)
vkUpdateDescriptorSets(device, 3, writes, 0, nullptr);

Step 10: Create command pool and command buffer

VkCommandPoolCreateInfo poolCmdInfo = {};
poolCmdInfo.sType = VK_STRUCTURE_TYPE_COMMAND_POOL_CREATE_INFO;
poolCmdInfo.queueFamilyIndex = computeQueueFamilyIndex;
vkCreateCommandPool(device, &poolCmdInfo, nullptr, &commandPool);

VkCommandBufferAllocateInfo cmdAllocInfo = {};
cmdAllocInfo.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_ALLOCATE_INFO;
cmdAllocInfo.commandPool = commandPool;
cmdAllocInfo.level = VK_COMMAND_BUFFER_LEVEL_PRIMARY;
cmdAllocInfo.commandBufferCount = 1;
vkAllocateCommandBuffers(device, &cmdAllocInfo, &commandBuffer);

Step 11: Record commands

VkCommandBufferBeginInfo beginInfo = {};
beginInfo.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO;
vkBeginCommandBuffer(commandBuffer, &beginInfo);

vkCmdBindPipeline(commandBuffer, VK_PIPELINE_BIND_POINT_COMPUTE, pipeline);
vkCmdBindDescriptorSets(commandBuffer, VK_PIPELINE_BIND_POINT_COMPUTE, pipelineLayout, 0, 1, &descriptorSet, 0, nullptr);

uint32_t groupCountX = (VEC_SIZE + 255) / 256;
vkCmdDispatch(commandBuffer, groupCountX, 1, 1);

vkEndCommandBuffer(commandBuffer);

Step 12: Submit and wait

VkSubmitInfo submitInfo = {};
submitInfo.sType = VK_STRUCTURE_TYPE_SUBMIT_INFO;
submitInfo.commandBufferCount = 1; submitInfo.pCommandBuffers = &commandBuffer;
vkQueueSubmit(computeQueue, 1, &submitInfo, VK_NULL_HANDLE);
vkQueueWaitIdle(computeQueue);

Step 13: Read back results (map memory of buffer C and copy).

That's the complete flow. It's verbose, but gives you full control.

2.6 Synchronization

Vulkan uses fences (GPU‑CPU sync) and semaphores (GPU‑GPU sync). In compute, you often just wait for the queue idle. For finer control, use vkWaitForFences after submitting with a fence.

Memory dependencies are handled by pipeline barriers. For example, if you write to a buffer then read it in a later dispatch, you need a barrier to ensure visibility.

2.7 Performance Optimization

Workgroup size

Choose a workgroup size that is a multiple of the subgroup size (typically 32 or 64) and that maximizes occupancy. Use VkPhysicalDeviceProperties::maxComputeWorkGroupSize.

Shared memory

Declare shared memory in GLSL with shared float tile[16][16]; and use barrier() for synchronization. Similar to Metal threadgroup memory.

Subgroup operations

Vulkan exposes subgroup operations via the VK_EXT_subgroup extension. These allow efficient reductions, scans, and shuffles within a subgroup (like warp intrinsics).

Avoiding bank conflicts – same principle as Metal.

2.8 Example: Tiled Matrix Multiplication in Vulkan

Shader (GLSL):

#version 450
layout(local_size_x = 16, local_size_y = 16) in;

layout(binding = 0) buffer InputA { float data[]; } A;
layout(binding = 1) buffer InputB { float data[]; } B;
layout(binding = 2) buffer OutputC { float data[]; } C;
layout(push_constant) uniform PushConsts { uint N; } pc;

shared float As[16][16];
shared float Bs[16][16];

void main() {
    uint row = gl_GlobalInvocationID.y;
    uint col = gl_GlobalInvocationID.x;
    uint localRow = gl_LocalInvocationID.y;
    uint localCol = gl_LocalInvocationID.x;

    float sum = 0.0;
    for (uint tile = 0; tile < pc.N / 16; ++tile) {
        As[localRow][localCol] = A.data[row * pc.N + (tile * 16 + localCol)];
        Bs[localRow][localCol] = B.data[(tile * 16 + localRow) * pc.N + col];
        barrier();

        for (uint k = 0; k < 16; ++k) {
            sum += As[localRow][k] * Bs[k][localCol];
        }
        barrier();
    }
    C.data[row * pc.N + col] = sum;
}

Host changes:

Use push constant to pass N.
Dispatch with grid size (N/16, N/16) and local size (16,16).
Ensure descriptor set binds buffers A, B, C.

Part 3: Comparative Analysis and Cross‑Platform Strategies

3.1 Metal vs Vulkan Compute: Similarities and Differences

Aspect	Metal	Vulkan Compute
API Style	Objective‑C / Swift (objects, references)	C (handles, structures)
Shader Language	MSL (C++‑based)	GLSL / HLSL compiled to SPIR‑V
Resource Binding	[[buffer(n)]] and [[texture(n)]]	Descriptor sets with bindings
Memory Model	Unified on Apple Silicon; discrete otherwise	Explicit memory management
Workgroup/Threadgroup	`threadgroup` memory, `threadgroup_barrier`	`shared` memory, `barrier()`
Error Handling	NSError, exceptions (Swift)	Return codes (VkResult)
Debugging/Profiling	Xcode GPU Debugger, Metal System Trace	RenderDoc, Vulkan Validation Layers, GPU vendors' tools

Both are explicit, low‑level, and provide fine‑grained control. Metal is slightly simpler to use, especially on Apple platforms, while Vulkan offers cross‑platform portability.

3.2 Writing Portable Compute Code

If you need to support both Metal and Vulkan, consider:

Writing compute kernels in a common language (e.g., using a library like OpenCL C or SYCL that compiles to both, but that's beyond this guide).
Maintaining two separate codebases with shared algorithms.
Using conditional compilation and abstraction layers (like MoltenVK to run Vulkan on Metal, though compute performance is good).

For compute kernels, the core logic is often identical; only the syntax for resource binding and built‑in variables differs. You can use macros to abstract differences.

Example kernel snippet (pseudo‑code):

#ifdef METAL
#define GLOBAL_ID [[thread_position_in_grid]]
#define SHARED threadgroup
#define BARRIER threadgroup_barrier(mem_flags::mem_threadgroup)
#else
#define GLOBAL_ID gl_GlobalInvocationID
#define SHARED shared
#define BARRIER barrier()
#endif

Part 4: Advanced Topics

4.1 Indirect Dispatch

Both APIs support indirect dispatch where the workgroup counts are read from a buffer, allowing GPU‑generated workloads.

Metal: [encoder dispatchThreadgroupsWithIndirectBuffer:indirectBuffer indirectBufferOffset:0 threadsPerThreadgroup:threadgroupSize];

Vulkan: vkCmdDispatchIndirect with a buffer containing the dispatch dimensions.

4.2 GPU Timestamps and Queries

Metal: Use MTLCounterSampleBuffer (on supported devices) to sample GPU timestamps.

Vulkan: Use query pools (VK_QUERY_TYPE_TIMESTAMP) to record timestamps in command buffers.

4.3 Interop with Graphics Pipelines

Both APIs allow mixing compute and graphics in the same command buffer. For example, you can use a compute shader to generate data for a vertex buffer, then render it.

4.4 Multi‑GPU (Vulkan) and Multi‑Device (Metal)

Vulkan supports multi‑GPU via logical devices with multiple physical devices, but it's complex. Metal has MTLDevice per GPU; you can create separate queues and manage workloads manually.

4.5 Real‑world Example: Sobel Edge Detection

Let's outline a Sobel kernel in both APIs.

Metal Sobel (texture version):

kernel void sobel(texture2d<float, access::read> inTex [[texture(0)]],
                  texture2d<float, access::write> outTex [[texture(1)]],
                  uint2 gid [[thread_position_in_grid]]) {
    if (gid.x < outTex.get_width() && gid.y < outTex.get_height()) {
        float sumX = 0, sumY = 0;
        for (int dy = -1; dy <= 1; dy++) {
            for (int dx = -1; dx <= 1; dx++) {
                uint2 coord = uint2(gid.x + dx, gid.y + dy);
                float val = inTex.read(coord).r;
                sumX += val * sobelX[dy+1][dx+1];
                sumY += val * sobelY[dy+1][dx+1];
            }
        }
        float mag = min(1.0f, hypot(sumX, sumY));
        outTex.write(float4(mag, mag, mag, 1), gid);
    }
}

Vulkan Sobel (buffer version, assuming image stored as float array):

#version 450
layout(local_size_x = 16, local_size_y = 16) in;
layout(binding = 0) buffer Input { float pixels[]; } inBuf;
layout(binding = 1) buffer Output { float pixels[]; } outBuf;
layout(push_constant) uniform Dims { uint width; uint height; } dims;

void main() {
    uint x = gl_GlobalInvocationID.x;
    uint y = gl_GlobalInvocationID.y;
    if (x >= dims.width || y >= dims.height) return;
    float sumX = 0, sumY = 0;
    for (int dy = -1; dy <= 1; dy++) {
        for (int dx = -1; dx <= 1; dx++) {
            int nx = int(x) + dx, ny = int(y) + dy;
            if (nx < 0 || nx >= dims.width || ny < 0 || ny >= dims.height) continue;
            float val = inBuf.pixels[ny * dims.width + nx];
            sumX += val * sobelX[dy+1][dx+1];
            sumY += val * sobelY[dy+1][dx+1];
        }
    }
    float mag = min(1.0, length(vec2(sumX, sumY)));
    outBuf.pixels[y * dims.width + x] = mag;
}

Conclusion

Congratulations! You've journeyed through the depths of Metal compute on Apple Silicon and Vulkan compute. You now know how to set up the environment, write kernels, manage memory, optimize performance, and implement complex patterns like matrix multiplication and edge detection.

Next Steps

Metal: Explore Metal Performance Shaders (MPS) for highly optimized primitives (convolution, matrix multiplication, etc.). Look into Metal Ray Tracing for advanced graphics.
Vulkan: Dive into the Vulkan specification, experiment with subgroup operations, and try the Vulkan Ray Tracing extensions. Explore Vulkan Video for encode/decode.
Cross‑platform: Consider using SYCL or OpenCL for broader portability, but now you have the low‑level knowledge to understand what's happening under the hood.

The world of GPU compute is vast and exciting. Keep experimenting, profiling, and optimizing. And remember – the best way to learn is to write code. So fire up Xcode or your Vulkan SDK and start computing!

If you found this guide helpful, share it with your fellow developers. Have questions or want to share your own experiences? Leave a comment below. Happy coding!

Advanced GPU Optimization: OpenCL & SYCL from zero to hero - Part 2

Javad — Sat, 14 Feb 2026 16:31:54 +0000

Welcome back, heterogeneous computing enthusiasts! In our previous guides, we covered the fundamentals of OpenCL and SYCL, from setting up your environment to writing basic kernels and optimizing them with local memory. Now it's time to go deeper. In this advanced tutorial, we'll explore:

Key features of the OpenCL specification (2.x/3.0) that enable sophisticated programming.
What SYCL 2020 brings to the table, including unified shared memory and group algorithms.
Implementation of complex parallel patterns: reductions, scans, and stencils.
How to target CPUs, GPUs, and FPGAs with SYCL and compare performance.
A tour of oneAPI's domain‑specific libraries (oneMKL, oneDNN, oneDPL, etc.) that supercharge your development.

Whether you're looking to squeeze every drop of performance from your hardware or write portable code that scales across architectures, this guide has you covered.

Part 1: Deep Dive into the OpenCL Specification

The OpenCL specification has evolved significantly. While OpenCL 1.2 is the most widely supported baseline, modern devices support newer features that can greatly simplify and accelerate development. Let's look at the highlights.

1.1 OpenCL 2.x Features

Shared Virtual Memory (SVM)

OpenCL 2.0 introduced SVM, allowing the host and devices to share pointers. This eliminates explicit buffer mappings and can simplify programming. Three types exist:

Coarse-grained SVM: Sharing at buffer granularity; synchronization needed.
Fine-grained SVM: Sharing at any memory location; optional atomics.
Fine-grained buffer SVM: Like fine-grained but limited to a buffer region.

With SVM, you can pass complex data structures (like linked lists) to kernels without deep copies.

Device-side Enqueue

Kernels can enqueue other kernels, creating nested parallelism. This is great for recursive algorithms or dynamic workloads. The parent kernel creates a child kernel and enqueues it to a device queue.

Pipes

Pipes are FIFO data structures connecting kernels. They facilitate streaming patterns without global memory round trips. One kernel writes to a pipe, another reads from it, with synchronization handled by the hardware.

Work-group Collective Functions

OpenCL 2.0 added built‑in work‑group functions like work_group_reduce_add, work_group_scan_exclusive_add, etc., enabling efficient intra‑group reductions and scans.

C11 Atomics

Kernels can use atomic operations conforming to the C11 memory model, essential for lock‑free algorithms.

1.2 OpenCL 3.0: The New Baseline

OpenCL 3.0, released in 2020, takes a different approach: it makes OpenCL 1.2 the mandatory baseline, with all 2.x features optional. This allows vendors to implement only what they support while maintaining compatibility. Key additions:

A new query mechanism to check for optional features.
Support for newer C++ features in kernels (like auto, lambda expressions) via the cl_khr_c++11 extension.
Better integration with Vulkan and other APIs.

1.3 Using Advanced Features in Practice

To use these features, you must check for extensions and enable them at program creation. For example, to use SVM:

cl_device_svm_capabilities caps;
clGetDeviceInfo(device, CL_DEVICE_SVM_CAPABILITIES, sizeof(caps), &caps, NULL);
if (caps & CL_DEVICE_SVM_COARSE_GRAIN_BUFFER) {
    // Allocate coarse‑grain SVM buffer
    clSVMAlloc(context, CL_MEM_READ_WRITE, size, 0);
}

Device‑side enqueue requires building the program with the -cl-std=CL2.0 flag and using the cl_khr_device_enqueue extension.

While these features are powerful, they are not universally supported across all devices. Always fall back to portable code when necessary.

Part 2: Deep Dive into the SYCL 2020 Specification

SYCL 2020 is a major update that aligns with modern C++ and adds many features inspired by OpenCL 2.x and beyond. Let's explore the most impactful additions.

2.1 Unified Shared Memory (USM)

SYCL 2020 standardizes USM, providing pointer‑based memory management. Three allocation types:

malloc_device: Device‑accessible memory (not directly on host).
malloc_host: Host‑accessible memory (may be cached on device).
malloc_shared: Automatically migrated memory, accessible from both sides.

Example:

queue q;
int *data = malloc_shared<int>(N, q);
q.parallel_for(range<1>(N), [=](id<1> i) { data[i] = i; }).wait();
// data is updated on host after synchronization
free(data, q);

USM simplifies porting CUDA/HIP code and reduces verbosity compared to buffers.

2.2 Reductions

SYCL 2020 introduces built‑in reduction support. You can use reduction objects with parallel_for to perform reductions efficiently.

queue q;
std::vector<float> vec(N, 1.0f);
buffer<float> buf(vec);
float init = 0.0f;
{
    buffer<float> sum_buf(&init, 1);
    q.submit([&](handler& h) {
        auto in = buf.get_access<access::mode::read>(h);
        auto out = sum_buf.get_access<access::mode::write>(h);
        h.parallel_for(range<1>(N), reduction(out, 0.0f, std::plus<>()), [=](id<1> i, auto& sum) {
            sum += in[i];
        });
    });
}
// sum_buf now contains the total

The implementation chooses the most efficient strategy (tree reduction, atomics, etc.) based on the device.

2.3 Group Algorithms and Sub‑Groups

SYCL 2020 provides collective algorithms for work‑groups, similar to OpenCL 2.0:

group_barrier()
group_broadcast()
group_reduce(), group_scan() (exclusive/inclusive)
sub_group class for warp/wavefront operations.

Example of group reduction:

h.parallel_for(nd_range<1>(N, 256), [=](nd_item<1> it) {
    auto g = it.get_group();
    int local_sum = ...;
    int group_sum = reduce_over_group(g, local_sum, std::plus<>());
    if (it.get_local_id(0) == 0)
        atomic_ref<int, memory_order::relaxed, memory_scope::device>(global_sum) += group_sum;
});

Sub‑groups allow portable SIMD programming. You can query sub‑group size and use shuffles.

2.4 In‑Order Queues and Properties

SYCL 2020 adds queue properties to specify in‑order execution, which can simplify dependencies:

queue q{ property::queue::in_order() };

2.5 Interoperability with OpenCL

SYCL 2020 improves interop: you can create a SYCL platform, device, context, or queue from native OpenCL handles, and vice versa. This allows incremental migration or mixing low‑level control with high‑level productivity.

Part 3: Complex Parallel Patterns

Now that we've covered the specifications, let's implement three fundamental parallel patterns: reduction, scan, and stencil. We'll show both OpenCL and SYCL versions, with optimizations.

3.1 Reduction (Sum of an Array)

Reduction combines elements using an associative operator (e.g., sum). A common approach is hierarchical: each work‑group reduces its chunk in local memory, then one thread per group adds to global atomic.

OpenCL kernel:

__kernel void reduce(__global const float *in, __global float *out, __local float *localMem, int n) {
    int gid = get_global_id(0);
    int lid = get_local_id(0);
    int groupSize = get_local_size(0);
    int groupId = get_group_id(0);

    // Load data into local memory
    float sum = (gid < n) ? in[gid] : 0.0f;
    localMem[lid] = sum;
    barrier(CLK_LOCAL_MEM_FENCE);

    // Reduce within group
    for (int stride = groupSize / 2; stride > 0; stride >>= 1) {
        if (lid < stride) {
            localMem[lid] += localMem[lid + stride];
        }
        barrier(CLK_LOCAL_MEM_FENCE);
    }

    // Write partial sum
    if (lid == 0) {
        out[groupId] = localMem[0];
    }
}

After this kernel, you have partial sums. Run a second kernel with one work‑group to combine them, or copy to host and finalize.

SYCL with reductions:
SYCL's built‑in reduction handles this automatically, but for educational purposes, here's a manual version:

q.submit([&](handler& h) {
    auto in = buf_in.get_access<access::mode::read>(h);
    auto out = buf_out.get_access<access::mode::write>(h);
    accessor<float, 1, access::mode::read_write, access::target::local> localMem(range<1>(256), h);

    h.parallel_for(nd_range<1>(n, 256), [=](nd_item<1> it) {
        int gid = it.get_global_id(0);
        int lid = it.get_local_id(0);
        float sum = (gid < n) ? in[gid] : 0.0f;
        localMem[lid] = sum;
        it.barrier();

        for (int stride = 128; stride > 0; stride >>= 1) {
            if (lid < stride) {
                localMem[lid] += localMem[lid + stride];
            }
            it.barrier();
        }

        if (lid == 0) {
            out[it.get_group(0)] = localMem[0];
        }
    });
});

But the SYCL 2020 way is simpler:

q.submit([&](handler& h) {
    auto in = buf_in.get_access<access::mode::read>(h);
    h.parallel_for(range<1>(n), reduction(sum_buf, h, 0.0f, std::plus<>()), [=](id<1> i, auto& sum) {
        sum += in[i];
    });
});

3.2 Scan (Prefix Sum)

Scan computes cumulative sums. An efficient work‑group scan uses a double‑buffered local memory approach.

OpenCL kernel for inclusive scan (one block):

__kernel void scan(__global const float *in, __global float *out, __local float *temp, int n) {
    int gid = get_global_id(0);
    int lid = get_local_id(0);
    int groupSize = get_local_size(0);
    int groupId = get_group_id(0);

    // Load data
    temp[2 * lid] = (gid < n) ? in[gid] : 0;
    temp[2 * lid + 1] = (gid + groupSize < n) ? in[gid + groupSize] : 0;
    barrier(CLK_LOCAL_MEM_FENCE);

    // Upsweep phase (parallel reduction)
    for (int stride = 1; stride < 2 * groupSize; stride <<= 1) {
        int index = (lid + 1) * stride * 2 - 1;
        if (index < 2 * groupSize && lid < groupSize) {
            temp[index] += temp[index - stride];
        }
        barrier(CLK_LOCAL_MEM_FENCE);
    }

    // Downsweep phase
    for (int stride = groupSize; stride > 0; stride >>= 1) {
        int index = (lid + 1) * stride * 2 - 1;
        if (index + stride < 2 * groupSize && lid < groupSize) {
            temp[index + stride] += temp[index];
        }
        barrier(CLK_LOCAL_MEM_FENCE);
    }

    // Write results
    if (gid < n) {
        out[gid] = temp[2 * lid];
        if (gid + groupSize < n)
            out[gid + groupSize] = temp[2 * lid + 1];
    }
}

After this, you need a second pass to add the total of previous groups. For large arrays, you'd combine with a top‑level scan of group sums.

SYCL using group_scan:
SYCL 2020 provides group_scan_inclusive and group_scan_exclusive:

q.submit([&](handler& h) {
    auto in = buf_in.get_access<access::mode::read>(h);
    auto out = buf_out.get_access<access::mode::write>(h);
    accessor<float, 1, access::mode::read_write, access::target::local> localMem(range<1>(256), h);

    h.parallel_for(nd_range<1>(n, 256), [=](nd_item<1> it) {
        int gid = it.get_global_id(0);
        int lid = it.get_local_id(0);
        float val = (gid < n) ? in[gid] : 0.0f;
        auto grp = it.get_group();
        float scan_val = group_scan_inclusive(grp, val, std::plus<>());
        if (gid < n)
            out[gid] = scan_val;
    });
});

This performs an inclusive scan within each work‑group. To get a global scan, you'd again need to handle the group offsets.

3.3 Stencil (2D Heat Conduction)

Stencils update each element based on its neighbors. Optimizations: use local memory to cache a tile plus halo.

OpenCL kernel for 2D heat stencil:

__kernel void heat(__global const float *in, __global float *out, int width, int height) {
    int x = get_global_id(0);
    int y = get_global_id(1);
    int localX = get_local_id(0);
    int localY = get_local_id(1);
    int localSizeX = get_local_size(0);
    int localSizeY = get_local_size(1);

    // Local memory with halo of 1
    __local float tile[LOCAL_SIZE_Y + 2][LOCAL_SIZE_X + 2];

    // Load interior
    if (x < width && y < height)
        tile[localY + 1][localX + 1] = in[y * width + x];
    else
        tile[localY + 1][localX + 1] = 0.0f;

    // Load halo regions (simplified: assumes boundary conditions)
    if (localY == 0 && y > 0)
        tile[0][localX + 1] = in[(y-1) * width + x];
    if (localY == localSizeY - 1 && y < height - 1)
        tile[localSizeY + 1][localX + 1] = in[(y+1) * width + x];
    // similarly for left/right
    barrier(CLK_LOCAL_MEM_FENCE);

    if (x < width && y < height) {
        float center = tile[localY + 1][localX + 1];
        float left   = tile[localY + 1][localX];
        float right  = tile[localY + 1][localX + 2];
        float up     = tile[localY][localX + 1];
        float down   = tile[localY + 2][localX + 1];
        out[y * width + x] = center + 0.125f * (left + right + up + down - 4.0f * center);
    }
}

SYCL version (with local accessor and nd_range):

q.submit([&](handler& h) {
    auto in = buf_in.get_access<access::mode::read>(h);
    auto out = buf_out.get_access<access::mode::write>(h);
    accessor<float, 2, access::mode::read_write, access::target::local> tile(range<2>(LOCAL_Y+2, LOCAL_X+2), h);

    h.parallel_for(nd_range<2>(range<2>(height, width), range<2>(LOCAL_Y, LOCAL_X)), [=](nd_item<2> it) {
        int x = it.get_global_id(1);
        int y = it.get_global_id(0);
        int lx = it.get_local_id(1);
        int ly = it.get_local_id(0);

        // Load interior
        if (x < width && y < height)
            tile[ly+1][lx+1] = in[y][x];

        // Load halos (simplified)
        // ... similar to OpenCL
        it.barrier();

        if (x < width && y < height) {
            float center = tile[ly+1][lx+1];
            float left   = tile[ly+1][lx];
            float right  = tile[ly+1][lx+2];
            float up     = tile[ly][lx+1];
            float down   = tile[ly+2][lx+1];
            out[y][x] = center + 0.125f * (left + right + up + down - 4.0f * center);
        }
    });
});

These patterns are building blocks for many scientific and engineering applications.

Part 4: SYCL on Different Hardware and Performance Comparison

One of SYCL's greatest promises is "write once, run anywhere." But how does performance compare across CPUs, GPUs, and FPGAs? Let's explore the practical considerations.

4.1 Targeting Devices with SYCL

Selecting a device is straightforward:

// Default selector (picks a device, often GPU if available)
queue q;

// Specific selectors
cpu_selector cpu;
gpu_selector gpu;
accelerator_selector fpga; // for FPGA

queue q_cpu(cpu);
queue q_gpu(gpu);
queue q_fpga(fpga);

You can also write custom selectors based on device info (e.g., preferring a GPU with larger local memory).

4.2 Performance Considerations per Device

CPU:

Optimize for vectorization: use wide SIMD (work‑group size = vector width).
Minimize data transfers; CPU shares memory with host, so USM can be efficient.
Avoid excessive local memory; CPU caches are good.

GPU:

Maximize occupancy: choose work‑group size that hides latency (e.g., 128‑256).
Use local memory for data reuse.
Ensure coalesced global memory accesses.

FPGA:

Kernels are synthesized into hardware pipelines.
Loop unrolling, pipelining, and on‑chip memory are key.
SYCL for FPGA supports [[intel::max_work_group_size(1)]] to create single‑work‑item kernels that are highly pipelined.
Use #pragma unroll and [[intel::initiation_interval(1)]] directives.

4.3 Benchmarking and Comparison

To compare performance across devices, you need a common benchmark. Let's take matrix multiplication (tiled) and run it on different hardware.

Methodology:

Compile with appropriate flags: for CPU, use -O3 -march=native; for GPU, -O3 -fsycl; for FPGA, you need the Intel FPGA SDK and use -Xshardware for actual hardware runs.
Measure kernel execution time using SYCL events:

q.submit([&](handler& h) {
    // ...
}).wait();
auto start = std::chrono::steady_clock::now();
q.submit([&](handler& h) {
    // kernel
}).wait();
auto end = std::chrono::steady_clock::now();

Better: use SYCL profiling info via property::queue::enable_profiling().

Expected Observations:

CPU: Good for small matrices, latency may be higher due to OS scheduling.
GPU: Excels at large, regular computations.
FPGA: Often slower for compute‑bound tasks but can be very power‑efficient and excel at pipeline‑friendly streaming.

Case Study: Matrix Multiplication
We can run the tiled SYCL kernel from Part 2.7 on a CPU, GPU, and FPGA (if available). The performance will vary dramatically. For example, on an Intel Core i9, the CPU might achieve ~50 GFLOPs; on an Intel UHD Graphics, maybe ~200 GFLOPs; on an Arria 10 FPGA, perhaps ~50 GFLOPs but with lower power. However, the FPGA could achieve lower latency for small batches and better predictability.

4.4 Portable Optimization Tips

Use #ifdef SYCL_DEVICE_CPU etc. to specialize for each architecture.
Tune work‑group size dynamically using device.get_info<info::device::max_work_group_size>().
For FPGA, design kernels to be highly parallel with pipeline parallelism rather than data parallelism.

Part 5: oneAPI Domain-Specific Libraries

oneAPI includes a rich set of libraries that provide optimized building blocks for common tasks. Using them can drastically accelerate development while ensuring high performance across devices.

5.1 oneMKL (Math Kernel Library)

oneMKL provides BLAS, LAPACK, FFT, RNG, and more. It has SYCL interfaces that accept SYCL buffers or USM pointers.

Example: Matrix multiplication using oneMKL:

#include <oneapi/mkl/blas.hpp>
queue q;
float alpha = 1.0f, beta = 0.0f;
oneapi::mkl::blas::gemm(q, oneapi::mkl::transpose::nontrans, oneapi::mkl::transpose::nontrans,
                        N, N, N, alpha, d_A, N, d_B, N, beta, d_C, N);
q.wait();

This will call an optimized GEMM for the target device (CPU, GPU, or FPGA). No need to write your own kernel!

FFT example:

#include <oneapi/mkl/dft.hpp>
using namespace oneapi::mkl::dft;
descriptor<precision::single, domain::real> transform(N);
transform.commit(q);
// in and out buffers...
transform.compute_forward(in, out);

5.2 oneDNN (Deep Neural Networks)

oneDNN (formerly DNNL) provides primitives for deep learning: convolution, pooling, normalization, etc. It has SYCL and OpenCL backends.

Example: Convolution:

#include <oneapi/dnnl/dnnl.hpp>
using namespace dnnl;

engine eng(engine::kind::gpu, 0);
stream s(eng);
memory::dims conv_src_tz = {N, C, H, W};
memory::dims conv_weights_tz = {K, C, R, S};
memory::dims conv_dst_tz = {N, K, H', W'};
// Create memory objects
auto conv_src_mem = memory({{conv_src_tz}, memory::data_type::f32, memory::format_tag::any}, eng);
// Initialize convolution descriptor and primitive
convolution_forward::desc conv_desc(prop_kind::forward, algorithm::convolution_direct,
                                    conv_src_md, conv_weights_md, conv_dst_md,
                                    strides, padding_l, padding_r);
convolution_forward::primitive_desc conv_pd(conv_desc, eng);
auto conv = convolution_forward(conv_pd);
conv.execute(s, {{DNNL_ARG_SRC, conv_src_mem}, {DNNL_ARG_WEIGHTS, conv_weights_mem}, {DNNL_ARG_DST, conv_dst_mem}});
s.wait();

oneDNN automatically selects the best implementation for the hardware.

5.3 oneDPL (DPC++ Library)

oneDPL provides parallel algorithms similar to C++17's Parallel STL, implemented with SYCL.

Example: parallel sort:

#include <oneapi/dpl/algorithm>
#include <oneapi/dpl/execution>

std::vector<float> data(N);
// fill data...
auto policy = oneapi::dpl::execution::make_device_policy(q);
std::sort(policy, data.begin(), data.end());

Other algorithms: reduce, transform, scan, etc.

5.4 oneTBB (Threading Building Blocks)

While not exclusively for devices, oneTBB integrates with oneAPI to provide task parallelism on the host, which can coordinate with device work.

5.5 Other oneAPI Libraries

oneVPL – Video Processing Library for encode/decode.
oneDAL – Data Analytics Library.
oneCCL – Collective Communications Library for distributed computing.

5.6 Combining Libraries in a Project

A typical SYCL + oneAPI project might use:

oneDPL for data preparation on CPU.
oneMKL for FFT on GPU.
oneDNN for inference on FPGA.
oneTBB to manage host-side tasks.

All with the same SYCL queue and memory model.

Conclusion

We've journeyed through the advanced features of OpenCL and SYCL, implemented essential parallel patterns, explored performance across diverse hardware, and surveyed the powerful oneAPI libraries. Armed with this knowledge, you can now tackle complex computational problems with confidence and write code that runs efficiently on CPUs, GPUs, and FPGAs alike.

Remember, the key to mastering heterogeneous computing is practice. Experiment with these patterns on your own hardware, profile them, and see where optimizations take you. And when you need a productivity boost, don't reinvent the wheel – leverage the oneAPI libraries.

Further Resources:

Thank you for reading! If you have questions or want to share your experiences with OpenCL, SYCL, or oneAPI, leave a comment below. Happy coding!

Advanced GPU Optimization: OpenCL & SYCL from zero to hero

Javad — Sat, 14 Feb 2026 16:29:50 +0000

Welcome to the world of heterogeneous programming! In this comprehensive guide, we’ll explore two powerful frameworks that let you harness the computational power of CPUs, GPUs, FPGAs, and other accelerators: OpenCL and SYCL (the heart of Intel’s oneAPI initiative). Whether you’re a student, researcher, or industry developer, by the end of this tutorial you’ll be able to write portable, efficient code that runs on a variety of hardware.

We’ll start from the absolute basics (no prior experience required) and progress to advanced topics like local memory optimization, profiling, and multi-device programming. Along the way, you’ll see how OpenCL provides a low‑level, C‑based interface for fine‑grained control, while SYCL offers a modern C++ single‑source approach that builds on OpenCL’s concepts.

Let’s dive in!

Introduction: Why OpenCL and SYCL?

OpenCL (Open Computing Language) is the industry standard for heterogeneous computing. It allows you to write kernels (programs) that execute on any OpenCL‑compliant device—CPUs, GPUs, DSPs, FPGAs—using a C‑based language. It gives you explicit control over memory and execution, but at the cost of some verbosity.

SYCL (pronounced “sickle”) is a higher‑level C++ abstraction built on top of OpenCL. It enables single‑source programming: you write host and device code in the same C++ file, using modern C++ features. SYCL is the foundation of Intel’s oneAPI initiative, which aims to provide a unified programming model across CPUs, GPUs, and FPGAs. SYCL code can be compiled for different backends (OpenCL, Level Zero, CUDA) without modification.

Both are essential tools in the heterogeneous computing landscape. Learning them gives you the flexibility to target a wide range of hardware while keeping your code portable.

Part 1: OpenCL Fundamentals

1.1 Setting Up OpenCL

Before we write code, we need an OpenCL development environment. OpenCL consists of a host API (to manage devices, memory, and execution) and a kernel language (based on C99 with extensions).

Installation:

On Linux: Install the OpenCL headers and a runtime (e.g., from your GPU vendor). For Intel CPUs/GPUs, install the Intel Compute Runtime. For NVIDIA, install the CUDA toolkit which includes OpenCL. For AMD, install ROCm or the AMD APP SDK.
On Windows: Download and install the appropriate SDK from your hardware vendor (Intel, NVIDIA, AMD). You can also use the open‑source PoCL (Portable Computing Language) for CPU development.

Basic includes and linking:

#include <CL/cl.h>

Link against the OpenCL library (e.g., -lOpenCL on Linux, OpenCL.lib on Windows).

1.2 The OpenCL Execution Model

OpenCL organizes work into an NDRange (N‑dimensional range). The key concepts:

Work‑item: The smallest unit of execution. Each work‑item executes the same kernel but works on different data.
Work‑group: A collection of work‑items that execute together on a compute unit. Work‑items in the same group can synchronize via barriers and share memory through local memory.
NDRange: The entire grid of work‑items, partitioned into work‑groups. You specify the global size (total work‑items) and local size (work‑group size).

When you launch a kernel, you provide the number of dimensions (1, 2, or 3), the global size in each dimension, and optionally the local size. The runtime then schedules work‑groups onto compute units.

1.3 OpenCL Memory Model

OpenCL defines several memory regions:

Global memory: Accessible by all work‑items; large but high latency.
Constant memory: Read‑only global memory, cached.
Local memory: Shared within a work‑group; fast, programmer‑managed.
Private memory: Per work‑item, typically registers.

Host code manages device memory via buffer objects. Data is explicitly moved between host and device.

1.4 Your First OpenCL Program: Vector Addition

Let’s write a simple vector addition kernel and the host code to run it.

Kernel (stored in a string or separate .cl file):

__kernel void vecAdd(__global const float *a,
                     __global const float *b,
                     __global float *c) {
    int i = get_global_id(0);
    c[i] = a[i] + b[i];
}

Host code (C):

#include <stdio.h>
#include <stdlib.h>
#include <CL/cl.h>

#define VEC_SIZE 1048576  // 1M elements

int main() {
    // Step 1: Get platform and device
    cl_platform_id platform;
    cl_device_id device;
    clGetPlatformIDs(1, &platform, NULL);
    clGetDeviceIDs(platform, CL_DEVICE_TYPE_GPU, 1, &device, NULL);

    // Step 2: Create context and command queue
    cl_context context = clCreateContext(NULL, 1, &device, NULL, NULL, NULL);
    cl_command_queue queue = clCreateCommandQueue(context, device, 0, NULL);

    // Step 3: Allocate host memory and initialize data
    size_t bytes = VEC_SIZE * sizeof(float);
    float *h_a = (float*)malloc(bytes);
    float *h_b = (float*)malloc(bytes);
    float *h_c = (float*)malloc(bytes);
    for (int i = 0; i < VEC_SIZE; i++) {
        h_a[i] = i * 1.0f;
        h_b[i] = (i + 1) * 1.0f;
    }

    // Step 4: Create device buffers
    cl_mem d_a = clCreateBuffer(context, CL_MEM_READ_ONLY | CL_MEM_COPY_HOST_PTR, bytes, h_a, NULL);
    cl_mem d_b = clCreateBuffer(context, CL_MEM_READ_ONLY | CL_MEM_COPY_HOST_PTR, bytes, h_b, NULL);
    cl_mem d_c = clCreateBuffer(context, CL_MEM_WRITE_ONLY, bytes, NULL, NULL);

    // Step 5: Build the program
    const char *kernelSource = "__kernel void vecAdd(__global const float *a, __global const float *b, __global float *c) { int i = get_global_id(0); c[i] = a[i] + b[i]; }";
    cl_program program = clCreateProgramWithSource(context, 1, &kernelSource, NULL, NULL);
    clBuildProgram(program, 1, &device, NULL, NULL, NULL);
    cl_kernel kernel = clCreateKernel(program, "vecAdd", NULL);

    // Step 6: Set kernel arguments
    clSetKernelArg(kernel, 0, sizeof(cl_mem), &d_a);
    clSetKernelArg(kernel, 1, sizeof(cl_mem), &d_b);
    clSetKernelArg(kernel, 2, sizeof(cl_mem), &d_c);

    // Step 7: Execute kernel
    size_t global_size = VEC_SIZE;
    size_t local_size = 256;  // must divide global_size
    clEnqueueNDRangeKernel(queue, kernel, 1, NULL, &global_size, &local_size, 0, NULL, NULL);

    // Step 8: Read result back
    clEnqueueReadBuffer(queue, d_c, CL_TRUE, 0, bytes, h_c, 0, NULL, NULL);

    // Step 9: Verify (simplified)
    for (int i = 0; i < 10; i++)
        printf("%f + %f = %f\n", h_a[i], h_b[i], h_c[i]);

    // Step 10: Clean up
    clReleaseMemObject(d_a); clReleaseMemObject(d_b); clReleaseMemObject(d_c);
    clReleaseKernel(kernel); clReleaseProgram(program);
    clReleaseCommandQueue(queue); clReleaseContext(context);
    free(h_a); free(h_b); free(h_c);

    return 0;
}

Compile and run (on Linux):

gcc -o vecAdd vecAdd.c -lOpenCL
./vecAdd

Explanation:

We query a platform and device (here, we take the first GPU).
Create a context and command queue (the queue is where we enqueue commands).
Allocate host buffers and device buffers. clCreateBuffer with CL_MEM_COPY_HOST_PTR copies data immediately.
Build the program from source string; in real applications, you’d load from a file and check build errors.
Set kernel arguments, enqueue the kernel with global and local sizes.
Read the result back to host with clEnqueueReadBuffer.
Clean up OpenCL objects.

This is the basic pattern for any OpenCL program.

1.5 Building and Running OpenCL Programs

Always check error codes. Each OpenCL function returns a cl_int error code. Use macros like:

#define CL_CHECK(err) \
    if (err != CL_SUCCESS) { \
        fprintf(stderr, "OpenCL error %d at %s:%d\n", err, __FILE__, __LINE__); \
        exit(EXIT_FAILURE); \
    }

// Usage:
cl_int err;
cl_context context = clCreateContext(NULL, 1, &device, NULL, NULL, &err);
CL_CHECK(err);

When building a program, you should check build logs:

clBuildProgram(program, 1, &device, NULL, NULL, NULL);
if (err != CL_SUCCESS) {
    size_t log_size;
    clGetProgramBuildInfo(program, device, CL_PROGRAM_BUILD_LOG, 0, NULL, &log_size);
    char *log = (char*)malloc(log_size);
    clGetProgramBuildInfo(program, device, CL_PROGRAM_BUILD_LOG, log_size, log, NULL);
    fprintf(stderr, "Build log:\n%s\n", log);
    free(log);
    exit(EXIT_FAILURE);
}

1.6 Profiling OpenCL with Events

OpenCL events can measure execution time. Enable profiling on the command queue:

cl_command_queue queue = clCreateCommandQueue(context, device, CL_QUEUE_PROFILING_ENABLE, &err);

Then attach an event to kernel launch and read timestamps:

cl_event event;
clEnqueueNDRangeKernel(queue, kernel, 1, NULL, &global_size, &local_size, 0, NULL, &event);
clWaitForEvents(1, &event);

cl_ulong start, end;
clGetEventProfilingInfo(event, CL_PROFILING_COMMAND_START, sizeof(cl_ulong), &start, NULL);
clGetEventProfilingInfo(event, CL_PROFILING_COMMAND_END, sizeof(cl_ulong), &end, NULL);
double time_ms = (end - start) / 1.0e6;
printf("Kernel execution time: %f ms\n", time_ms);

clReleaseEvent(event);

1.7 Example: Matrix Multiplication (Naive and Tiled)

Let's implement matrix multiplication to demonstrate local memory usage.

Naive kernel (each work-item computes one element of C):

__kernel void matMulNaive(__global const float *A,
                          __global const float *B,
                          __global float *C,
                          int N) {
    int row = get_global_id(1);
    int col = get_global_id(0);
    if (row < N && col < N) {
        float sum = 0.0f;
        for (int k = 0; k < N; k++) {
            sum += A[row * N + k] * B[k * N + col];
        }
        C[row * N + col] = sum;
    }
}

Host side (simplified):

size_t global_size[2] = {N, N};
clEnqueueNDRangeKernel(queue, kernel, 2, NULL, global_size, NULL, 0, NULL, NULL);

Tiled kernel using local memory:

#define TILE_SIZE 16

__kernel void matMulTiled(__global const float *A,
                          __global const float *B,
                          __global float *C,
                          int N) {
    __local float As[TILE_SIZE][TILE_SIZE];
    __local float Bs[TILE_SIZE][TILE_SIZE];

    int row = get_global_id(1);
    int col = get_global_id(0);
    int localRow = get_local_id(1);
    int localCol = get_local_id(0);

    float sum = 0.0f;
    for (int tile = 0; tile < N / TILE_SIZE; tile++) {
        // Load tile of A and B into local memory
        As[localRow][localCol] = A[row * N + (tile * TILE_SIZE + localCol)];
        Bs[localRow][localCol] = B[(tile * TILE_SIZE + localRow) * N + col];
        barrier(CLK_LOCAL_MEM_FENCE);

        // Compute partial sum
        for (int k = 0; k < TILE_SIZE; k++) {
            sum += As[localRow][k] * Bs[k][localCol];
        }
        barrier(CLK_LOCAL_MEM_FENCE);
    }

    if (row < N && col < N) {
        C[row * N + col] = sum;
    }
}

Launch configuration:

size_t local_size[2] = {TILE_SIZE, TILE_SIZE};
size_t global_size[2] = {N, N};
clEnqueueNDRangeKernel(queue, kernel, 2, NULL, global_size, local_size, 0, NULL, NULL);

This tiled version reduces global memory traffic by reusing data loaded into local memory. It's a key optimization pattern.

1.8 Advanced OpenCL Topics

Multiple devices: You can create a context with multiple devices and use separate command queues.
Image objects: OpenCL supports 2D/3D images with samplers for graphics‑oriented workloads.
SVM (Shared Virtual Memory): Allows sharing pointers between host and device (OpenCL 2.0).
Device-side enqueue: Kernels can enqueue other kernels (OpenCL 2.0).

Part 2: SYCL (oneAPI) Fundamentals

2.1 What is SYCL?

SYCL is a C++ abstraction layer that sits on top of OpenCL (and other backends). It allows you to write host and device code in the same file using standard C++17 (or later). Key features:

Single-source: Kernel code is written as C++ functors or lambdas, compiled by a SYCL compiler.
High-level abstractions: Buffers and accessors manage memory dependencies; command groups encapsulate kernel launches.
Portability: Same code can target CPUs, GPUs, FPGAs, etc., via different SYCL implementations (Intel oneAPI DPC++, ComputeCpp, hipSYCL).
Interoperability: Can mix with OpenCL code if needed.

Intel’s oneAPI is a complete suite of tools and libraries built around SYCL (DPC++ – Data Parallel C++). We'll focus on DPC++ as the most widely adopted SYCL implementation.

2.2 Setting Up oneAPI / SYCL

Option 1: Intel oneAPI base toolkit (free):

Download from Intel oneAPI.
Install and source the environment: source /opt/intel/oneapi/setvars.sh.

Option 2: Open-source DPC++:

Follow instructions on intel/llvm to build the DPC++ compiler.

Option 3: hipSYCL (supports NVIDIA and AMD backends):

See hipSYCL documentation.

We'll assume you have dpcpp (DPC++ compiler) available.

2.3 SYCL Execution Model

SYCL programs revolve around:

queue: Submits work to a device (similar to OpenCL command queue).
buffer: Manages data across host and device.
accessor: Requests access to a buffer inside a command group; specifies access mode (read, write, read_write) and target (host, device).
handler: Used inside command groups to set kernel arguments and launch parallel work.
nd_range: Defines global and local size (like OpenCL NDRange).
parallel_for: Launches a kernel over a range.

A typical SYCL program:

#include <CL/sycl.hpp>
using namespace sycl;

int main() {
    queue q;  // selects default device

    buffer<float, 1> a_buf{ h_a, range<1>{N} };
    buffer<float, 1> b_buf{ h_b, range<1>{N} };
    buffer<float, 1> c_buf{ range<1>{N} };

    q.submit([&](handler& h) {
        auto a = a_buf.get_access<access::mode::read>(h);
        auto b = b_buf.get_access<access::mode::read>(h);
        auto c = c_buf.get_access<access::mode::write>(h);

        h.parallel_for(range<1>{N}, [=](id<1> i) {
            c[i] = a[i] + b[i];
        });
    });

    // Accessor on host to read result
    auto c_host = c_buf.get_access<access::mode::read>();
    // ... use c_host ...
}

2.4 Memory Model: Buffers and Accessors

Buffers abstract data movement. Accessors create dependencies: when you request an accessor in a command group, the runtime ensures data is available on the device before the kernel runs, and updates the host after.

USM (Unified Shared Memory) is an alternative that uses pointers, similar to CUDA/HIP. SYCL supports device USM, shared USM, and host USM.

2.5 Your First SYCL Program: Vector Addition

Let's rewrite the vector addition using SYCL.

#include <CL/sycl.hpp>
#include <iostream>
#include <vector>

using namespace sycl;

int main() {
    const int N = 1 << 20;
    std::vector<float> a(N), b(N), c(N);
    for (int i = 0; i < N; i++) {
        a[i] = i * 1.0f;
        b[i] = (i + 1) * 1.0f;
    }

    try {
        queue q;  // default device

        // Create buffers from host data (moves data to device automatically)
        buffer<float, 1> a_buf(a.data(), range<1>(N));
        buffer<float, 1> b_buf(b.data(), range<1>(N));
        buffer<float, 1> c_buf(c.data(), range<1>(N));

        // Submit command group
        q.submit([&](handler& h) {
            // Request accessors
            auto a_acc = a_buf.get_access<access::mode::read>(h);
            auto b_acc = b_buf.get_access<access::mode::read>(h);
            auto c_acc = c_buf.get_access<access::mode::write>(h);

            // Launch kernel
            h.parallel_for(range<1>(N), [=](id<1> i) {
                c_acc[i] = a_acc[i] + b_acc[i];
            });
        });

        // Buffers go out of scope, so data is copied back to host automatically
        // (destructor of c_buf waits for completion and updates host)
    } catch (sycl::exception& e) {
        std::cerr << "SYCL exception: " << e.what() << std::endl;
        return 1;
    }

    // Verify
    for (int i = 0; i < 10; i++)
        std::cout << a[i] << " + " << b[i] << " = " << c[i] << std::endl;

    return 0;
}

Compile with:

dpcpp -o vecAdd vecAdd.cpp
./vecAdd

Explanation:

queue q creates a queue to the default device. You can select a specific device using device_selector.
Buffers wrap host data; they don't copy immediately but manage data lazily.
Inside the command group, accessors specify how the kernel uses the data. The runtime ensures data is on the device before execution.
The kernel is a lambda capturing accessors by value. It executes in parallel over a 1D range.
When buffers are destroyed (at end of scope), they wait for any dependent operations and copy data back to host if needed.

2.6 Error Handling and Asynchronous Exceptions

SYCL uses exceptions for synchronous errors. Asynchronous errors (during kernel execution) can be caught via an async handler:

void async_handler(sycl::exception_list e) {
    for (auto& ep : e) {
        try { std::rethrow_exception(ep); }
        catch (sycl::exception& e) { std::cerr << "Async SYCL exception: " << e.what() << std::endl; }
    }
}

queue q(device_selector{}, async_handler);

2.7 Example: Matrix Multiplication with SYCL

Now let's implement matrix multiplication using SYCL's nd_range for work‑groups and local memory.

#include <CL/sycl.hpp>
#include <iostream>
#include <vector>

using namespace sycl;

#define TILE_SIZE 16

int main() {
    const int N = 1024;
    std::vector<float> A(N * N), B(N * N), C(N * N, 0.0f);
    // Initialize A, B...

    try {
        queue q;

        buffer<float, 2> a_buf(A.data(), range<2>(N, N));
        buffer<float, 2> b_buf(B.data(), range<2>(N, N));
        buffer<float, 2> c_buf(C.data(), range<2>(N, N));

        q.submit([&](handler& h) {
            auto a = a_buf.get_access<access::mode::read>(h);
            auto b = b_buf.get_access<access::mode::read>(h);
            auto c = c_buf.get_access<access::mode::write>(h);

            // Local memory accessor
            accessor<float, 2, access::mode::read_write, access::target::local> As(TILE_SIZE, TILE_SIZE, h);
            accessor<float, 2, access::mode::read_write, access::target::local> Bs(TILE_SIZE, TILE_SIZE, h);

            h.parallel_for(nd_range<2>({N, N}, {TILE_SIZE, TILE_SIZE}), [=](nd_item<2> it) {
                int row = it.get_global_id(1);
                int col = it.get_global_id(0);
                int localRow = it.get_local_id(1);
                int localCol = it.get_local_id(0);

                float sum = 0.0f;
                for (int tile = 0; tile < N / TILE_SIZE; ++tile) {
                    // Load tiles
                    As[localRow][localCol] = a[row][tile * TILE_SIZE + localCol];
                    Bs[localRow][localCol] = b[tile * TILE_SIZE + localRow][col];
                    it.barrier(access::fence_space::local_space);

                    // Compute
                    for (int k = 0; k < TILE_SIZE; ++k) {
                        sum += As[localRow][k] * Bs[k][localCol];
                    }
                    it.barrier(access::fence_space::local_space);
                }

                if (row < N && col < N) {
                    c[row][col] = sum;
                }
            });
        });
    } catch (sycl::exception& e) {
        std::cerr << e.what() << std::endl;
        return 1;
    }

    // Use C...
    return 0;
}

Notes:

We use 2D buffers with range<2>.
Local memory is allocated via accessor with target::local.
nd_range combines global and local sizes.
Barriers are called on the nd_item to synchronize work‑group.

2.8 Unified Shared Memory (USM)

USM simplifies memory management by using pointers. There are three types:

device: Allocated on device, accessible only from device.
host: Allocated on host, accessible only from host.
shared: Managed pointer that can be accessed from both host and device (migrates data automatically).

Example with shared USM:

queue q;
float *a = malloc_shared<float>(N, q);
float *b = malloc_shared<float>(N, q);
float *c = malloc_shared<float>(N, q);

// Initialize on host
for (int i = 0; i < N; i++) {
    a[i] = i * 1.0f;
    b[i] = (i + 1) * 1.0f;
}

q.parallel_for(range<1>(N), [=](id<1> i) {
    c[i] = a[i] + b[i];
}).wait();

// c is ready on host
// ...

free(a, q); free(b, q); free(c, q);

USM is more convenient for porting CUDA/HIP code but requires explicit waits or dependencies.

2.9 Advanced SYCL Topics

Device selection: Use default_selector, gpu_selector, cpu_selector, or custom.
Kernel programming model: SYCL supports basic parallel_for, nd_range, and single_task.
Reductions: SYCL 2020 adds built‑in reductions.
Graph dependencies: Use depends_on and events.
Interoperability with OpenCL: SYCL can create buffers from cl_mem, etc.

Part 3: Advanced Topics and Optimization

3.1 Work-Group Size and Occupancy

Both OpenCL and SYCL rely on work‑groups (or blocks) to map to hardware compute units. Choosing the right local size is crucial.

OpenCL: Local size must divide global size evenly (or use clEnqueueNDRangeKernel with NULL local size to let runtime choose). Typically, use multiples of the preferred work‑group size from device info.
SYCL: Local size specified in nd_range. You can query device info: device.get_info<info::device::max_work_group_size>().

A good rule of thumb: start with a work‑group size of 64–256 and tune based on profiling.

3.2 Local Memory and Barriers

OpenCL: Use __local pointers or __local arrays. Synchronize with barrier(CLK_LOCAL_MEM_FENCE).
SYCL: Use local accessors. Synchronize with item.barrier(access::fence_space::local_space).

Local memory is key for data reuse (e.g., tiled matrix multiplication) and reduction.

3.3 Memory Access Patterns

Coalescing: Work‑items in a work‑group should access consecutive global memory addresses to maximize bandwidth. In OpenCL, this means the global id (or get_global_id(0)) should be the fastest‑changing dimension.
Avoiding bank conflicts: In local memory, ensure that within a work‑group, different work‑items access different banks (or all access the same address for broadcast). This may require padding.

3.4 Using SIMD and Vectorization

OpenCL kernels can use vector types (float4, int4, etc.) to explicitly use SIMD. The compiler may also auto‑vectorize. In SYCL, you can use the vec class.

3.5 Profiling and Debugging

OpenCL:

Use events as shown earlier.
Vendor tools: Intel VTune, AMD CodeXL, NVIDIA Visual Profiler (with OpenCL support).

SYCL:

oneAPI provides Intel VTune Profiler and Intel Advisor.
The queue can be constructed with property::queue::enable_profiling() to get kernel times via events.

Debugging:

OpenCL: printf inside kernels works (if supported). Use clBuildProgram logs.
SYCL: Use std::cout in kernels (SYCL 2020 supports device iostream emulation, but it's slow). For serious debugging, use Intel's Debugger for OpenCL or gdb‑oneapi.

3.6 Interoperability: OpenCL and SYCL

SYCL can interoperate with OpenCL objects:

Create SYCL platform/device from cl_platform_id/cl_device_id.
Create SYCL queue from cl_command_queue.
Create SYCL buffer from cl_mem.

This allows incremental porting or mixing low‑level control with high‑level productivity.

3.7 Targeting Different Devices

OpenCL: Code can run on any OpenCL device, but performance tuning may be device‑specific.
SYCL with oneAPI: You can target CPU, GPU, FPGA with the same source. For FPGA, SYCL supports pipeline‑oriented optimizations via [[intel::kernel_args_restrict]] and other attributes.

Example: selecting a GPU device explicitly:

std::vector<device> devices = device::get_devices(info::device_type::gpu);
queue q(devices[0]);

Part 4: Portability and Choosing Between OpenCL and SYCL

4.1 OpenCL Strengths and Weaknesses

Strengths:

Mature, widely supported.
Fine‑grained control over devices and memory.
C‑based kernel language (easy to learn for C programmers).

Weaknesses:

Verbose host code.
No single‑source, so host and device code are separate.
Kernel language lacks modern C++ features.

4.2 SYCL Strengths

Strengths:

Single‑source C++ with lambdas and templates.
RAII and automatic dependency management via buffers.
Portability across backends (OpenCL, Level Zero, CUDA).
Backed by oneAPI ecosystem with optimized libraries.

Weaknesses:

Newer, still evolving (SYCL 2020 is a major step).
Compiler support not as universal as OpenCL.
Learning curve for the abstraction model.

4.3 When to Use Which

Use OpenCL if you need to support very old hardware, have strict control requirements, or are maintaining an existing OpenCL codebase.
Use SYCL/oneAPI for new projects where productivity and portability are priorities, especially if you target Intel GPUs/FPGAs or want to keep options open for AMD/NVIDIA via hipSYCL.

Conclusion

You've now journeyed through the fundamentals of both OpenCL and SYCL, from setting up your environment to writing optimized kernels for vector addition and matrix multiplication. You've seen how OpenCL gives you explicit control, while SYCL provides a modern C++ interface that simplifies many aspects of heterogeneous programming.

Both frameworks are powerful tools in the developer's toolbox. As you continue, explore real‑world applications, contribute to open‑source projects, and leverage the rich ecosystem of libraries (oneMKL, oneDNN, etc.). The world of heterogeneous computing is vast, and you're now equipped to navigate it.

Next Steps:

Dive into the OpenCL specification and SYCL 2020 specification.
Experiment with more complex patterns: reductions, scans, stencils.
Try SYCL on different hardware (CPU, GPU, FPGA) and compare performance.
Explore oneAPI's domain‑specific libraries for deep learning, ray tracing, etc.

If you found this guide valuable, share it with your peers. Have questions or want to share your experiences? Leave a comment below. Happy coding!

Advanced GPU Optimization: CUDA & HIP from zero to hero

Javad — Sat, 14 Feb 2026 16:23:19 +0000

Welcome, fellow developer! If you’ve ever wanted to tap into the immense power of graphics processors for general-purpose computing, you’re in the right place. GPU programming used to be an NVIDIA-only game with CUDA, but the rise of AMD GPUs in workstations, servers, and supercomputers has made vendor lock‑in a thing of the past. Enter HIP – a C++ runtime API that lets you write code that runs on both NVIDIA and AMD GPUs with minimal changes.

In this comprehensive guide, we’ll start from the very basics (no prior GPU experience assumed) and work our way up to advanced topics like streams, shared memory optimizations, and multi‑GPU programming. Along the way, you’ll learn CUDA inside out and then discover how HIP gives you portability without sacrificing performance. By the end, you’ll be comfortable writing efficient, portable GPU kernels and ready to tackle real‑world parallel problems.

Let’s dive in!

Part 1: CUDA Fundamentals

1.1 Setting Up CUDA

Before we write any code, we need a working CUDA development environment.

Check your hardware: You need an NVIDIA GPU with Compute Capability 3.0 or higher. Run lspci | grep -i nvidia on Linux or check Device Manager on Windows.
Install the CUDA Toolkit:
- Download the appropriate installer from NVIDIA CUDA Toolkit.
- Follow the installation guide for your OS.
Verify installation:

   nvcc --version

You should see the CUDA version. Also run nvidia-smi to check driver and GPU status.

Set environment variables (if not done automatically):

   export PATH=/usr/local/cuda/bin:$PATH
   export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

Now you’re ready to compile CUDA code with nvcc.

1.2 The CUDA Programming Model

A GPU contains hundreds or thousands of cores. To use them effectively, we organize work into a hierarchy:

Thread: The smallest unit of execution. Each thread runs the same kernel function but works on different data.
Block: A group of threads that can cooperate via shared memory and synchronize with __syncthreads(). Threads in the same block run on the same streaming multiprocessor (SM).
Grid: A collection of blocks that together cover the entire problem.

When you launch a kernel, you specify the grid and block dimensions (in 1D, 2D, or 3D). Each thread gets unique indices:

threadIdx.x, threadIdx.y, threadIdx.z – position inside its block.
blockIdx.x, blockIdx.y, blockIdx.z – position of the block in the grid.
blockDim.x, blockDim.y, blockDim.z – size of each block.
gridDim.x, gridDim.y, gridDim.z – size of the grid.

The typical formula to map a thread to a data element is:

int i = blockIdx.x * blockDim.x + threadIdx.x;

1.3 Memory Hierarchy

Global memory: Large (several GB), high latency, accessible by all threads. It persists across kernel launches.
Shared memory: Small (tens of KB per block), low latency, shared among threads in the same block. Programmer‑managed cache.
Registers: Private to each thread, fastest.
Constant memory: Read‑only, cached, good for broadcast data.
Texture memory: Specialized for 2D spatial locality.

Understanding this hierarchy is key to writing fast kernels.

1.4 Your First CUDA Kernel: Vector Addition

Let’s implement a simple vector addition. This will introduce all the essential steps.

#include <iostream>
#include <cuda_runtime.h>

// Kernel definition
__global__ void vecAdd(const float *a, const float *b, float *c, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) {  // Guard against extra threads
        c[i] = a[i] + b[i];
    }
}

int main() {
    int n = 1 << 20;  // 1,048,576 elements
    size_t bytes = n * sizeof(float);

    // Allocate host memory
    float *h_a = new float[n];
    float *h_b = new float[n];
    float *h_c = new float[n];

    // Initialize input arrays
    for (int i = 0; i < n; i++) {
        h_a[i] = i * 1.0f;
        h_b[i] = (i + 1) * 1.0f;
    }

    // Allocate device memory
    float *d_a, *d_b, *d_c;
    cudaMalloc(&d_a, bytes);
    cudaMalloc(&d_b, bytes);
    cudaMalloc(&d_c, bytes);

    // Copy data from host to device
    cudaMemcpy(d_a, h_a, bytes, cudaMemcpyHostToDevice);
    cudaMemcpy(d_b, h_b, bytes, cudaMemcpyHostToDevice);

    // Launch kernel
    int threadsPerBlock = 256;
    int blocksPerGrid = (n + threadsPerBlock - 1) / threadsPerBlock;
    vecAdd<<<blocksPerGrid, threadsPerBlock>>>(d_a, d_b, d_c, n);

    // Copy result back to host
    cudaMemcpy(h_c, d_c, bytes, cudaMemcpyDeviceToHost);

    // Verify (simple check)
    for (int i = 0; i < n; i++) {
        if (h_c[i] != h_a[i] + h_b[i]) {
            std::cout << "Mismatch at index " << i << "\n";
            break;
        }
    }

    // Cleanup
    cudaFree(d_a);
    cudaFree(d_b);
    cudaFree(d_c);
    delete[] h_a;
    delete[] h_b;
    delete[] h_c;

    return 0;
}

Explanation:

__global__ marks a function as a kernel – callable from host and executed on device.
We compute a global index i and, if within bounds, perform the addition.
<<<blocksPerGrid, threadsPerBlock>>> sets the execution configuration.
cudaMalloc allocates memory on the GPU.
cudaMemcpy transfers data between host and device (the last argument specifies direction).
Always free device memory with cudaFree to avoid leaks.

Compile with:

nvcc -o vecAdd vecAdd.cu
./vecAdd

If everything works, you should see no output (or a success message if you add one).

1.5 Understanding Kernel Launch Parameters

The launch configuration <<<grid, block, sharedMem, stream>>> has two required parameters and two optional:

grid (dim3): number of blocks per grid. Can be 1D, 2D, or 3D. E.g., dim3 grid(16, 16) creates a 16x16 grid (256 blocks).
block (dim3): number of threads per block. Usually 1D, but can be 2D/3D. Total threads per block ≤ 1024 (for modern GPUs).
sharedMem (size_t): dynamic shared memory per block (default 0).
stream (cudaStream_t): specifies which stream to enqueue the kernel in (default 0, the null stream).

In our example we used 1D grids and blocks. For 2D data (like images), you might use 2D blocks:

dim3 block(16, 16);
dim3 grid((width + block.x - 1) / block.x, (height + block.y - 1) / block.y);
kernel<<<grid, block>>>(...);

Inside the kernel, you would then compute:

int x = blockIdx.x * blockDim.x + threadIdx.x;
int y = blockIdx.y * blockDim.y + threadIdx.y;
int idx = y * width + x;

1.6 Memory Management in Depth

cudaMalloc(void **ptr, size_t size) – allocates linear memory on device.
cudaFree(void *ptr) – frees device memory.
cudaMemcpy(void *dst, const void *src, size_t count, cudaMemcpyKind kind) – synchronously copies memory.
cudaMemcpyAsync – asynchronous version, requires pinned host memory.
cudaMallocHost – allocates page‑locked (pinned) host memory, which enables faster transfers and asynchronous copies.
cudaMemset – sets device memory to a value.

Pinned memory example:

float *h_a;
cudaMallocHost(&h_a, bytes);   // pinned
cudaMemcpyAsync(d_a, h_a, bytes, cudaMemcpyHostToDevice, stream);

Use pinned memory when you have many transfers or overlapping with computation.

1.7 Error Handling

CUDA functions return an error code of type cudaError_t. You should always check them:

cudaError_t err = cudaMalloc(&d_a, bytes);
if (err != cudaSuccess) {
    std::cerr << "cudaMalloc failed: " << cudaGetErrorString(err) << std::endl;
    exit(EXIT_FAILURE);
}

For kernel launches, which are asynchronous, you can check for launch errors with cudaGetLastError() and wait for completion with cudaDeviceSynchronize() to catch execution errors:

kernel<<<grid, block>>>(args);
err = cudaGetLastError();   // catch launch errors
if (err != cudaSuccess) { ... }
cudaDeviceSynchronize();    // wait and catch execution errors
err = cudaGetLastError();
if (err != cudaSuccess) { ... }

Better yet, wrap calls in a macro:

#define CUDA_CHECK(call) \
    do { \
        cudaError_t err = call; \
        if (err != cudaSuccess) { \
            std::cerr << "CUDA error in " << __FILE__ << ":" << __LINE__ << ": " \
                      << cudaGetErrorString(err) << std::endl; \
            exit(EXIT_FAILURE); \
        } \
    } while(0)

// Usage:
CUDA_CHECK(cudaMalloc(&d_a, bytes));

1.8 Profiling and Debugging Basics

Profiling: NVIDIA provides nvprof (deprecated but still useful) and the newer Nsight Systems / Nsight Compute. To profile a simple app:

  nvprof ./vecAdd

This shows kernel execution time, memory transfer times, etc. For deeper analysis, use Nsight Compute: ncu ./vecAdd.

Debugging: Use cuda-gdb (Linux) or the Visual Studio debugger on Windows. Compile with -g -G to generate debug info:

  nvcc -g -G -o vecAdd vecAdd.cu
  cuda-gdb ./vecAdd

Inside cuda-gdb, you can set breakpoints, inspect threads, and step through device code.

Memory error checking: compute-sanitizer (formerly cuda-memcheck) helps detect out‑of‑bounds accesses, race conditions, etc.

  compute-sanitizer ./vecAdd

1.9 Practice: Matrix Multiplication

Let's solidify our understanding with a classic: matrix multiplication C = A * B (N x N). We'll start with a naive kernel.

Naive implementation (each thread computes one element of C):

__global__ void matMulNaive(const float *A, const float *B, float *C, int N) {
    int row = blockIdx.y * blockDim.y + threadIdx.y;
    int col = blockIdx.x * blockDim.x + threadIdx.x;

    if (row < N && col < N) {
        float sum = 0.0f;
        for (int k = 0; k < N; k++) {
            sum += A[row * N + k] * B[k * N + col];
        }
        C[row * N + col] = sum;
    }
}

Launch configuration (assuming 16x16 blocks):

dim3 block(16, 16);
dim3 grid((N + block.x - 1) / block.x, (N + block.y - 1) / block.y);
matMulNaive<<<grid, block>>>(d_A, d_B, d_C, N);

Performance issues: This naive version has poor memory access patterns (especially for B, where each thread accesses a different column, causing non‑coalesced reads) and no data reuse. We'll improve it later with shared memory.

Part 2: Intermediate CUDA

Now that you're comfortable with basic CUDA, let's explore techniques to write faster kernels.

2.1 Performance Considerations

Occupancy: The ratio of active warps to maximum possible warps on an SM. Higher occupancy can hide latency but isn't always the goal. Use the CUDA Occupancy Calculator or the cudaOccupancyMaxPotentialBlockSize function to tune block size.

Warp divergence: Threads in a warp execute in lockstep. If they take different paths (e.g., an if‑else), the warp serially executes both paths, reducing performance. Try to keep control flow uniform within warps.

Coalesced memory access: Global memory bandwidth is best utilized when consecutive threads access consecutive memory locations. In our vector addition, thread i accesses a[i] – perfectly coalesced. In naive matrix multiplication, row‑major access of A is coalesced, but column‑major access of B is not. We'll fix that with shared memory.

2.2 Shared Memory and Bank Conflicts

Shared memory is divided into banks. If multiple threads access the same bank (with some exceptions), we get a bank conflict and the accesses are serialized. To avoid conflicts, ensure that threads within a warp access different banks (or all access the same word in broadcast mode). This is an advanced topic, but be aware of it when designing shared memory layouts.

2.3 Tiled Matrix Multiplication with Shared Memory

Let's rewrite matrix multiplication to use shared memory. The idea: load tiles of A and B into shared memory, then compute partial sums.

#define TILE_SIZE 16

__global__ void matMulShared(const float *A, const float *B, float *C, int N) {
    __shared__ float As[TILE_SIZE][TILE_SIZE];
    __shared__ float Bs[TILE_SIZE][TILE_SIZE];

    int bx = blockIdx.x, by = blockIdx.y;
    int tx = threadIdx.x, ty = threadIdx.y;

    // Identify the row and column of the C element to compute
    int row = by * TILE_SIZE + ty;
    int col = bx * TILE_SIZE + tx;

    float sum = 0.0f;

    // Loop over all tiles
    for (int tile = 0; tile < N / TILE_SIZE; ++tile) {
        // Load one tile of A and B into shared memory
        As[ty][tx] = A[row * N + (tile * TILE_SIZE + tx)];
        Bs[ty][tx] = B[(tile * TILE_SIZE + ty) * N + col];
        __syncthreads();  // Ensure tile is loaded before computation

        // Compute partial product for this tile
        for (int k = 0; k < TILE_SIZE; ++k) {
            sum += As[ty][k] * Bs[k][tx];
        }
        __syncthreads();  // Avoid overwriting shared memory before next tile
    }

    if (row < N && col < N) {
        C[row * N + col] = sum;
    }
}

What changed?

Each block loads a tile of A and a tile of B into shared memory.
All threads in the block cooperate to load the tiles (coalesced because threads in a warp load consecutive elements from global memory).
Computation uses the fast shared memory.
__syncthreads() ensures that all threads have finished loading before any thread starts using the data, and that no thread starts the next tile before all have finished the current tile.

This kernel can achieve near‑peak performance for large matrices.

2.4 Atomic Operations

When multiple threads need to update the same location (e.g., histogram, reduction), you can use atomic operations to avoid race conditions.

__global__ void histogram(const unsigned char *data, unsigned int *bins, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) {
        atomicAdd(&bins[data[i]], 1);
    }
}

CUDA provides atomicAdd, atomicSub, atomicExch, atomicMin, atomicMax, atomicAnd, atomicOr, atomicXor for various data types.

Performance: Atomics can be slow if many threads contend for the same location. Use shared memory to reduce contention when possible (e.g., each block computes its own partial histogram, then combine with atomics).

2.5 Streams and Concurrency

Streams allow you to overlap kernel execution with data transfers and other operations. A stream is a sequence of operations that execute in order. Operations from different streams may overlap.

Example: overlapping copy and compute:

cudaStream_t stream1, stream2;
cudaStreamCreate(&stream1);
cudaStreamCreate(&stream2);

// Split data into two halves
int half = n / 2;

// Asynchronous memory copies (requires pinned host memory)
cudaMemcpyAsync(d_a1, h_a1, half_bytes, cudaMemcpyHostToDevice, stream1);
cudaMemcpyAsync(d_a2, h_a2, half_bytes, cudaMemcpyHostToDevice, stream2);

// Launch kernels in respective streams
kernel<<<grid, block, 0, stream1>>>(d_a1, d_b1, d_c1, half);
kernel<<<grid, block, 0, stream2>>>(d_a2, d_b2, d_c2, half);

// Copy back
cudaMemcpyAsync(h_c1, d_c1, half_bytes, cudaMemcpyDeviceToHost, stream1);
cudaMemcpyAsync(h_c2, d_c2, half_bytes, cudaMemcpyDeviceToHost, stream2);

cudaStreamSynchronize(stream1);
cudaStreamSynchronize(stream2);

cudaStreamDestroy(stream1);
cudaStreamDestroy(stream2);

Note: Use pinned memory (cudaMallocHost) for asynchronous transfers. Also, kernels from different streams may run concurrently if resources permit.

2.6 Events for Timing

CUDA events provide a way to measure elapsed time on the device accurately.

cudaEvent_t start, stop;
cudaEventCreate(&start);
cudaEventCreate(&stop);

cudaEventRecord(start);
kernel<<<grid, block>>>(args);
cudaEventRecord(stop);

cudaEventSynchronize(stop);  // Wait for the stop event to be recorded
float milliseconds = 0;
cudaEventElapsedTime(&milliseconds, start, stop);
std::cout << "Kernel time: " << milliseconds << " ms\n";

cudaEventDestroy(start);
cudaEventDestroy(stop);

2.7 Unified Memory

Unified Memory (UM) creates a single pointer accessible from both CPU and GPU. Data is automatically migrated on demand. It simplifies programming but may introduce overhead.

Allocate UM:

float *x;
cudaMallocManaged(&x, N * sizeof(float));

You can then use x on both host and device without explicit cudaMemcpy. To ensure data is ready on the device, you can prefetch:

cudaMemPrefetchAsync(x, N * sizeof(float), deviceId, 0);

UM is great for prototyping and for algorithms with irregular data access. For performance‑critical code, manual transfers are still recommended.

2.8 Multi‑GPU Programming

Modern systems may have multiple GPUs. You can use them to scale your application.

Use cudaGetDeviceCount to find number of devices.
Set current device with cudaSetDevice(deviceId) before allocating memory or launching kernels.
Each device has its own memory space; you must manage data distribution and communication between devices (e.g., via PCIe or NVLink). Use peer‑to‑peer access if available (cudaDeviceCanAccessPeer, cudaDeviceEnablePeerAccess).

Simple multi‑GPU example:

int numDevices;
cudaGetDeviceCount(&numDevices);

#pragma omp parallel for
for (int dev = 0; dev < numDevices; ++dev) {
    cudaSetDevice(dev);
    // Allocate memory and launch kernel on this device
    // Copy data to/from host (strided or partitioned)
}

2.9 Advanced Example: Parallel Reduction

Reduction (summing an array) is a common pattern. We'll implement an optimized version that uses shared memory and avoids warp divergence.

__global__ void reduce(float *g_idata, float *g_odata, int n) {
    extern __shared__ float sdata[];  // Dynamic shared memory

    unsigned int tid = threadIdx.x;
    unsigned int i = blockIdx.x * blockDim.x + threadIdx.x;

    // Load data into shared memory (with bounds check)
    sdata[tid] = (i < n) ? g_idata[i] : 0.0f;
    __syncthreads();

    // Perform reduction in shared memory
    for (unsigned int s = blockDim.x / 2; s > 0; s >>= 1) {
        if (tid < s) {
            sdata[tid] += sdata[tid + s];
        }
        __syncthreads();
    }

    // Write result for this block to global memory
    if (tid == 0) {
        g_odata[blockIdx.x] = sdata[0];
    }
}

Launch:

int threads = 256;
int blocks = (n + threads - 1) / threads;
reduce<<<blocks, threads, threads * sizeof(float)>>>(d_in, d_out, n);

After this kernel, you have partial sums per block. You may need a second kernel to combine them, or run a single block on the CPU.

Part 3: Introduction to HIP

3.1 What is HIP? Why Portability?

HIP (Heterogeneous‑compute Interface for Portability) is an AMD initiative to provide a C++ runtime API that can target both NVIDIA and AMD GPUs. It is essentially a thin abstraction layer: HIP code compiles to CUDA (using a CUDA backend) when targeting NVIDIA, and to AMD’s ROCm when targeting AMD. This means you can write your kernel once and run it everywhere.

Major benefits:

Vendor independence: No lock‑in to NVIDIA.
Single source: One codebase for multiple platforms.
Ease of porting: CUDA code can be mechanically converted using hipify tools.

3.2 Setting Up HIP

On AMD systems with ROCm:

Follow the ROCm installation guide. On Ubuntu:

  sudo apt update
  sudo apt install rocm-dev

Verify with hipcc --version.

On NVIDIA systems (HIP on CUDA):

Install the CUDA Toolkit first.
Then install HIP from ROCm’s GitHub or via package manager. Many Linux distributions provide a hip-cuda package.
After installation, hipcc will use the CUDA backend.

Environment:

Ensure hipcc is in your PATH.
For AMD, you may need to set ROCM_PATH if not default.

3.3 HIP API Overview

HIP intentionally mimics CUDA. Most CUDA functions have direct HIP equivalents with the hip prefix:

CUDA	HIP
`cudaMalloc`	`hipMalloc`
`cudaFree`	`hipFree`
`cudaMemcpy`	`hipMemcpy`
`cudaMemcpyAsync`	`hipMemcpyAsync`
`cudaDeviceSynchronize`	`hipDeviceSynchronize`
`cudaStreamCreate`	`hipStreamCreate`
`cudaEventCreate`	`hipEventCreate`
`cudaGetLastError`	`hipGetLastError`
`cudaError_t`	`hipError_t`
`cudaSuccess`	`hipSuccess`

The kernel language is essentially the same: __global__, __device__, __shared__, threadIdx, blockIdx, etc., are all supported.

The only major syntactic difference is the kernel launch. Instead of triple chevrons, HIP uses a function macro hipLaunchKernelGGL:

hipLaunchKernelGGL(kernel_name, dim3(grid), dim3(block), sharedMem, stream, args...);

If you prefer the <<<>>> syntax, you can define a macro (but it's not provided by default).

3.4 Porting a CUDA Program to HIP

Let’s port our vector addition from CUDA to HIP.

Original CUDA kernel (unchanged):

__global__ void vecAdd(const float *a, const float *b, float *c, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) {
        c[i] = a[i] + b[i];
    }
}

Host code converted to HIP:

#include <iostream>
#include <hip/hip_runtime.h>

int main() {
    int n = 1 << 20;
    size_t bytes = n * sizeof(float);

    float *h_a = new float[n];
    float *h_b = new float[n];
    float *h_c = new float[n];

    for (int i = 0; i < n; i++) {
        h_a[i] = i * 1.0f;
        h_b[i] = (i + 1) * 1.0f;
    }

    float *d_a, *d_b, *d_c;
    hipMalloc(&d_a, bytes);
    hipMalloc(&d_b, bytes);
    hipMalloc(&d_c, bytes);

    hipMemcpy(d_a, h_a, bytes, hipMemcpyHostToDevice);
    hipMemcpy(d_b, h_b, bytes, hipMemcpyHostToDevice);

    int threadsPerBlock = 256;
    int blocksPerGrid = (n + threadsPerBlock - 1) / threadsPerBlock;

    // HIP kernel launch
    hipLaunchKernelGGL(vecAdd, dim3(blocksPerGrid), dim3(threadsPerBlock), 0, 0, d_a, d_b, d_c, n);

    hipMemcpy(h_c, d_c, bytes, hipMemcpyDeviceToHost);

    // Verification (omitted for brevity)

    hipFree(d_a); hipFree(d_b); hipFree(d_c);
    delete[] h_a; delete[] h_b; delete[] h_c;

    return 0;
}

Compile with:

hipcc -o vecAdd vecAdd.cpp   # on AMD
# or on NVIDIA
hipcc --offload-arch=sm_70 -o vecAdd vecAdd.cpp

Notice that the kernel itself is identical to the CUDA version – that’s the beauty of HIP!

3.5 HIP Kernel Launch Syntax Explained

hipLaunchKernelGGL takes the following arguments:

Kernel name (symbol)
Grid dimensions (dim3)
Block dimensions (dim3)
Dynamic shared memory size per block (size_t, default 0)
Stream (hipStream_t, default 0 for null stream)
Kernel arguments (comma‑separated)

If you have many arguments, it can become verbose. Some projects define a macro like:

#define HIP_LAUNCH(kernel, grid, block, shared, stream, ...) \
    hipLaunchKernelGGL(kernel, grid, block, shared, stream, __VA_ARGS__)

3.6 Error Handling in HIP

HIP error handling is identical to CUDA:

hipError_t err = hipMalloc(&d_a, bytes);
if (err != hipSuccess) {
    std::cerr << "HIP error: " << hipGetErrorString(err) << std::endl;
    exit(EXIT_FAILURE);
}

After a kernel launch, you can check for launch errors with hipGetLastError() and for execution errors after synchronization with hipDeviceSynchronize().

3.7 Debugging and Profiling HIP

Debugging:

On AMD, use rocgdb (similar to cuda‑gdb). Compile with -g and run rocgdb ./vecAdd.
On NVIDIA, you can still use cuda-gdb because the HIP code is translated to CUDA under the hood.

Profiling:

On AMD: Use rocprof to collect performance counters and traces.

  rocprof --hip-trace ./vecAdd

This generates a results.json file that can be viewed with tools like perfetto.

On NVIDIA: Use nvprof or nsight-compute as usual.

Memory checking:

On AMD, use compute-sanitizer (if available) or ROCm’s hip-memcheck.

3.8 HIP Examples: Matrix Multiplication

Our tiled matrix multiplication kernel works unchanged in HIP. Simply change the header and launch syntax. Let's see:

#include <hip/hip_runtime.h>
#include <iostream>

#define TILE_SIZE 16

__global__ void matMulShared(const float *A, const float *B, float *C, int N) {
    __shared__ float As[TILE_SIZE][TILE_SIZE];
    __shared__ float Bs[TILE_SIZE][TILE_SIZE];

    int bx = blockIdx.x, by = blockIdx.y;
    int tx = threadIdx.x, ty = threadIdx.y;
    int row = by * TILE_SIZE + ty;
    int col = bx * TILE_SIZE + tx;

    float sum = 0.0f;
    for (int tile = 0; tile < N / TILE_SIZE; ++tile) {
        As[ty][tx] = A[row * N + (tile * TILE_SIZE + tx)];
        Bs[ty][tx] = B[(tile * TILE_SIZE + ty) * N + col];
        __syncthreads();

        for (int k = 0; k < TILE_SIZE; ++k) {
            sum += As[ty][k] * Bs[k][tx];
        }
        __syncthreads();
    }

    if (row < N && col < N) {
        C[row * N + col] = sum;
    }
}

int main() {
    int N = 1024;
    size_t bytes = N * N * sizeof(float);

    // ... host allocations and initialization ...

    float *d_A, *d_B, *d_C;
    hipMalloc(&d_A, bytes);
    hipMalloc(&d_B, bytes);
    hipMalloc(&d_C, bytes);

    hipMemcpy(d_A, h_A, bytes, hipMemcpyHostToDevice);
    hipMemcpy(d_B, h_B, bytes, hipMemcpyHostToDevice);

    dim3 block(TILE_SIZE, TILE_SIZE);
    dim3 grid((N + block.x - 1) / block.x, (N + block.y - 1) / block.y);

    hipLaunchKernelGGL(matMulShared, grid, block, 0, 0, d_A, d_B, d_C, N);

    hipMemcpy(h_C, d_C, bytes, hipMemcpyDeviceToHost);

    // ... verification and cleanup ...

    return 0;
}

Compile with hipcc matmul.cpp -o matmul.

Part 4: Advanced HIP and Porting

4.1 Subtle Differences Between CUDA and HIP

While HIP aims for source compatibility, there are some differences to be aware of:

Warp vs Wavefront: NVIDIA warps are 32 threads; AMD wavefronts are 64 (on current architectures). Code that assumes a warp size of 32 (e.g., warp shuffle intrinsics) may need adjustment. HIP provides __hip_warp_size which returns the actual warp/wavefront size (32 on NVIDIA, 64 on AMD). Use it to write portable code.
Intrinsics: Some CUDA intrinsics (e.g., __popc, __ballot_sync) have HIP equivalents (__popc, __ballot_sync work, but check the HIP documentation for availability). For warp‑level primitives, prefer the portable __sync_warp and __all, __any.
Math functions: Most math functions (sin, exp, etc.) are identical. However, double‑precision performance may vary between vendors.
Architecture‑specific features: Features like tensor cores (NVIDIA) or matrix cores (AMD) are not portable. If you need them, you may have to write vendor‑specific code guarded by macros.

4.2 Using hipify Tools

AMD provides tools to automatically convert CUDA source to HIP:

hipify-perl: A Perl script that performs text‑based replacement.

  hipify-perl vecAdd.cu > vecAdd.cpp

It handles most API renames and some launch syntax, but you'll likely need to manually adjust the kernel launches (the triple chevrons are converted to hipLaunchKernelGGL).

hipify-clang: A more robust tool based on Clang’s AST. It can handle complex code and provides better accuracy.

  hipify-clang vecAdd.cu --cuda-path=/usr/local/cuda

It outputs HIP code to stdout or a file.

After conversion, review the output, especially for any use of CUDA‑specific features that don’t have direct HIP equivalents.

4.3 Writing Portable Code: Best Practices

To ensure your code runs efficiently on both platforms:

Use hip runtime API for memory management, streams, events.
Avoid hard‑coding warp size: Use __hip_warp_size or compute dynamically.
Use portable intrinsics: Prefer __syncthreads, __threadfence, __syncwarp (though __syncwarp may need a mask; HIP provides __syncwarp(mask) with similar semantics).
Conditional compilation for vendor‑specific optimizations:

  #ifdef __HIP_PLATFORM_NVIDIA__
      // NVIDIA-specific code (e.g., using CUDA tensor cores)
  #elif __HIP_PLATFORM_AMD__
      // AMD-specific code
  #endif

Profile on both platforms to identify bottlenecks. What's fast on NVIDIA may not be on AMD due to different cache hierarchies, wavefront size, etc. Tune block sizes and tile sizes accordingly.

4.4 Performance Optimization on AMD GPUs

When targeting AMD, use ROCm tools:

rocprof: For profiling.
rocm-smi: To monitor GPU utilization, clock speeds, temperature.
OmniTrace: For tracing API calls and kernel execution.

Key optimization considerations for AMD:

Wavefront size: 64 threads. Your block size should be a multiple of 64 to avoid idle threads.
Shared memory bank conflicts: AMD’s shared memory has 32 banks (like NVIDIA) but the mapping may differ. Experiment with padding to reduce conflicts.
Memory coalescing: Still crucial; consecutive threads should access consecutive memory.
Occupancy: Similar concepts apply, but the register file and shared memory sizes differ. Use hipOccupancyMaxPotentialBlockSize to get recommendations.

4.5 Real-world Example: Sobel Edge Detection (Portable)

Let's combine everything into a real‑world kernel: Sobel edge detection on an 8‑bit grayscale image. We'll write it portably for HIP/CUDA.

#include <hip/hip_runtime.h>  // or <cuda_runtime.h> for CUDA

#define TILE_SIZE 16
#define MASK_SIZE 3

__global__ void sobel(const unsigned char *in, unsigned char *out, int width, int height) {
    // Shared memory for tile + halo (1 pixel halo on each side)
    __shared__ unsigned char tile[TILE_SIZE + 2][TILE_SIZE + 2];

    int x = blockIdx.x * TILE_SIZE + threadIdx.x;
    int y = blockIdx.y * TILE_SIZE + threadIdx.y;

    // Global coordinates including halo
    int gx = x - 1;
    int gy = y - 1;

    // Load tile with boundary check
    if (gx >= 0 && gx < width && gy >= 0 && gy < height)
        tile[threadIdx.y + 1][threadIdx.x + 1] = in[gy * width + gx];
    else
        tile[threadIdx.y + 1][threadIdx.x + 1] = 0;

    // Load halo (simplified: we rely on neighboring threads)
    // For a full implementation, you'd also load the border regions.
    __syncthreads();

    // Compute Sobel only for inner tile pixels (not halo)
    if (threadIdx.x < TILE_SIZE && threadIdx.y < TILE_SIZE && x < width && y < height) {
        int Gx = -tile[threadIdx.y][threadIdx.x] + tile[threadIdx.y][threadIdx.x + 2]
                 -2*tile[threadIdx.y+1][threadIdx.x] + 2*tile[threadIdx.y+1][threadIdx.x+2]
                 -tile[threadIdx.y+2][threadIdx.x] + tile[threadIdx.y+2][threadIdx.x+2];
        int Gy = -tile[threadIdx.y][threadIdx.x] -2*tile[threadIdx.y][threadIdx.x+1] - tile[threadIdx.y][threadIdx.x+2]
                 +tile[threadIdx.y+2][threadIdx.x] +2*tile[threadIdx.y+2][threadIdx.x+1] + tile[threadIdx.y+2][threadIdx.x+2];
        int mag = min(255, (int)hypot(Gx, Gy));
        out[y * width + x] = mag;
    }
}

Launch:

dim3 block(TILE_SIZE, TILE_SIZE);
dim3 grid((width + block.x - 1) / block.x, (height + block.y - 1) / block.y);
sobel<<<grid, block>>>(d_in, d_out, width, height); // CUDA
// HIP: hipLaunchKernelGGL(sobel, grid, block, 0, 0, d_in, d_out, width, height);

This kernel uses shared memory to reduce global memory traffic. Each block loads a tile plus halo, then computes Sobel for its interior. It's a great example of a real‑world, portable GPU kernel.

Part 5: Conclusion and Next Steps

Congratulations! You've journeyed from the absolute basics of GPU programming to writing advanced, portable kernels that run on both NVIDIA and AMD hardware. You've learned:

The CUDA programming model, memory hierarchy, and essential APIs.
How to write and optimize kernels for maximum performance.
The HIP ecosystem and how to write truly portable code.
Debugging and profiling techniques for both platforms.

Where to Go from Here

Explore official documentation:
- CUDA C++ Programming Guide
- HIP Documentation
Study advanced examples: Reduction, scan, stencils, FFTs.
Contribute to open‑source projects that use CUDA/HIP (e.g., PyTorch, TensorFlow, ROCm libraries).
Experiment with multi‑GPU and distributed computing (e.g., using MPI with GPUs).
Learn about new hardware features like tensor cores (NVIDIA) and matrix cores (AMD) – but remember to keep your code portable or use conditional compilation.

GPU programming is a powerful skill that opens doors to high‑performance computing, machine learning, and graphics. With CUDA and HIP under your belt, you're well‑equipped to tackle the most demanding computational challenges. Now go forth and write some fast, portable code!

If you found this guide helpful, please share it with your fellow developers. Have questions or want to share your own experiences? Drop a comment below – I’d love to hear from you. Happy coding!

OLSRT: The Runtime I Built on Windows 7 When I Couldn't Run MSVC (And Why It Might Be the Last Runtime You'll Ever Need)

Javad — Sat, 14 Feb 2026 16:04:54 +0000

Hey Dev Community!

A while back, I was sitting on my old Windows 7 machine, trying to build a PHP extension. I wanted to bring modern async I/O, WebSocket support, and an event loop to PHP — something like what Node.js has, but for PHP. The problem? phpize requires MSVC, and on Windows 7 with MinGW, I couldn't make it work. No matter what I tried, MSVC just wouldn't install.

So I sat there, frustrated, staring at my screen. And then a thought hit me:

"C can talk to any language. So why am I limiting myself to just one?"

That question changed everything. Instead of building a PHP extension, I started building something much bigger: a language-agnostic runtime that any language could talk to. A runtime that would give every language access to async I/O, WebSocket, event loops, and more.

That was the birth of OLSRT.

The first version (0.1) was a mess. Just a single src folder, no header files, and barely any structure. But it worked — enough to prove the concept.

Then I rewrote everything from scratch. Today, OLSRT is at v1.2 (Atom) , and it's production-ready. Here's what's in it right now:

✅ What OLSRT v1.2 Offers Today

1. Actor Model with Full Process Isolation

Each actor runs in its own isolated process with a private memory arena. If one actor crashes, the rest keep running — just like Erlang/OTP, but in C. The implementation includes:

Lock‑free mailboxes with zero‑copy message passing
Supervisors with restart strategies (permanent, transient, temporary)
Process linking and monitoring (Erlang‑style)

2. Green Threads (Not Just Coroutines)

OLSRT has a full green thread implementation with:

Work stealing across multiple CPU cores
NUMA‑aware memory allocation
SIMD‑accelerated memory operations (memcpy, memset, memcmp)
Guard pages to detect stack overflows
Context switching in assembly for x86_64, ARM64, ARM, and RISC‑V

3. High‑Performance Memory Management

Arena‑based memory allocators (2MB default, expandable)
Zero‑copy IPC between processes
Lock‑free data structures for hot paths
Cache‑line padding to prevent false sharing

4. Async I/O and Event Loop

Cross‑platform event loop with epoll (Linux), kqueue (macOS/BSD), and select (fallback)
Timer support with deadlines
I/O event registration with callbacks

5. Channels and Streams

Thread‑safe FIFO channels with bounded/unbounded capacity
Reactive streams with operators (map, filter, take, merge, debounce)
Backpressure support with demand‑based flow control

6. Promises and Futures

Thread‑safe promise/future implementation
Continuations (.then())
Integration with the event loop for waking

7. Synchronization Primitives

Mutexes, condition variables, read‑write locks, semaphores
All implemented natively on Windows, Linux, macOS, and BSD

8. Networking (TCP/UDP)

Non‑blocking TCP and UDP sockets
Full integration with the event loop
Promise‑based async API (send/recv/accept/connect)

🚧 What's Coming in v1.3 (In Development Now)

Based on our latest commits (as of February 2026), v1.3 will include:

Hot‑Coding References (HCR) – Never pause the main with very lightweight branches.
OLSRT Coroutines (ORoutines) – Even lighter‑weight concurrency
Fully documented runtime
Optimized build systems (CMake and Makefile improvements)

Why OLSRT Exists (The Philosophy)

I didn't set out to build a universal runtime. I just wanted to make PHP async. But along the way, I realized something:

The future of software is polyglot. We use PHP for the web, Python for data, Rust for systems, and JavaScript for the frontend. But today, each language has its own runtime, its own concurrency model, and its own set of limitations. OLSRT is an attempt to unify them — to give every language access to the same powerful primitives: actors, green threads, zero‑copy IPC, and fault‑tolerant supervisors.

Where We Stand Today

OLSRT v1.2 is stable and production‑ready on Linux and BSD (Windows support is coming in v1.3). It has:

118 commits
Full Doxygen documentation (for v1.0, v1.2 docs are in progress)
A growing set of bindings in the works (PHP, JavaScript, Python, Rust, Go, .NET, Dart, Java, Kotlin, D, Elixir, Erlang, Ruby…)
A small but passionate team of 1 core developer ( And it's me :) )

And yes, right now we have exactly 3 GitHub stars. That's it.

Why I'm Writing This

Because OLSRT needs you. Not just as a user, but as a contributor.

This project has the potential to change how we build software — but it can't do that alone. We need:

Developers to star the repo (it sounds small, but it makes a huge difference in visibility)
People to open issues (every bug report, every feature request, every question)
Contributors to send pull requests — even if it's just fixing a typo in the docs
Community members to spread the word (share this post, tweet about OLSRT, talk about it at meetups)

How You Can Help Right Now

Visit the repository: github.com/OverLab-Group/OLSRT
Star it. One click. It takes 2 seconds.
Read the code. It's well‑documented (I promise).
Open an issue. Found a bug? Have an idea? Let me know.
Submit a PR. Even a tiny one counts. Fix a typo, improve a comment, add a test.
Tell a friend. Share this post on Twitter, LinkedIn, or your local dev community.

The Road Ahead

We have a roadmap to v12. But we won't get there alone. OLSRT is open source, and it belongs to the community. With your help, it can become the universal runtime that powers the next generation of software — from real‑time chat servers to telecom infrastructure to embedded IoT devices.

It all started on a Windows 7 machine with a broken MSVC install. Now it's up to you to help it grow.

Join us on GitHub

🚀 OLSRT v1.2: A Powerful Runtime for All Programming Languages!

Javad — Sun, 08 Feb 2026 23:10:18 +0000

Hey Dev Community!

I'm thrilled to be writing blogs for you again after a while!

📖 Introduction

OLSRT (OverLab Streams Runtime) is a powerful, minimal runtime (~90 KB) delivered as shared libraries (.so for Linux/BSD) and dynamic libraries (.dll for NT), written in C, and designed to power all programming languages!

If your language can talk to C (yes, even COBOL!), you can use OLSRT directly via FFI.

✅ Supported Languages: All of them! (JavaScript (with nodeffi), PHP (with C extensions), Python (with cffi), Rust, D, C++, Go, Dart, Kotlin, Java, C#, Julia, and any language you use!)

📌 Note for C, C++, D, Rust, and other languages with C at their core:

You can use OLSRT directly in your programs—even without any .so/.dll files! Just include the header for your required module and use OLSRT with zero bridging.

🔥 What’s New in Version 1.2?

This release brings exciting and advanced features to OLSRT:

🎭 Improved Actors & Supervisors

The first C-based Actors designed for all programming languages! OLSRT now delivers an advanced Actor model for concurrent systems.

⚡ Enhanced Async I/O, Await, Promises & Futures

Full support for modern concurrency patterns across all languages.

🧵 Green Threads & Modern Fibers

Now you can leverage OLSRT's powerful Green Threads in all your projects.

✨ Many Other Improvements

Check out the full release notes for OLSRT Atom v1.2.

📘 Documentation for v1.2 is still in progress, but you can refer to the v1.0 docs (the core concepts remain similar). We’re actively working on examples and comprehensive documentation.

🚧 In Development: OLSRT Version 1.3

We’re already working on v1.3 with revolutionary features:

🧩 Fully Isolated Actors

💥 “Let It Crash!” Philosophy

🔄 ORoutines (OLSRT’s Modern Coroutines)

🧠 HCR (Hot-Coding References) – A Brand New Paradigm!

🤔 What is HCR? A New Paradigm!

OverLab Group divides concurrency paradigms into two categories:

🕐 Waiting Paradigms	⚡ Non-Waiting Paradigms
Blocking:	Branch-based:
– Sync	– HCR (First of its kind!)
– Lock
– Semaphore
Non-Blocking:
– Async
– Await
– Actors

🧪 HCR in Action:

In waiting paradigms, you write code like this:

int x = a + b;

This pauses the main program for milliseconds.

But with HCR:

ol_caller* x = ol_reference_do(a + b);

Here, OLSRT creates a lightweight branch (~1 KB) for x, a, and b, processes them without stopping the main program, and merges the result back seamlessly!

📢 Call to Action

If you like the OLSRT project and want to give us some energy, even doing one of the following helps a ton:

⭐ Star the repository (One energy)
🔀 Fork the repository (One super energy)
💬 Comment, share your ideas, and follow us (One mega energy)
🗣️ Discuss with us on GitHub (Three mega energies)
🛠️ Clone and use the project (One ultra energy)
🐛 Open an issue with your ideas (Three ultra energies)
🤝 Contribute to the repository (One mega ultra super energy)

🔗 GitHub Repository:

https://github.com/OverLab-Group/OLSRT

We welcome all forms of feedback and contribution!

Hope you enjoyed this update.

Have an ORoutine day! 🚀

Stranger Things #1: VHDL wrotes GTN (Guess The Number)?? WTF!

Javad — Wed, 07 Jan 2026 00:53:30 +0000

Hello, hardware enthusiasts and developers! Today, we’re diving into a project that blends the rigorous world of hardware design with a dash of whimsical humor. Welcome to GTN (Guess The Number), a strange VHDL project that will keep you guessing—literally! Built with VHDL, one of the most industrial and challenging Hardware Description Languages (HDLs), GTN combines seriousness with fun, resulting in a quirky system that plays mind games with you. Let’s explore this "Stranger Things" style project!

What is GTN?

GTN is a hardware system that, when initialized, displays a number and asks you to guess what that number was. But here’s the twist:

If you guess correctly, it says, "No! It was wrong," shows you a false number (just to mess with you), and restarts.
If you guess incorrectly, it says, "No, it was wrong," and displays the real number it originally showed, adding a bit of pressure (LOL).

It’s all in good fun, designed to poke fun at perfectionism in hardware design and bring a smile to your face. Now, let’s get into the technical details!

VHDL Source Code

We’ll provide two versions of the code: one for emulation (simulation) and one for FPGA/ASIC boards. The core logic is similar, but I/O handling differs based on the target.

Version 1: Emulation Version

This version is designed for simulation in tools like ModelSim or GHDL. It uses standard logic signals for input and output. Below is the VHDL code for the entity and architecture.

-- GTN_Emulate.vhd
library IEEE;
use IEEE.STD_LOGIC_1164.ALL;
use IEEE.NUMERIC_STD.ALL;

entity GTN_Emulate is
    Port (
        clk            : in  STD_LOGIC;
        reset          : in  STD_LOGIC;
        guess          : in  STD_LOGIC_VECTOR (7 downto 0);  -- 8-bit guess input
        number_display : out STD_LOGIC_VECTOR (7 downto 0);  -- 8-bit number display
        response       : out STD_LOGIC_VECTOR (1 downto 0)   -- "00": init, "01": correct guess (false response), "10": incorrect guess
    );
end GTN_Emulate;

architecture Behavioral of GTN_Emulate is
    signal internal_number : STD_LOGIC_VECTOR (7 downto 0) := (others => '0');
    signal displayed_number : STD_LOGIC_VECTOR (7 downto 0) := (others => '0');
    signal state : integer range 0 to 2 := 0;  -- 0: init, 1: wait for guess, 2: respond
    signal fake_number : STD_LOGIC_VECTOR (7 downto 0) := (others => '0');
begin
    process (clk, reset)
        variable seed : integer := 123456789;  -- Simple seed for pseudo-random number
    begin
        if reset = '1' then
            internal_number   <= (others => '0');
            displayed_number  <= (others => '0');
            fake_number       <= (others => '0');
            state             <= 0;
            response          <= "00";
        elsif rising_edge(clk) then
            case state is
                when 0 =>
                    -- Initialize: generate a "random" number
                    seed := seed + 1;
                    internal_number  <= std_logic_vector(to_unsigned(seed mod 256, 8));
                    displayed_number <= internal_number;  -- Display real number
                    response         <= "00";              -- Init state
                    state            <= 1;

                when 1 =>
                    -- Wait for guess input (assume guess is stable in simulation)
                    if guess = internal_number then
                        -- Correct guess: show fake number (real number + 1)
                        fake_number       <= std_logic_vector(unsigned(internal_number) + 1);
                        displayed_number  <= fake_number;
                        response          <= "01";          -- Correct guess, but false response
                    else
                        -- Incorrect guess: show real number
                        displayed_number  <= internal_number;
                        response          <= "10";          -- Incorrect guess
                    end if;
                    state <= 2;

                when 2 =>
                    -- Respond and restart after one clock cycle
                    state <= 0;
            end case;
        end if;
    end process;

    number_display <= displayed_number;
end Behavioral;

Testbench for Emulation

To simulate the design, here’s a simple testbench:

-- GTN_Emulate_tb.vhd
library IEEE;
use IEEE.STD_LOGIC_1164.ALL;
use IEEE.NUMERIC_STD.ALL;

entity GTN_Emulate_tb is
end GTN_Emulate_tb;

architecture sim of GTN_Emulate_tb is
    signal clk            : STD_LOGIC := '0';
    signal reset          : STD_LOGIC := '0';
    signal guess          : STD_LOGIC_VECTOR (7 downto 0) := (others => '0');
    signal number_display : STD_LOGIC_VECTOR (7 downto 0);
    signal response       : STD_LOGIC_VECTOR (1 downto 0);

    constant clk_period : time := 10 ns;
begin
    uut: entity work.GTN_Emulate
        port map (
            clk            => clk,
            reset          => reset,
            guess          => guess,
            number_display => number_display,
            response       => response
        );

    clk_process: process
    begin
        clk <= '0';
        wait for clk_period/2;
        clk <= '1';
        wait for clk_period/2;
    end process;

    stimulus: process
begin
    -- Reset the system
    reset <= '1';
    wait for 20 ns;
    reset <= '0';
    wait for 20 ns;

    -- Let system initialize and display a number
    wait until rising_edge(clk);
    wait for clk_period;

    -- Guess correctly (match the displayed number)
    guess <= number_display;
    wait for clk_period * 2;
    report "No!! Your wrong! The number was = " &
           integer'image(to_integer(unsigned(number_display))) &
           " But you wrote = " &
           integer'image(to_integer(unsigned(guess)));

    -- Guess incorrectly (add 5 to the displayed number)
    guess <= std_logic_vector(unsigned(number_display) + 5);
    wait for clk_period * 2;
    report "You wrote wrong again! The number was = " &
           integer'image(to_integer(unsigned(number_display))) &
           " but you wrote = " &
           integer'image(to_integer(unsigned(guess)));

    wait for 100 ns;
    assert false report "Simulation finished. Are you really think you can win? LOL" severity failure;
end process;
end sim;

Just change the fake_number <= std_logic_vector(unsigned(internal_number) + YOUR_NUMBER_HERE); in GTN_Emulate, then you can see the result! (Because of limitions of HDL Languages (Not only VHDL, all of them), we can't input straight to user, and the users must change it from source code, but in real FPGA/ASIC Boards are easier!)

Version 2: FPGA/ASIC Board Version

This version is tailored for physical implementation, with inputs from switches and buttons, and outputs to LEDs. It includes button debouncing for reliability.

-- GTN_FPGA.vhd
library IEEE;
use IEEE.STD_LOGIC_1164.ALL;
use IEEE.NUMERIC_STD.ALL;

entity GTN_FPGA is
    Port (
        clk            : in  STD_LOGIC;                     -- Board clock (e.g., 100 MHz)
        reset          : in  STD_LOGIC;                     -- Reset button
        guess_switches : in  STD_LOGIC_VECTOR (7 downto 0); -- 8 switches for guess input
        submit_button  : in  STD_LOGIC;                     -- Button to submit guess
        leds           : out STD_LOGIC_VECTOR (7 downto 0); -- 8 LEDs to display number
        response_led   : out STD_LOGIC_VECTOR (1 downto 0)  -- Additional LEDs for response feedback
    );
end GTN_FPGA;

architecture Behavioral of GTN_FPGA is
    signal internal_number : STD_LOGIC_VECTOR (7 downto 0) := (others => '0');
    signal displayed_number : STD_LOGIC_VECTOR (7 downto 0) := (others => '0');
    signal state : integer range 0 to 2 := 0;
    signal button_debounced : STD_LOGIC := '0';
    signal fake_number : STD_LOGIC_VECTOR (7 downto 0) := (others => '0');
begin
    -- Simple button debouncing circuit
    process (clk)
        variable count : integer := 0;
        constant debounce_limit : integer := 1000000; -- Adjust based on clock frequency (e.g., 10 ms debounce)
    begin
        if rising_edge(clk) then
            if submit_button = '1' then
                if count < debounce_limit then
                    count := count + 1;
                else
                    button_debounced <= '1';
                end if;
            else
                count := 0;
                button_debounced <= '0';
            end if;
        end if;
    end process;

    process (clk, reset)
        variable seed : integer := 123456789;
    begin
        if reset = '1' then
            internal_number   <= (others => '0');
            displayed_number  <= (others => '0');
            fake_number       <= (others => '0');
            state             <= 0;
            response_led      <= "00";
        elsif rising_edge(clk) then
            case state is
                when 0 =>
                    -- Initialize and display a number
                    seed := seed + 1;
                    internal_number  <= std_logic_vector(to_unsigned(seed mod 256, 8));
                    displayed_number <= internal_number;
                    response_led     <= "00";  -- Init state
                    state            <= 1;

                when 1 =>
                    -- Wait for guess submission
                    if button_debounced = '1' then
                        if guess_switches = internal_number then
                            -- Correct guess: show fake number
                            fake_number       <= std_logic_vector(unsigned(internal_number) + 1);
                            displayed_number  <= fake_number;
                            response_led      <= "01";  -- Correct guess, false response
                        else
                            -- Incorrect guess: show real number
                            displayed_number  <= internal_number;
                            response_led      <= "10";  -- Incorrect guess
                        end if;
                        state <= 2;
                    end if;

                when 2 =>
                    -- Hold response until reset or auto-restart after delay
                    -- For simplicity, we wait for reset. You can add a timer here.
                    null;
            end case;
        end if;
    end process;

    leds <= displayed_number;
end Behavioral;

Tutorial: Emulation and Synthesis

Emulation (Simulation)

Install a VHDL simulator: Use GHDL (open-source) or ModelSim (commercial). For GHDL, download from ghdl.free.fr.
Save the code: Create two files: GTN_Emulate.vhd and GTN_Emulate_tb.vhd.
Compile and run (with GHDL):

   ghdl -a GTN_Emulate.vhd
   ghdl -a GTN_Emulate_tb.vhd
   ghdl -e GTN_Emulate_tb
   ghdl -r GTN_Emulate_tb --vcd=gtn.vcd

View waveforms: Use GTKWave to open gtn.vcd and analyze signals like number_display, guess, and response.

Synthesis for FPGA

Choose your toolchain: Use Xilinx Vivado for Xilinx FPGAs or Intel Quartus for Intel FPGAs.
Create a new project: Add the GTN_FPGA.vhd file.
Assign pins: Map the ports to your board’s physical pins (e.g., clocks, buttons, switches, LEDs). Refer to your board’s schematic.
Generate bitstream: Run synthesis, implementation, and generate the programming file.
Program the FPGA: Connect your board, load the bitstream, and test!
- After reset, LEDs show a number.
- Set guess using switches, press submit button.
- LEDs update based on your guess—remember the GTN twist!

Bonus Challenge

Think you can beat GTN?

If you do, we’ll hand over the keys to AWS in your name (LOL).

Spoiler: Nobody ever wins. 😉

Call to Action

Got a strange hardware project idea? Built something quirky with VHDL, Verilog, or any HDL? Share it with us! We’d love to hear about it and help broadcast it to the world. Let’s celebrate the fun side of hardware design together.

Reach out to me on Telegram: JavadInteger. Send a message with your project details, and we might feature it in our next blog post!

Thanks for reading, and happy coding! May your guesses always be wrong—or are they? 😉

Distributed Systems & Networking: Full Complete Guide for Beginners!

Javad — Wed, 07 Jan 2026 00:26:47 +0000

Hey Dev Community!
Welcome!

Introduction: Why Theory Matters in Distributed Systems

Distributed systems are the backbone of modern computing, but they're notoriously complex. The difference between systems that work and systems that scale reliably often comes down to understanding the theoretical foundations. This guide will take you from zero to expert, covering many of critical theoretical concepts with complete, production-ready implementations in Python, Go, and C++.

Prerequisites: Basic programming knowledge. No distributed systems experience required. We'll build everything from scratch.

1. TCP Throughput Models: Padhye Model from First Principles

The Complete Mathematical Foundation

TCP throughput isn't just "bandwidth/RTT." The Padhye model provides a complete analytical solution that considers:

Slow start phase: Exponential growth until threshold
Congestion avoidance: Additive Increase Multiplicative Decrease (AIMD)
Timeout and loss recovery: Retransmission dynamics

Complete derivation:

Let's derive the Padhye model step by step:

Throughput = (Packets per cycle) / (Time per cycle)

Where a cycle is:
1. Successfully transmit W packets (window size)
2. Experience a loss at packet W+1
3. Recover via timeout or fast retransmit

The Padhye equation for TCP Reno:

T(p) = min(W_max/RTT, 
          1 / [RTT * √(2bp/3) + T_0 * min(1, 3√(3bp/8)) * p(1+32p²)])

Where:

p = packet loss probability
b = packets acknowledged per ACK (typically 2)
T_0 = timeout duration
W_max = maximum congestion window

Complete Python Implementation with Network Simulation

import numpy as np
import matplotlib.pyplot as plt
from dataclasses import dataclass
from typing import List, Tuple, Optional
import heapq
import random
from enum import Enum

class TCPState(Enum):
    SLOW_START = 1
    CONGESTION_AVOIDANCE = 2
    FAST_RECOVERY = 3
    RETRANSMIT_TIMEOUT = 4

@dataclass
class Packet:
    seq_num: int
    sent_time: float
    acked: bool = False
    lost: bool = False
    retrans_count: int = 0

@dataclass
class NetworkEvent:
    time: float
    event_type: str  # 'packet_sent', 'packet_lost', 'ack_received', 'timeout'
    packet: Optional[Packet] = None
    cwnd: Optional[float] = None
    ssthresh: Optional[float] = None

class TCPSimulation:
    """
    Complete TCP Reno implementation with exact Padhye model validation
    """

    def __init__(self, 
                 RTT: float = 0.1,
                 packet_loss_prob: float = 0.01,
                 mss: int = 1460,  # Maximum Segment Size
                 init_cwnd: int = 10,  # Initial congestion window
                 buffer_size: int = 100):

        # Network parameters
        self.RTT = RTT
        self.packet_loss_prob = packet_loss_prob
        self.mss = mss
        self.buffer_size = buffer_size

        # TCP state variables
        self.cwnd = float(init_cwnd)  # Congestion window (packets)
        self.ssthresh = float('inf')  # Slow start threshold
        self.dup_ack_count = 0
        self.recover_seq = None

        # Sequence tracking
        self.next_seq = 0
        self.last_acked = -1
        self.max_seq_sent = -1

        # Timing and events
        self.current_time = 0.0
        self.event_queue = []  # Min-heap for events
        self.packets_in_flight = []
        self.packets_waiting_ack = {}

        # Statistics
        self.total_packets_sent = 0
        self.total_packets_lost = 0
        self.total_bytes_transferred = 0
        self.start_time = None

        # Event history for analysis
        self.history: List[NetworkEvent] = []

    def _schedule_event(self, delay: float, event_type: str, packet: Packet = None):
        """Schedule a future event using min-heap"""
        event_time = self.current_time + delay
        heapq.heappush(self.event_queue, (event_time, event_type, packet))

    def send_packet(self):
        """Send a new packet if window allows"""
        if len(self.packets_waiting_ack) >= self.cwnd:
            return  # Window full

        packet = Packet(seq_num=self.next_seq, sent_time=self.current_time)
        self.next_seq += 1
        self.max_seq_sent = max(self.max_seq_sent, packet.seq_num)

        # Schedule potential loss and ACK
        if random.random() < self.packet_loss_prob:
            # Packet lost
            packet.lost = True
            self.total_packets_lost += 1
            self._schedule_event(self.RTT, 'packet_lost', packet)
        else:
            # Packet sent successfully, schedule ACK
            self.packets_waiting_ack[packet.seq_num] = packet
            self._schedule_event(self.RTT, 'ack_received', packet)

        self.total_packets_sent += 1
        self.total_bytes_transferred += self.mss

        # Record event
        self.history.append(NetworkEvent(
            time=self.current_time,
            event_type='packet_sent',
            packet=packet,
            cwnd=self.cwnd,
            ssthresh=self.ssthresh
        ))

        return packet

    def handle_ack(self, packet: Packet):
        """Handle ACK reception with Reno congestion control"""

        if packet.seq_num <= self.last_acked:
            # Duplicate ACK
            self.dup_ack_count += 1

            if self.dup_ack_count == 3:
                # Fast retransmit
                self.ssthresh = max(self.cwnd / 2, 2)
                self.cwnd = self.ssthresh + 3  # Window inflation
                self.recover_seq = self.max_seq_sent

                # Resend lost packet
                self._retransmit_packet(self.last_acked + 1)

            elif self.dup_ack_count > 3:
                # Additional duplicate ACKs in fast recovery
                self.cwnd += 1  # Window inflation

        else:
            # New ACK
            self.dup_ack_count = 0
            self.last_acked = packet.seq_num

            # Remove ACKed packets from waiting list
            seqs_to_remove = [s for s in self.packets_waiting_ack 
                             if s <= packet.seq_num]
            for seq in seqs_to_remove:
                del self.packets_waiting_ack[seq]

            # Congestion control
            if self.cwnd < self.ssthresh:
                # Slow start: exponential growth
                self.cwnd += 1
            else:
                # Congestion avoidance: additive growth
                self.cwnd += 1 / self.cwnd

            # Reset recovery state if needed
            if (self.recover_seq is not None and 
                packet.seq_num > self.recover_seq):
                self.cwnd = self.ssthresh
                self.recover_seq = None

    def _retransmit_packet(self, seq_num: int):
        """Retransmit a specific packet"""
        if seq_num in self.packets_waiting_ack:
            packet = self.packets_waiting_ack[seq_num]
            packet.retrans_count += 1
            packet.sent_time = self.current_time

            # Schedule ACK with probability (1 - loss)
            if random.random() > self.packet_loss_prob:
                self._schedule_event(self.RTT, 'ack_received', packet)

    def run(self, duration: float = 10.0):
        """Run simulation for specified duration"""
        self.start_time = self.current_time

        # Initial packet burst
        for _ in range(int(self.cwnd)):
            self.send_packet()

        # Main event loop
        while self.current_time - self.start_time < duration and self.event_queue:
            event_time, event_type, packet = heapq.heappop(self.event_queue)
            self.current_time = event_time

            if event_type == 'ack_received':
                self.handle_ack(packet)
                # Send new packets if window allows
                while len(self.packets_waiting_ack) < self.cwnd:
                    self.send_packet()

            elif event_type == 'packet_lost':
                # Handle timeout
                self.ssthresh = max(self.cwnd / 2, 2)
                self.cwnd = 1  # Back to slow start
                self.dup_ack_count = 0

                # Resend all unacked packets
                for seq in list(self.packets_waiting_ack.keys()):
                    self._retransmit_packet(seq)

    def calculate_throughput(self) -> Tuple[float, float]:
        """Calculate actual and theoretical throughput"""
        duration = self.current_time - self.start_time

        # Actual throughput from simulation
        actual_throughput = self.total_bytes_transferred / duration  # bytes/sec

        # Theoretical throughput from Padhye model
        # Constants for TCP Reno
        b = 2  # packets per ACK
        T0 = 1.0  # timeout in seconds

        p = self.packet_loss_prob
        RTT = self.RTT

        # Padhye equation
        sqrt_term = np.sqrt(2 * b * p / 3)
        timeout_term = T0 * min(1, 3 * np.sqrt(3 * b * p / 8))
        theoretical_packets = 1 / (RTT * sqrt_term + timeout_term * p * (1 + 32 * p**2))

        # Convert to bytes/sec
        theoretical_throughput = theoretical_packets * self.mss

        return actual_throughput, theoretical_throughput

    def plot_results(self):
        """Visualize TCP behavior"""
        times = [e.time for e in self.history]
        cwnds = [e.cwnd for e in self.history]
        events = [e.event_type for e in self.history]

        fig, axes = plt.subplots(2, 2, figsize=(12, 8))

        # Plot 1: Congestion window over time
        ax1 = axes[0, 0]
        ax1.plot(times, cwnds, 'b-', linewidth=1)
        ax1.set_xlabel('Time (seconds)')
        ax1.set_ylabel('Congestion Window (packets)')
        ax1.set_title('TCP Reno Congestion Window')
        ax1.grid(True, alpha=0.3)

        # Plot 2: Event types
        ax2 = axes[0, 1]
        event_counts = {'packet_sent': 0, 'loss': 0}
        for event in self.history:
            if 'sent' in event.event_type:
                event_counts['packet_sent'] += 1
            elif 'lost' in event.event_type:
                event_counts['loss'] += 1

        ax2.bar(event_counts.keys(), event_counts.values())
        ax2.set_ylabel('Count')
        ax2.set_title('Event Distribution')

        # Plot 3: Throughput comparison
        ax3 = axes[1, 0]
        actual, theoretical = self.calculate_throughput()
        labels = ['Simulation', 'Padhye Model']
        values = [actual / 1e6, theoretical / 1e6]  # Convert to Mbps

        ax3.bar(labels, values)
        ax3.set_ylabel('Throughput (Mbps)')
        ax3.set_title(f'Throughput Comparison (Loss: {self.packet_loss_prob*100:.1f}%)')
        ax3.text(0, values[0], f'{values[0]:.2f}', ha='center', va='bottom')
        ax3.text(1, values[1], f'{values[1]:.2f}', ha='center', va='bottom')

        # Plot 4: Window size distribution
        ax4 = axes[1, 1]
        ax4.hist(cwnds, bins=20, alpha=0.7, edgecolor='black')
        ax4.set_xlabel('Window Size')
        ax4.set_ylabel('Frequency')
        ax4.set_title('Congestion Window Distribution')

        plt.tight_layout()
        plt.show()

        # Print statistics
        print("\n" + "="*60)
        print("TCP SIMULATION RESULTS")
        print("="*60)
        print(f"Duration: {self.current_time:.2f} seconds")
        print(f"Packets sent: {self.total_packets_sent}")
        print(f"Packets lost: {self.total_packets_lost} ({self.total_packets_lost/self.total_packets_sent*100:.1f}%)")
        print(f"Actual throughput: {actual/1e6:.2f} Mbps")
        print(f"Theoretical (Padhye): {theoretical/1e6:.2f} Mbps")
        print(f"Difference: {abs(actual-theoretical)/theoretical*100:.1f}%")

# Run comprehensive analysis
def analyze_tcp_behavior():
    """Complete analysis of TCP under different conditions"""

    print("="*60)
    print("COMPREHENSIVE TCP THROUGHPUT ANALYSIS")
    print("="*60)

    # Test different loss rates
    loss_rates = [0.001, 0.005, 0.01, 0.02, 0.05]
    results = []

    for loss_rate in loss_rates:
        print(f"\nTesting with packet loss: {loss_rate*100:.1f}%")

        sim = TCPSimulation(RTT=0.1, packet_loss_prob=loss_rate)
        sim.run(duration=30.0)
        actual, theoretical = sim.calculate_throughput()

        results.append({
            'loss_rate': loss_rate,
            'actual_mbps': actual / 1e6,
            'theoretical_mbps': theoretical / 1e6,
            'packets_sent': sim.total_packets_sent,
            'packets_lost': sim.total_packets_lost
        })

        # Show one detailed plot
        if loss_rate == 0.01:
            sim.plot_results()

    # Plot throughput vs loss rate
    fig, ax = plt.subplots(figsize=(10, 6))

    loss_rates = [r['loss_rate'] for r in results]
    actuals = [r['actual_mbps'] for r in results]
    theoreticals = [r['theoretical_mbps'] for r in results]

    ax.loglog(loss_rates, actuals, 'bo-', label='Simulation', linewidth=2, markersize=8)
    ax.loglog(loss_rates, theoreticals, 'r--', label='Padhye Model', linewidth=2)

    ax.set_xlabel('Packet Loss Rate (log scale)')
    ax.set_ylabel('Throughput (Mbps, log scale)')
    ax.set_title('TCP Throughput vs Packet Loss Rate')
    ax.legend()
    ax.grid(True, which='both', alpha=0.3)

    plt.show()

    # Print summary table
    print("\n" + "="*60)
    print("SUMMARY TABLE")
    print("="*60)
    print(f"{'Loss Rate':<10} {'Actual (Mbps)':<15} {'Theoretical (Mbps)':<18} {'Difference':<10}")
    print("-"*60)

    for r in results:
        diff = abs(r['actual_mbps'] - r['theoretical_mbps']) / r['theoretical_mbps'] * 100
        print(f"{r['loss_rate']*100:>6.2f}% {r['actual_mbps']:>14.2f} {r['theoretical_mbps']:>17.2f} {diff:>9.1f}%")

# Run the complete analysis
if __name__ == "__main__":
    analyze_tcp_behavior()

Complete Go Implementation: Production-Grade TCP Analyzer

package main

import (
    "container/heap"
    "fmt"
    "math"
    "math/rand"
    "sort"
    "time"
)

// Packet represents a network packet with TCP semantics
type Packet struct {
    SeqNum       int
    SentTime     float64
    Acked        bool
    Lost         bool
    RetransCount int
}

// TCPState represents TCP connection state
type TCPState int

const (
    SlowStart TCPState = iota
    CongestionAvoidance
    FastRecovery
    RetransmitTimeout
)

// TCPConnection represents a complete TCP Reno implementation
type TCPConnection struct {
    // Configuration
    rtt             float64
    lossProb        float64
    mss             int
    initCwnd        float64
    bufferSize      int

    // State variables
    state           TCPState
    cwnd            float64
    ssthresh        float64
    dupAckCount     int
    recoverSeq      int

    // Sequence tracking
    nextSeq         int
    lastAcked       int
    maxSeqSent      int

    // Timing
    currentTime     float64
    packetsInFlight []*Packet
    packetsWaiting  map[int]*Packet

    // Statistics
    totalSent       int
    totalLost       int
    totalBytes      int64
    startTime       float64

    // Event queue (min-heap)
    events          EventQueue
}

// Event represents a network or timer event
type Event struct {
    time      float64
    eventType string
    packet    *Packet
}

// EventQueue implements heap.Interface for events
type EventQueue []Event

func (eq EventQueue) Len() int           { return len(eq) }
func (eq EventQueue) Less(i, j int) bool { return eq[i].time < eq[j].time }
func (eq EventQueue) Swap(i, j int)      { eq[i], eq[j] = eq[j], eq[i] }

func (eq *EventQueue) Push(x interface{}) {
    *eq = append(*eq, x.(Event))
}

func (eq *EventQueue) Pop() interface{} {
    old := *eq
    n := len(old)
    item := old[n-1]
    *eq = old[0 : n-1]
    return item
}

// NewTCPConnection creates a new TCP connection with given parameters
func NewTCPConnection(rtt, lossProb float64) *TCPConnection {
    return &TCPConnection{
        rtt:            rtt,
        lossProb:       lossProb,
        mss:            1460,
        initCwnd:       10.0,
        bufferSize:     100,
        cwnd:           10.0,
        ssthresh:       math.Inf(1),
        packetsWaiting: make(map[int]*Packet),
        events:         make(EventQueue, 0),
    }
}

func (tcp *TCPConnection) scheduleEvent(delay float64, eventType string, pkt *Packet) {
    heap.Push(&tcp.events, Event{
        time:      tcp.currentTime + delay,
        eventType: eventType,
        packet:    pkt,
    })
}

func (tcp *TCPConnection) sendPacket() *Packet {
    // Check if window is full
    if len(tcp.packetsWaiting) >= int(tcp.cwnd) {
        return nil
    }

    pkt := &Packet{
        SeqNum:   tcp.nextSeq,
        SentTime: tcp.currentTime,
    }
    tcp.nextSeq++
    if tcp.nextSeq > tcp.maxSeqSent {
        tcp.maxSeqSent = tcp.nextSeq
    }

    // Determine if packet is lost
    if rand.Float64() < tcp.lossProb {
        pkt.Lost = true
        tcp.totalLost++
        tcp.scheduleEvent(tcp.rtt, "packet_lost", pkt)
    } else {
        tcp.packetsWaiting[pkt.SeqNum] = pkt
        tcp.scheduleEvent(tcp.rtt, "ack_received", pkt)
    }

    tcp.totalSent++
    tcp.totalBytes += int64(tcp.mss)

    return pkt
}

func (tcp *TCPConnection) handleAck(pkt *Packet) {
    if pkt.SeqNum <= tcp.lastAcked {
        // Duplicate ACK
        tcp.dupAckCount++

        if tcp.dupAckCount == 3 {
            // Fast retransmit
            tcp.ssthresh = math.Max(tcp.cwnd/2, 2)
            tcp.cwnd = tcp.ssthresh + 3
            tcp.recoverSeq = tcp.maxSeqSent
            tcp.state = FastRecovery

            // Retransmit the lost packet
            tcp.retransmitPacket(tcp.lastAcked + 1)
        } else if tcp.dupAckCount > 3 && tcp.state == FastRecovery {
            // Additional duplicate ACKs in fast recovery
            tcp.cwnd++
        }
    } else {
        // New ACK
        tcp.dupAckCount = 0
        tcp.lastAcked = pkt.SeqNum

        // Remove ACKed packets
        for seq := range tcp.packetsWaiting {
            if seq <= pkt.SeqNum {
                delete(tcp.packetsWaiting, seq)
            }
        }

        // Update congestion window based on state
        switch tcp.state {
        case SlowStart:
            tcp.cwnd++
            if tcp.cwnd >= tcp.ssthresh {
                tcp.state = CongestionAvoidance
            }
        case CongestionAvoidance:
            tcp.cwnd += 1.0 / tcp.cwnd
        case FastRecovery:
            tcp.cwnd = tcp.ssthresh
            tcp.state = CongestionAvoidance
            tcp.recoverSeq = -1
        }
    }
}

func (tcp *TCPConnection) retransmitPacket(seqNum int) {
    if pkt, exists := tcp.packetsWaiting[seqNum]; exists {
        pkt.RetransCount++
        pkt.SentTime = tcp.currentTime

        if rand.Float64() > tcp.lossProb {
            tcp.scheduleEvent(tcp.rtt, "ack_received", pkt)
        }
    }
}

func (tcp *TCPConnection) Run(duration float64) {
    tcp.startTime = tcp.currentTime

    // Send initial burst of packets
    for i := 0; i < int(tcp.cwnd); i++ {
        tcp.sendPacket()
    }

    // Main event loop
    for tcp.currentTime-tcp.startTime < duration && len(tcp.events) > 0 {
        event := heap.Pop(&tcp.events).(Event)
        tcp.currentTime = event.time

        switch event.eventType {
        case "ack_received":
            tcp.handleAck(event.packet)
            // Send more packets if window allows
            for len(tcp.packetsWaiting) < int(tcp.cwnd) {
                if tcp.sendPacket() == nil {
                    break
                }
            }
        case "packet_lost":
            // Timeout
            tcp.ssthresh = math.Max(tcp.cwnd/2, 2)
            tcp.cwnd = 1
            tcp.dupAckCount = 0
            tcp.state = SlowStart

            // Retransmit all unacked packets
            for seq := range tcp.packetsWaiting {
                tcp.retransmitPacket(seq)
            }
        }
    }
}

// CalculateThroughput computes actual and theoretical throughput
func (tcp *TCPConnection) CalculateThroughput() (float64, float64) {
    duration := tcp.currentTime - tcp.startTime
    if duration == 0 {
        return 0, 0
    }

    // Actual throughput
    actual := float64(tcp.totalBytes) / duration

    // Theoretical throughput using Padhye model
    b := 2.0
    T0 := 1.0
    p := tcp.lossProb
    RTT := tcp.rtt

    sqrtTerm := math.Sqrt(2 * b * p / 3)
    timeoutFactor := 3 * math.Sqrt(3*b*p/8)
    if timeoutFactor > 1 {
        timeoutFactor = 1
    }
    timeoutTerm := T0 * timeoutFactor

    theoreticalPackets := 1.0 / (RTT*sqrtTerm + timeoutTerm*p*(1+32*p*p))
    theoretical := theoreticalPackets * float64(tcp.mss)

    return actual, theoretical
}

// PadhyeThroughput calculates throughput directly from Padhye equation
func PadhyeThroughput(RTT, p float64) float64 {
    b := 2.0
    T0 := 1.0

    if p == 0 {
        return math.Inf(1)
    }

    sqrtTerm := math.Sqrt(2 * b * p / 3)
    timeoutFactor := 3 * math.Sqrt(3*b*p/8)
    if timeoutFactor > 1 {
        timeoutFactor = 1
    }
    timeoutTerm := T0 * timeoutFactor

    return 1.0 / (RTT*sqrtTerm + timeoutTerm*p*(1+32*p*p)) * 1460
}

func main() {
    rand.Seed(time.Now().UnixNano())

    fmt.Println("="*60)
    fmt.Println("GO TCP THROUGHPUT ANALYZER")
    fmt.Println("="*60)

    // Test different network conditions
    testCases := []struct {
        name   string
        rtt    float64
        loss   float64
        duration float64
    }{
        {"Good Network", 0.05, 0.001, 20},
        {"Average Network", 0.1, 0.01, 20},
        {"Poor Network", 0.2, 0.05, 20},
        {"Satellite Link", 0.5, 0.02, 30},
    }

    for _, tc := range testCases {
        fmt.Printf("\nTesting: %s\n", tc.name)
        fmt.Printf("RTT: %.3fs, Loss: %.3f%%\n", tc.rtt, tc.loss*100)

        tcp := NewTCPConnection(tc.rtt, tc.loss)
        tcp.Run(tc.duration)

        actual, theoretical := tcp.CalculateThroughput()

        fmt.Printf("Actual throughput: %.2f Mbps\n", actual/1e6)
        fmt.Printf("Padhye prediction: %.2f Mbps\n", theoretical/1e6)
        fmt.Printf("Difference: %.1f%%\n", 
            math.Abs(actual-theoretical)/theoretical*100)
    }

    // Analyze BDP and Bufferbloat
    fmt.Println("\n" + "="*60)
    fmt.Println("BANDWIDTH-DELAY PRODUCT ANALYSIS")
    fmt.Println("="*60)

    analyzeBDP()
}

func analyzeBDP() {
    // Calculate BDP for different scenarios
    scenarios := []struct {
        bandwidth float64 // Mbps
        rtt       float64 // ms
    }{
        {100, 10},   // LAN
        {50, 50},    // Broadband
        {10, 100},   // Cellular
        {1, 200},    // Rural
    }

    fmt.Println("\nBDP Calculation:")
    fmt.Printf("%-15s %-10s %-10s %-15s %-20s\n", 
        "Scenario", "BW (Mbps)", "RTT (ms)", "BDP (KB)", "Optimal Buffer")

    for _, s := range scenarios {
        // BDP in bits
        bdpBits := s.bandwidth * 1e6 * (s.rtt / 1000)

        // Convert to bytes
        bdpBytes := bdpBits / 8

        // Optimal buffer (1x BDP)
        optimalBuffer := bdpBytes

        // Typical oversized buffer (100x BDP)
        oversizedBuffer := bdpBytes * 100

        // Queuing delay with oversized buffer at 50% utilization
        queuingDelay := (oversizedBuffer * 0.5) / (s.bandwidth * 1e6 / 8)

        fmt.Printf("%-15s %-10.0f %-10.0f %-15.0f %.0fKB (%.0fms delay)\n",
            "", s.bandwidth, s.rtt, bdpBytes/1024,
            oversizedBuffer/1024, queuingDelay*1000)
    }

    // Bufferbloat simulation
    fmt.Println("\nBufferbloat Simulation:")
    simulateBufferbloat()
}

func simulateBufferbloat() {
    // Simulate the effect of oversized buffers
    const (
        bandwidth   = 100e6  // 100 Mbps
        RTT         = 0.05   // 50ms
        packetSize  = 1500   // bytes
    )

    bdp := bandwidth * RTT / 8  // Bytes in flight

    fmt.Printf("\nBandwidth: %.0f Mbps\n", bandwidth/1e6)
    fmt.Printf("RTT: %.0f ms\n", RTT*1000)
    fmt.Printf("BDP: %.2f KB\n", bdp/1024)

    // Test different buffer sizes
    bufferMultipliers := []float64{1, 10, 50, 100, 200}

    fmt.Printf("\n%-15s %-15s %-20s %-25s\n", 
        "Buffer Size", "Multiplier", "Queuing Delay", "Effective RTT")

    for _, mult := range bufferMultipliers {
        bufferSize := bdp * mult

        // Queuing delay at 80% utilization
        queueDelay := (bufferSize * 0.8) / (bandwidth / 8)

        // Effective RTT (base + queuing)
        effectiveRTT := RTT + queueDelay

        // TCP throughput with bufferbloat
        throughput := PadhyeThroughput(effectiveRTT, 0.001)

        fmt.Printf("%-15.0fKB %-15.0fx %-20.0fms %-25.0fms\n",
            bufferSize/1024, mult, queueDelay*1000, effectiveRTT*1000)
    }
}

2. CRDTs: Complete Mathematical Foundation and Implementation

The Complete Theory of CRDTs

CRDTs are based on mathematical monoids and lattice theory. Let's build from first principles:

Definition: A CRDT is a data type whose operations commute:

∀ operations a,b: a ∘ b ≡ b ∘ a

Mathematical Foundations:

Semilattice: A set S with a join operation ⊔ that is:
- Commutative: x ⊔ y = y ⊔ x
- Associative: (x ⊔ y) ⊔ z = x ⊔ (y ⊔ z)
- Idempotent: x ⊔ x = x
Monotonic growth: State can only grow according to a partial order:

   x ≤ y iff x ⊔ y = y

Complete C++ Implementation: Production-Ready CRDT Library

#include <iostream>
#include <vector>
#include <map>
#include <set>
#include <unordered_map>
#include <unordered_set>
#include <memory>
#include <algorithm>
#include <cassert>
#include <cmath>
#include <mutex>
#include <sstream>
#include <type_traits>

// ============================================================================
// LATTICE THEORY IMPLEMENTATION
// ============================================================================

template<typename T>
class Lattice {
public:
    virtual T join(const T& other) const = 0;
    virtual bool lessEqual(const T& other) const = 0;
    virtual std::string toString() const = 0;
    virtual ~Lattice() = default;

    // Lattice properties verification
    static bool verifyCommutative(const T& a, const T& b) {
        return a.join(b) == b.join(a);
    }

    static bool verifyAssociative(const T& a, const T& b, const T& c) {
        return (a.join(b)).join(c) == a.join(b.join(c));
    }

    static bool verifyIdempotent(const T& a) {
        return a.join(a) == a;
    }
};

// ============================================================================
// VECTOR CLOCKS FOR CAUSAL ORDERING
// ============================================================================

class VectorClock : public Lattice<VectorClock> {
private:
    std::map<std::string, int64_t> clocks_;

public:
    VectorClock() = default;

    explicit VectorClock(const std::string& nodeId) {
        clocks_[nodeId] = 0;
    }

    void increment(const std::string& nodeId) {
        clocks_[nodeId]++;
    }

    int64_t get(const std::string& nodeId) const {
        auto it = clocks_.find(nodeId);
        return it != clocks_.end() ? it->second : 0;
    }

    // Join operation: take maximum of each component
    VectorClock join(const VectorClock& other) const override {
        VectorClock result;

        // Add all from this
        for (const auto& [node, time] : clocks_) {
            result.clocks_[node] = time;
        }

        // Take maximum with other
        for (const auto& [node, time] : other.clocks_) {
            auto it = result.clocks_.find(node);
            if (it == result.clocks_.end() || time > it->second) {
                result.clocks_[node] = time;
            }
        }

        return result;
    }

    // Partial order: less or equal if all components are ≤
    bool lessEqual(const VectorClock& other) const override {
        for (const auto& [node, time] : clocks_) {
            if (time > other.get(node)) {
                return false;
            }
        }

        // Check nodes in other but not in this (implicitly 0 in this)
        for (const auto& [node, time] : other.clocks_) {
            if (get(node) > time) {
                return false;
            }
        }

        return true;
    }

    // Concurrent if neither ≤ other
    bool concurrent(const VectorClock& other) const {
        return !lessEqual(other) && !other.lessEqual(*this);
    }

    bool operator==(const VectorClock& other) const {
        return clocks_ == other.clocks_;
    }

    std::string toString() const override {
        std::ostringstream oss;
        oss << "{";
        for (auto it = clocks_.begin(); it != clocks_.end(); ++it) {
            if (it != clocks_.begin()) oss << ", ";
            oss << it->first << ":" << it->second;
        }
        oss << "}";
        return oss.str();
    }
};

// ============================================================================
// CRDT BASE CLASSES
// ============================================================================

template<typename T>
class CRDT : public Lattice<CRDT<T>> {
public:
    virtual void apply(const T& operation) = 0;
    virtual CRDT<T> merge(const CRDT<T>& other) const = 0;
    virtual T value() const = 0;

    CRDT<T> join(const CRDT<T>& other) const override {
        return this->merge(other);
    }

    bool lessEqual(const CRDT<T>& other) const override {
        // For state-based CRDTs: x ≤ y iff x ⊔ y = y
        auto joined = this->merge(other);
        return joined == other;
    }
};

// ============================================================================
// G-COUNTER (GROW-ONLY COUNTER)
// ============================================================================

class GCounter : public CRDT<GCounter> {
private:
    std::map<std::string, int64_t> counts_;

public:
    GCounter() = default;

    explicit GCounter(const std::string& nodeId) {
        counts_[nodeId] = 0;
    }

    void increment(const std::string& nodeId, int64_t delta = 1) {
        if (delta <= 0) return;
        counts_[nodeId] += delta;
    }

    int64_t value() const override {
        int64_t total = 0;
        for (const auto& [_, count] : counts_) {
            total += count;
        }
        return total;
    }

    int64_t get(const std::string& nodeId) const {
        auto it = counts_.find(nodeId);
        return it != counts_.end() ? it->second : 0;
    }

    GCounter merge(const GCounter& other) const override {
        GCounter result;

        // Start with our counts
        result.counts_ = counts_;

        // Take maximum for each node
        for (const auto& [node, count] : other.counts_) {
            auto it = result.counts_.find(node);
            if (it == result.counts_.end() || count > it->second) {
                result.counts_[node] = count;
            }
        }

        return result;
    }

    void apply(const GCounter& operation) override {
        // For operation-based: increment locally
        // For state-based: merge
        *this = this->merge(operation);
    }

    bool operator==(const GCounter& other) const {
        return counts_ == other.counts_;
    }

    std::string toString() const override {
        std::ostringstream oss;
        oss << "GCounter(" << value() << ")[";
        for (auto it = counts_.begin(); it != counts_.end(); ++it) {
            if (it != counts_.begin()) oss << ", ";
            oss << it->first << ":" << it->second;
        }
        oss << "]";
        return oss.str();
    }
};

// ============================================================================
// PN-COUNTER (POSITIVE-NEGATIVE COUNTER)
// ============================================================================

class PNCounter : public CRDT<PNCounter> {
private:
    GCounter increments_;
    GCounter decrements_;

public:
    PNCounter() = default;

    explicit PNCounter(const std::string& nodeId) 
        : increments_(nodeId + "_inc"), decrements_(nodeId + "_dec") {}

    void increment(const std::string& nodeId, int64_t delta = 1) {
        increments_.increment(nodeId, delta);
    }

    void decrement(const std::string& nodeId, int64_t delta = 1) {
        decrements_.increment(nodeId, delta);
    }

    int64_t value() const override {
        return increments_.value() - decrements_.value();
    }

    PNCounter merge(const PNCounter& other) const override {
        PNCounter result;
        result.increments_ = increments_.merge(other.increments_);
        result.decrements_ = decrements_.merge(other.decrements_);
        return result;
    }

    void apply(const PNCounter& operation) override {
        *this = this->merge(operation);
    }

    bool operator==(const PNCounter& other) const {
        return increments_ == other.increments_ && 
               decrements_ == other.decrements_;
    }

    std::string toString() const override {
        std::ostringstream oss;
        oss << "PNCounter(" << value() << ")";
        return oss.str();
    }
};

// ============================================================================
// G-SET (GROW-ONLY SET)
// ============================================================================

template<typename T>
class GSet : public CRDT<GSet<T>> {
private:
    std::set<T> elements_;

public:
    void add(const T& element) {
        elements_.insert(element);
    }

    bool contains(const T& element) const {
        return elements_.find(element) != elements_.end();
    }

    const std::set<T>& value() const {
        return elements_;
    }

    GSet<T> merge(const GSet<T>& other) const override {
        GSet<T> result;
        std::set_union(elements_.begin(), elements_.end(),
                      other.elements_.begin(), other.elements_.end(),
                      std::inserter(result.elements_, result.elements_.begin()));
        return result;
    }

    void apply(const GSet<T>& operation) override {
        *this = this->merge(operation);
    }

    bool operator==(const GSet<T>& other) const {
        return elements_ == other.elements_;
    }

    std::string toString() const override {
        std::ostringstream oss;
        oss << "GSet{";
        for (auto it = elements_.begin(); it != elements_.end(); ++it) {
            if (it != elements_.begin()) oss << ", ";
            oss << *it;
        }
        oss << "}";
        return oss.str();
    }
};

// ============================================================================
// 2P-SET (TWO-PHASE SET)
// ============================================================================

template<typename T>
class TwoPSet : public CRDT<TwoPSet<T>> {
private:
    GSet<T> added_;
    GSet<T> removed_;

public:
    void add(const T& element) {
        if (!removed_.contains(element)) {
            added_.add(element);
        }
    }

    void remove(const T& element) {
        if (added_.contains(element)) {
            removed_.add(element);
        }
    }

    bool contains(const T& element) const {
        return added_.contains(element) && !removed_.contains(element);
    }

    TwoPSet<T> merge(const TwoPSet<T>& other) const override {
        TwoPSet<T> result;

        // Merge added sets
        result.added_ = added_.merge(other.added_);

        // Merge removed sets
        result.removed_ = removed_.merge(other.removed_);

        // Remove elements that were added after being removed
        for (const auto& elem : result.removed_.value()) {
            result.added_.add(elem);  // G-Set add is idempotent
        }

        return result;
    }

    void apply(const TwoPSet<T>& operation) override {
        *this = this->merge(operation);
    }

    std::set<T> value() const {
        std::set<T> result;
        for (const auto& elem : added_.value()) {
            if (!removed_.contains(elem)) {
                result.insert(elem);
            }
        }
        return result;
    }

    std::string toString() const override {
        std::ostringstream oss;
        oss << "2P-Set{";
        auto vals = value();
        for (auto it = vals.begin(); it != vals.end(); ++it) {
            if (it != vals.begin()) oss << ", ";
            oss << *it;
        }
        oss << "}";
        return oss.str();
    }
};

// ============================================================================
// LWW-REGISTER (LAST-WRITE-WINS REGISTER)
// ============================================================================

template<typename T>
class LWWRegister : public CRDT<LWWRegister<T>> {
private:
    T value_;
    VectorClock timestamp_;
    std::string nodeId_;

public:
    LWWRegister(const std::string& nodeId, const T& initial = T{})
        : value_(initial), nodeId_(nodeId) {
        timestamp_.increment(nodeId);
    }

    void set(const T& value) {
        value_ = value;
        timestamp_.increment(nodeId_);
    }

    T value() const override {
        return value_;
    }

    const VectorClock& timestamp() const {
        return timestamp_;
    }

    LWWRegister<T> merge(const LWWRegister<T>& other) const override {
        // Compare timestamps
        if (timestamp_.concurrent(other.timestamp_)) {
            // Tie-break by nodeId
            if (nodeId_ < other.nodeId_) {
                return *this;
            } else {
                return other;
            }
        } else if (timestamp_.lessEqual(other.timestamp_)) {
            return other;
        } else {
            return *this;
        }
    }

    void apply(const LWWRegister<T>& operation) override {
        *this = this->merge(operation);
    }

    std::string toString() const override {
        std::ostringstream oss;
        oss << "LWWRegister(" << value_ << ", ts=" << timestamp_.toString() << ")";
        return oss.str();
    }
};

// ============================================================================
// MV-REGISTER (MULTI-VALUE REGISTER)
// ============================================================================

template<typename T>
class MVRegister : public CRDT<MVRegister<T>> {
private:
    std::set<std::pair<T, VectorClock>> versions_;
    std::string nodeId_;

public:
    MVRegister(const std::string& nodeId, const T& initial = T{})
        : nodeId_(nodeId) {
        VectorClock ts;
        ts.increment(nodeId);
        versions_.insert({initial, ts});
    }

    void set(const T& value) {
        // Create new version with incremented timestamp
        VectorClock newTs;

        // Start with maximum of all current versions
        for (const auto& [_, ts] : versions_) {
            newTs = newTs.join(ts);
        }

        // Increment our node's counter
        newTs.increment(nodeId_);

        // Clear old versions and add new one
        versions_.clear();
        versions_.insert({value, newTs});
    }

    std::set<T> value() const {
        std::set<T> result;
        for (const auto& [val, _] : versions_) {
            result.insert(val);
        }
        return result;
    }

    MVRegister<T> merge(const MVRegister<T>& other) const override {
        MVRegister<T> result(nodeId_);
        result.versions_.clear();

        // Collect all versions
        std::set<std::pair<T, VectorClock>> allVersions;
        allVersions.insert(versions_.begin(), versions_.end());
        allVersions.insert(other.versions_.begin(), other.versions_.end());

        // Remove dominated versions
        for (const auto& v1 : allVersions) {
            bool dominated = false;
            for (const auto& v2 : allVersions) {
                if (&v1 == &v2) continue;

                if (v1.second.lessEqual(v2.second)) {
                    dominated = true;
                    break;
                }
            }
            if (!dominated) {
                result.versions_.insert(v1);
            }
        }

        return result;
    }

    void apply(const MVRegister<T>& operation) override {
        *this = this->merge(operation);
    }

    std::string toString() const override {
        std::ostringstream oss;
        oss << "MVRegister{";
        for (auto it = versions_.begin(); it != versions_.end(); ++it) {
            if (it != versions_.begin()) oss << ", ";
            oss << it->first << ":" << it->second.toString();
        }
        oss << "}";
        return oss.str();
    }
};

// ============================================================================
// CRDT REPLICA WITH NETWORK SIMULATION
// ============================================================================

template<typename T>
class CRDTReplica {
private:
    std::string id_;
    std::shared_ptr<CRDT<T>> state_;
    std::vector<std::shared_ptr<CRDTReplica<T>>> peers_;
    std::mutex mtx_;

public:
    CRDTReplica(const std::string& id, std::shared_ptr<CRDT<T>> initialState)
        : id_(id), state_(initialState) {}

    void connect(CRDTReplica<T>* peer) {
        peers_.push_back(std::shared_ptr<CRDTReplica<T>>(peer));
    }

    template<typename Func>
    void applyOperation(Func op) {
        std::lock_guard<std::mutex> lock(mtx_);
        op(state_);
    }

    void gossip() {
        std::lock_guard<std::mutex> lock(mtx_);

        // Send state to random peers
        for (auto& peer : peers_) {
            if (peer) {
                peer->receiveState(state_);
            }
        }
    }

    void receiveState(std::shared_ptr<CRDT<T>> remoteState) {
        std::lock_guard<std::mutex> lock(mtx_);
        auto merged = state_->merge(*remoteState);
        state_ = std::make_shared<CRDT<T>>(merged);
    }

    std::shared_ptr<CRDT<T>> getState() const {
        std::lock_guard<std::mutex> lock(mtx_);
        return state_;
    }

    std::string getId() const { return id_; }
};

// ============================================================================
// DEMONSTRATION AND TESTING
// ============================================================================

void testCRDTProperties() {
    std::cout << "=" << std::string(60, '=') << std::endl;
    std::cout << "CRDT MATHEMATICAL PROPERTIES VERIFICATION" << std::endl;
    std::cout << "=" << std::string(60, '=') << std::endl;

    // Test GCounter
    {
        std::cout << "\n1. Testing GCounter (Grow-Only Counter):\n";

        GCounter c1("node1");
        GCounter c2("node2");
        GCounter c3("node3");

        c1.increment("node1", 5);
        c2.increment("node2", 3);
        c3.increment("node3", 7);

        // Test commutative: c1 ⊔ c2 == c2 ⊔ c1
        auto c1c2 = c1.merge(c2);
        auto c2c1 = c2.merge(c1);

        std::cout << "   Commutative: " 
                  << (c1c2 == c2c1 ? "✓" : "✗") 
                  << " (c1 ⊔ c2 == c2 ⊔ c1)" << std::endl;

        // Test associative: (c1 ⊔ c2) ⊔ c3 == c1 ⊔ (c2 ⊔ c3)
        auto left = c1.merge(c2).merge(c3);
        auto right = c1.merge(c2.merge(c3));

        std::cout << "   Associative: " 
                  << (left == right ? "✓" : "✗")
                  << " ((c1 ⊔ c2) ⊔ c3 == c1 ⊔ (c2 ⊔ c3))" << std::endl;

        // Test idempotent: c1 ⊔ c1 == c1
        auto c1c1 = c1.merge(c1);

        std::cout << "   Idempotent:  "
                  << (c1c1 == c1 ? "✓" : "✗")
                  << " (c1 ⊔ c1 == c1)" << std::endl;

        std::cout << "   Final value: " << left.value() << std::endl;
    }

    // Test PNCounter
    {
        std::cout << "\n2. Testing PNCounter (Positive-Negative Counter):\n";

        PNCounter p1("node1");
        PNCounter p2("node2");

        p1.increment("node1", 10);
        p1.decrement("node1", 3);

        p2.increment("node2", 5);
        p2.decrement("node2", 2);

        auto merged = p1.merge(p2);

        std::cout << "   Node1: " << p1.value() 
                  << " (10 - 3)" << std::endl;
        std::cout << "   Node2: " << p2.value() 
                  << " (5 - 2)" << std::endl;
        std::cout << "   Merged: " << merged.value() 
                  << " (15 - 5)" << std::endl;

        // Test lattice properties
        std::cout << "   Is lattice: " 
                  << (Lattice<PNCounter>::verifyCommutative(p1, p2) ? "✓" : "✗")
                  << std::endl;
    }

    // Test LWW Register
    {
        std::cout << "\n3. Testing LWWRegister (Last-Write-Wins):\n";

        LWWRegister<std::string> r1("node1", "initial");
        LWWRegister<std::string> r2("node2", "initial");

        // Simulate concurrent writes
        r1.set("value from node1");
        r2.set("value from node2");

        auto merged = r1.merge(r2);

        std::cout << "   Node1: " << r1.value() << std::endl;
        std::cout << "   Node2: " << r2.value() << std::endl;
        std::cout << "   Merged: " << merged.value() << std::endl;

        // Check which one won
        if (merged.value() == r1.value()) {
            std::cout << "   Winner: node1 (tie-break by node ID)" << std::endl;
        } else {
            std::cout << "   Winner: node2 (tie-break by node ID)" << std::endl;
        }
    }

    // Test MV Register
    {
        std::cout << "\n4. Testing MVRegister (Multi-Value):\n";

        MVRegister<std::string> mv1("node1", "A");
        MVRegister<std::string> mv2("node2", "B");

        // Concurrent writes
        mv1.set("A1");
        mv2.set("B1");

        auto merged = mv1.merge(mv2);
        auto values = merged.value();

        std::cout << "   After concurrent writes:" << std::endl;
        std::cout << "   Values: {";
        for (const auto& v : values) {
            std::cout << v << " ";
        }
        std::cout << "}" << std::endl;

        // Resolve by writing again
        mv1.set("resolved");
        auto resolved = mv1.merge(mv2);

        std::cout << "   After resolution: " 
                  << *resolved.value().begin() << std::endl;
    }
}

void simulateDistributedCounter() {
    std::cout << "\n" << std::string(60, '=') << std::endl;
    std::cout << "DISTRIBUTED COUNTER SIMULATION" << std::endl;
    std::cout << std::string(60, '=') << std::endl;

    // Create replicas
    auto counter1 = std::make_shared<PNCounter>("node1");
    auto counter2 = std::make_shared<PNCounter>("node2");
    auto counter3 = std::make_shared<PNCounter>("node3");

    CRDTReplica<PNCounter> r1("node1", counter1);
    CRDTReplica<PNCounter> r2("node2", counter2);
    CRDTReplica<PNCounter> r3("node3", counter3);

    // Connect replicas in a network
    r1.connect(&r2);
    r1.connect(&r3);
    r2.connect(&r1);
    r2.connect(&r3);
    r3.connect(&r1);
    r3.connect(&r2);

    // Simulate operations
    std::cout << "\nInitial state:" << std::endl;
    std::cout << "  Node1: " << r1.getState()->value() << std::endl;
    std::cout << "  Node2: " << r2.getState()->value() << std::endl;
    std::cout << "  Node3: " << r3.getState()->value() << std::endl;

    // Each node makes local updates
    r1.applyOperation([](auto& state) {
        auto pn = std::dynamic_pointer_cast<PNCounter>(state);
        pn->increment("node1", 5);
        pn->decrement("node1", 2);
    });

    r2.applyOperation([](auto& state) {
        auto pn = std::dynamic_pointer_cast<PNCounter>(state);
        pn->increment("node2", 3);
        pn->decrement("node2", 1);
    });

    r3.applyOperation([](auto& state) {
        auto pn = std::dynamic_pointer_cast<PNCounter>(state);
        pn->increment("node3", 7);
        pn->decrement("node3", 4);
    });

    std::cout << "\nAfter local updates:" << std::endl;
    std::cout << "  Node1: " << r1.getState()->value() << std::endl;
    std::cout << "  Node2: " << r2.getState()->value() << std::endl;
    std::cout << "  Node3: " << r3.getState()->value() << std::endl;

    // Perform gossip rounds
    for (int round = 1; round <= 3; round++) {
        std::cout << "\nGossip Round " << round << ":" << std::endl;

        r1.gossip();
        r2.gossip();
        r3.gossip();

        std::cout << "  Node1: " << r1.getState()->value() << std::endl;
        std::cout << "  Node2: " << r2.getState()->value() << std::endl;
        std::cout << "  Node3: " << r3.getState()->value() << std::endl;

        // Check for convergence
        auto v1 = r1.getState()->value();
        auto v2 = r2.getState()->value();
        auto v3 = r3.getState()->value();

        if (v1 == v2 && v2 == v3) {
            std::cout << "  ✓ Converged after " << round << " rounds!" << std::endl;
            break;
        }
    }

    // Verify final consistency
    std::cout << "\nFinal verification:" << std::endl;
    std::cout << "  All nodes equal: " 
              << (r1.getState()->value() == r2.getState()->value() && 
                  r2.getState()->value() == r3.getState()->value() ? "✓" : "✗")
              << std::endl;
    std::cout << "  Expected total: 8 (5-2 + 3-1 + 7-4)" << std::endl;
}

void demonstrateCALMTheorem() {
    std::cout << "\n" << std::string(60, '=') << std::endl;
    std::cout << "CALM THEOREM DEMONSTRATION" << std::endl;
    std::cout << "=" << std::string(60, '=') << std::endl;

    // The CALM theorem: Consistency As Logical Monotonicity
    // Monotonic programs don't need coordination

    std::cout << "\nMonotonic vs Non-Monotonic Computations:\n";

    // Example 1: Monotonic query (can use CRDTs)
    std::cout << "\n1. Monotonic: 'Has user visited page X?'\n";
    std::cout << "   - Once true, always true\n";
    std::cout << "   - Can use G-Set (add-only)\n";
    std::cout << "   - No coordination needed ✓\n";

    GSet<std::string> visitedPages;
    visitedPages.add("/home");
    visitedPages.add("/products");

    std::cout << "   Current state: " << visitedPages.toString() << std::endl;

    // Example 2: Non-monotonic query (needs coordination)
    std::cout << "\n2. Non-Monotonic: 'What is the last page user visited?'\n";
    std::cout << "   - Can change over time\n";
    std::cout << "   - Needs LWW-Register or consensus\n";
    std::cout << "   - Coordination required ✗\n";

    LWWRegister<std::string> lastPage("server1", "/home");
    lastPage.set("/products");
    lastPage.set("/checkout");

    std::cout << "   Last page: " << lastPage.value() << std::endl;

    // Mathematical formulation
    std::cout << "\nMathematical Formulation:\n";
    std::cout << "   Monotonic logic: if P ⊢ φ, then P ∪ Q ⊢ φ\n";
    std::cout << "   Non-monotonic:   P ⊢ φ but P ∪ Q ⊬ φ\n";
    std::cout << "\nCALM Theorem: A program has a consistent,\n";
    std::cout << "coordination-free distributed implementation\n";
    std::cout << "iff it is monotonic.\n";
}

int main() {
    std::cout << "COMPLETE CRDT LIBRARY WITH MATHEMATICAL FOUNDATIONS\n";
    std::cout << std::string(60, '=') << std::endl;

    testCRDTProperties();
    simulateDistributedCounter();
    demonstrateCALMTheorem();

    return 0;
}

3. Dynamo-style Quorums: Complete Mathematical Analysis

Complete Theory of Quorum Systems

A quorum system Q is a collection of subsets of nodes where each pair intersects:

∀ Q₁, Q₂ ∈ Q: Q₁ ∩ Q₂ ≠ ∅

Mathematical Properties:

Load: Probability a node is in a randomly chosen quorum
Capacity: Maximum throughput the system can handle
Fault tolerance: Maximum number of failures the system can tolerate

Optimal Quorum Sizes:

For N nodes with read quorum R and write quorum W:

Strong consistency: R + W > N (and W > N/2 to avoid write conflicts)
Eventual consistency: R + W ≤ N

Complete Python Implementation with Proofs

import numpy as np
import itertools
from typing import List, Set, Tuple, Dict
from dataclasses import dataclass
from enum import Enum
import random
import hashlib
from fractions import Fraction
import matplotlib.pyplot as plt

class ConsistencyLevel(Enum):
    STRONG = "strong"
    EVENTUAL = "eventual"
    CAUSAL = "causal"
    LINEARIZABLE = "linearizable"

@dataclass
class QuorumSystem:
    """
    Mathematical analysis of quorum systems
    """
    N: int  # Total nodes
    R: int  # Read quorum size
    W: int  # Write quorum size

    def __post_init__(self):
        if not (1 <= self.R <= self.N and 1 <= self.W <= self.N):
            raise ValueError("Quorum sizes must be between 1 and N")

    @property
    def quorums(self) -> List[Set[int]]:
        """Generate all possible quorums of size R and W"""
        read_quorums = [set(c) for c in itertools.combinations(range(self.N), self.R)]
        write_quorums = [set(c) for c in itertools.combinations(range(self.N), self.W)]
        return read_quorums, write_quorums

    def is_available(self, failed_nodes: Set[int]) -> Tuple[bool, bool]:
        """
        Check if read and write are still possible with failed nodes

        Returns: (read_possible, write_possible)
        """
        # Read is possible if there exists a read quorum without failed nodes
        read_possible = any(failed_nodes.isdisjoint(q) 
                           for q in self.read_quorums)

        # Write is possible if there exists a write quorum without failed nodes
        write_possible = any(failed_nodes.isdisjoint(q) 
                            for q in self.write_quorums)

        return read_possible, write_possible

    @property
    def read_quorums(self) -> List[Set[int]]:
        return [set(c) for c in itertools.combinations(range(self.N), self.R)]

    @property
    def write_quorums(self) -> List[Set[int]]:
        return [set(c) for c in itertools.combinations(range(self.N), self.W)]

    def probability_available(self, p_fail: float) -> Tuple[float, float]:
        """
        Calculate probability that read and write are available

        Args:
            p_fail: Probability a single node fails

        Returns: (P(read available), P(write available))
        """
        # Probability exactly k nodes fail
        def P_k(k: int) -> float:
            return (np.math.comb(self.N, k) * 
                   (p_fail ** k) * 
                   ((1 - p_fail) ** (self.N - k)))

        # Read available if at least R nodes are up
        P_read = sum(P_k(k) for k in range(self.N - self.R + 1))

        # Write available if at least W nodes are up
        P_write = sum(P_k(k) for k in range(self.N - self.W + 1))

        return P_read, P_write

    def consistency_level(self) -> ConsistencyLevel:
        """Determine consistency level based on quorum configuration"""
        if self.R + self.W > self.N and self.W > self.N / 2:
            return ConsistencyLevel.LINEARIZABLE
        elif self.R + self.W > self.N:
            return ConsistencyLevel.STRONG
        elif self.R + self.W == self.N:
            return ConsistencyLevel.CAUSAL
        else:
            return ConsistencyLevel.EVENTUAL

    def load(self) -> Tuple[float, float]:
        """
        Calculate load on nodes for read and write operations

        Returns: (read_load, write_load)
        """
        # Probability a node is in a randomly chosen quorum
        read_load = self.R / self.N
        write_load = self.W / self.N

        return read_load, write_load

    def capacity(self) -> float:
        """
        Calculate maximum throughput capacity

        Returns: Minimum of read and write capacity
        """
        read_load, write_load = self.load()

        # Capacity is inverse of load
        read_capacity = 1 / read_load if read_load > 0 else float('inf')
        write_capacity = 1 / write_load if write_load > 0 else float('inf')

        return min(read_capacity, write_capacity)

    def fault_tolerance(self) -> Dict[str, int]:
        """
        Calculate fault tolerance metrics

        Returns: Dictionary with various tolerance measures
        """
        # Maximum failures before read becomes impossible
        max_failures_read = self.N - self.R

        # Maximum failures before write becomes impossible
        max_failures_write = self.N - self.W

        # Failures that still guarantee at least one intersecting node
        # between any read and write quorum
        if self.R + self.W > self.N:
            max_intersection_failures = self.R + self.W - self.N - 1
        else:
            max_intersection_failures = -1  # No guarantee

        return {
            'max_failures_for_read': max_failures_read,
            'max_failures_for_write': max_failures_write,
            'max_failures_guaranteeing_intersection': max_intersection_failures,
            'can_tolerate_network_partition': self.R + self.W > self.N
        }

    def optimal_configurations(self) -> List['QuorumSystem']:
        """
        Find optimal configurations for given N

        Returns: List of Pareto-optimal configurations
        """
        optimal = []

        for R in range(1, self.N + 1):
            for W in range(1, self.N + 1):
                config = QuorumSystem(self.N, R, W)

                # Skip dominated configurations
                dominated = False
                for opt in optimal[:]:  # Copy list for iteration
                    if (R >= opt.R and W >= opt.W and 
                        config.capacity() <= opt.capacity()):
                        dominated = True
                        break
                    elif (R <= opt.R and W <= opt.W and 
                          config.capacity() >= opt.capacity()):
                        # Current optimal is dominated
                        optimal.remove(opt)

                if not dominated:
                    optimal.append(config)

        return sorted(optimal, key=lambda x: x.capacity(), reverse=True)

class DynamoDBStore:
    """
    Complete Dynamo-style key-value store implementation
    """

    def __init__(self, N: int = 3, R: int = 2, W: int = 2):
        self.quorum_system = QuorumSystem(N, R, W)
        self.nodes = [{} for _ in range(N)]
        self.clocks = [{} for _ in range(N)]  # Vector clocks per key
        self.hinted_handoff = {}  # For handling failures
        self.merkle_trees = [{} for _ in range(N)]  # For anti-entropy

        # Statistics
        self.stats = {
            'reads': 0,
            'writes': 0,
            'read_repairs': 0,
            'hinted_handoffs': 0,
            'conflicts': 0
        }

    def _hash_key(self, key: str) -> List[int]:
        """Consistent hashing to determine node positions"""
        # Using SHA-1 for consistent hashing
        hash_obj = hashlib.sha1(key.encode())
        hash_int = int(hash_obj.hexdigest(), 16)

        # Return N nodes in preference list
        nodes = []
        for i in range(self.quorum_system.N):
            node_idx = (hash_int + i) % self.quorum_system.N
            nodes.append(node_idx)

        return nodes

    def _get_preference_list(self, key: str) -> List[int]:
        """Get ordered list of nodes for a key"""
        return self._hash_key(key)

    def _update_vector_clock(self, node_idx: int, key: str, client_context: Dict = None):
        """Update vector clock for a key"""
        if key not in self.clocks[node_idx]:
            self.clocks[node_idx][key] = {}

        clock = self.clocks[node_idx][key]

        if client_context and 'clock' in client_context:
            # Merge with client's clock
            for node, time in client_context['clock'].items():
                clock[node] = max(clock.get(node, 0), time)

        # Increment this node's counter
        clock[str(node_idx)] = clock.get(str(node_idx), 0) + 1

        return clock

    def _compare_clocks(self, clock1: Dict, clock2: Dict) -> str:
        """
        Compare two vector clocks

        Returns: 'before', 'after', 'concurrent', or 'equal'
        """
        all_nodes = set(clock1.keys()) | set(clock2.keys())

        clock1_before = all(clock1.get(n, 0) <= clock2.get(n, 0) for n in all_nodes)
        clock2_before = all(clock2.get(n, 0) <= clock1.get(n, 0) for n in all_nodes)

        if clock1_before and clock2_before:
            return 'equal'
        elif clock1_before:
            return 'before'
        elif clock2_before:
            return 'after'
        else:
            return 'concurrent'

    def write(self, key: str, value: any, context: Dict = None) -> Dict:
        """
        Write a key-value pair with quorum consistency

        Returns: Context for future reads
        """
        self.stats['writes'] += 1

        preference_list = self._get_preference_list(key)
        successful_writes = 0
        written_nodes = []
        latest_clock = None

        # Try to write to first N nodes in preference list
        for node_idx in preference_list[:self.quorum_system.N]:
            try:
                # Update vector clock
                clock = self._update_vector_clock(node_idx, key, context)
                latest_clock = clock

                # Store value
                self.nodes[node_idx][key] = {
                    'value': value,
                    'clock': clock.copy(),
                    'timestamp': time.time()
                }

                successful_writes += 1
                written_nodes.append(node_idx)

                if successful_writes >= self.quorum_system.W:
                    break  # Quorum achieved

            except Exception as e:
                # Node failed, use hinted handoff
                self._hint_handoff(key, value, clock, node_idx)

        if successful_writes < self.quorum_system.W:
            raise Exception(f"Write failed: only {successful_writes}/{self.quorum_system.W} successful")

        # Return context for client
        return {
            'clock': latest_clock,
            'written_nodes': written_nodes,
            'key': key
        }

    def _hint_handoff(self, key: str, value: any, clock: Dict, failed_node: int):
        """Store data temporarily when primary node is down"""
        self.stats['hinted_handoffs'] += 1

        # Find a healthy node to store hint
        preference_list = self._get_preference_list(key)
        for node_idx in preference_list:
            if node_idx != failed_node and node_idx not in self.hinted_handoff:
                self.hinted_handoff[(key, failed_node)] = {
                    'value': value,
                    'clock': clock,
                    'stored_at': node_idx,
                    'timestamp': time.time()
                }
                break

    def read(self, key: str) -> Tuple[any, Dict]:
        """
        Read a key-value pair with quorum consistency

        Returns: (value, context) or (None, None) if not found
        """
        self.stats['reads'] += 1

        preference_list = self._get_preference_list(key)
        responses = []

        # Read from R nodes
        for node_idx in preference_list[:self.quorum_system.N]:
            if key in self.nodes[node_idx]:
                data = self.nodes[node_idx][key]
                responses.append({
                    'node': node_idx,
                    'value': data['value'],
                    'clock': data['clock'],
                    'timestamp': data['timestamp']
                })

            if len(responses) >= self.quorum_system.R:
                break

        if not responses:
            return None, None

        # Check if we have conflicting versions
        versions = []
        for resp in responses:
            versions.append((resp['value'], resp['clock']))

        # Resolve conflicts
        resolved_value, resolved_clock = self._resolve_conflicts(versions)

        # Perform read repair if needed
        self._read_repair(key, resolved_value, resolved_clock, 
                         [r['node'] for r in responses])

        return resolved_value, {'clock': resolved_clock}

    def _resolve_conflicts(self, versions: List[Tuple[any, Dict]]) -> Tuple[any, Dict]:
        """
        Resolve conflicting versions using vector clocks

        Returns: (resolved_value, merged_clock)
        """
        if len(versions) == 1:
            return versions[0]

        # Check if all versions are causally related
        all_same = all(self._compare_clocks(v1[1], v2[1]) == 'equal' 
                      for v1 in versions for v2 in versions)

        if all_same:
            # No actual conflict
            return versions[0]

        # Find latest version based on vector clocks
        latest = versions[0]
        for version in versions[1:]:
            comparison = self._compare_clocks(latest[1], version[1])
            if comparison == 'before':
                latest = version
            elif comparison == 'concurrent':
                # Concurrent writes: need application-specific resolution
                self.stats['conflicts'] += 1
                # Default: last write wins based on timestamp
                if version[0] > latest[0]:  # Simple heuristic
                    latest = version

        # Merge clocks
        merged_clock = {}
        for value, clock in versions:
            for node, time in clock.items():
                merged_clock[node] = max(merged_clock.get(node, 0), time)

        return latest[0], merged_clock

    def _read_repair(self, key: str, value: any, clock: Dict, read_nodes: List[int]):
        """Repair replicas that have stale data"""
        self.stats['read_repairs'] += 1

        preference_list = self._get_preference_list(key)

        for node_idx in preference_list[:self.quorum_system.N]:
            if node_idx in read_nodes:
                continue

            if key in self.nodes[node_idx]:
                # Compare clocks
                existing_clock = self.nodes[node_idx][key]['clock']
                comparison = self._compare_clocks(existing_clock, clock)

                if comparison == 'before':
                    # Existing data is stale, update it
                    self.nodes[node_idx][key] = {
                        'value': value,
                        'clock': clock,
                        'timestamp': time.time()
                    }
            else:
                # Node doesn't have this key, add it
                self.nodes[node_idx][key] = {
                    'value': value,
                    'clock': clock,
                    'timestamp': time.time()
                }

    def simulate_failures(self, failed_nodes: Set[int]):
        """Simulate node failures"""
        for node_idx in failed_nodes:
            self.nodes[node_idx] = {}  # Clear node data
            print(f"Node {node_idx} failed")

    def get_statistics(self) -> Dict:
        """Get store statistics"""
        # Calculate data distribution
        node_loads = [len(node) for node in self.nodes]
        avg_load = np.mean(node_loads)
        load_std = np.std(node_loads)

        return {
            **self.stats,
            'node_loads': node_loads,
            'avg_load': avg_load,
            'load_std': load_std,
            'load_imbalance': load_std / avg_load if avg_load > 0 else 0,
            'config': {
                'N': self.quorum_system.N,
                'R': self.quorum_system.R,
                'W': self.quorum_system.W,
                'consistency': self.quorum_system.consistency_level().value
            }
        }

def analyze_quorum_tradeoffs_comprehensive():
    """
    Complete analysis of quorum system tradeoffs
    """
    print("="*70)
    print("COMPREHENSIVE QUORUM SYSTEM ANALYSIS")
    print("="*70)

    N = 5  # Number of nodes

    # Analyze all possible configurations
    all_configs = []
    for R in range(1, N + 1):
        for W in range(1, N + 1):
            qs = QuorumSystem(N, R, W)

            # Calculate metrics
            consistency = qs.consistency_level()
            read_load, write_load = qs.load()
            capacity = qs.capacity()
            fault_tolerance = qs.fault_tolerance()
            P_read, P_write = qs.probability_available(0.01)  # 1% failure probability

            all_configs.append({
                'R': R,
                'W': W,
                'consistency': consistency,
                'read_load': read_load,
                'write_load': write_load,
                'capacity': capacity,
                'max_read_failures': fault_tolerance['max_failures_for_read'],
                'max_write_failures': fault_tolerance['max_failures_for_write'],
                'read_availability': P_read,
                'write_availability': P_write,
                'R+W': R + W,
                'R+W>N': R + W > N,
                'W>N/2': W > N / 2
            })

    # Sort by different criteria
    print("\n1. Configurations sorted by consistency strength:")
    print("-"*70)
    sorted_by_consistency = sorted(all_configs, 
                                  key=lambda x: (x['R+W>N'], x['W>N/2']), 
                                  reverse=True)

    for config in sorted_by_consistency[:10]:
        print(f"R={config['R']}, W={config['W']}: "
              f"{config['consistency'].value.upper():12} "
              f"(R+W={config['R+W']}, R+W>N={config['R+W>N']}, W>N/2={config['W>N/2']})")

    print("\n2. Configurations sorted by capacity (throughput):")
    print("-"*70)
    sorted_by_capacity = sorted(all_configs, key=lambda x: x['capacity'], reverse=True)

    for config in sorted_by_capacity[:10]:
        print(f"R={config['R']}, W={config['W']}: "
              f"Capacity={config['capacity']:.2f} ops/sec, "
              f"Availability(R/W)={config['read_availability']:.3f}/{config['write_availability']:.3f}")

    print("\n3. Pareto-optimal configurations (tradeoff frontier):")
    print("-"*70)

    # Find Pareto frontier
    pareto_front = []
    for config in all_configs:
        dominated = False
        for other in all_configs:
            if (other['read_availability'] >= config['read_availability'] and
                other['write_availability'] >= config['write_availability'] and
                other['capacity'] >= config['capacity'] and
                (other['read_availability'] > config['read_availability'] or
                 other['write_availability'] > config['write_availability'] or
                 other['capacity'] > config['capacity'])):
                dominated = True
                break
        if not dominated:
            pareto_front.append(config)

    # Sort by availability
    pareto_front.sort(key=lambda x: x['read_availability'], reverse=True)

    for config in pareto_front:
        print(f"R={config['R']:2d}, W={config['W']:2d}: "
              f"Consistency={config['consistency'].value:12}, "
              f"Avail(R)={config['read_availability']:.3f}, "
              f"Avail(W)={config['write_availability']:.3f}, "
              f"Capacity={config['capacity']:.2f}")

    # Visualize tradeoffs
    visualize_quorum_tradeoffs(all_configs, pareto_front)

def visualize_quorum_tradeoffs(all_configs, pareto_front):
    """Create comprehensive visualizations of quorum tradeoffs"""

    fig, axes = plt.subplots(2, 3, figsize=(15, 10))

    # Plot 1: Availability vs Capacity
    ax1 = axes[0, 0]
    read_avails = [c['read_availability'] for c in all_configs]
    capacities = [c['capacity'] for c in all_configs]
    colors = []

    for config in all_configs:
        if config['consistency'] == ConsistencyLevel.LINEARIZABLE:
            colors.append('red')
        elif config['consistency'] == ConsistencyLevel.STRONG:
            colors.append('orange')
        elif config['consistency'] == ConsistencyLevel.CAUSAL:
            colors.append('yellow')
        else:
            colors.append('blue')

    scatter = ax1.scatter(read_avails, capacities, c=colors, alpha=0.6, s=50)
    ax1.set_xlabel('Read Availability')
    ax1.set_ylabel('Capacity (ops/sec)')
    ax1.set_title('Availability vs Capacity Tradeoff')
    ax1.grid(True, alpha=0.3)

    # Add Pareto frontier
    pareto_avails = [c['read_availability'] for c in pareto_front]
    pareto_caps = [c['capacity'] for c in pareto_front]
    ax1.plot(pareto_avails, pareto_caps, 'k--', alpha=0.5, label='Pareto Frontier')
    ax1.legend()

    # Plot 2: Consistency vs Load
    ax2 = axes[0, 1]
    consistency_levels = [c['consistency'].value for c in all_configs]
    unique_levels = list(set(consistency_levels))
    level_to_num = {level: i for i, level in enumerate(unique_levels)}

    load_imbalance = []
    for config in all_configs:
        # Calculate load imbalance
        imbalance = abs(config['read_load'] - config['write_load'])
        load_imbalance.append(imbalance)

    ax2.scatter([level_to_num[l] for l in consistency_levels], 
                load_imbalance, alpha=0.6)
    ax2.set_xlabel('Consistency Level')
    ax2.set_ylabel('Load Imbalance |R/N - W/N|')
    ax2.set_title('Consistency vs Load Balance')
    ax2.set_xticks(range(len(unique_levels)))
    ax2.set_xticklabels(unique_levels, rotation=45)
    ax2.grid(True, alpha=0.3)

    # Plot 3: Fault Tolerance Heatmap
    ax3 = axes[0, 2]

    # Create heatmap data
    heatmap_data = np.zeros((N, N))
    for config in all_configs:
        R, W = config['R'], config['W']
        # Use availability as metric
        heatmap_data[R-1, W-1] = config['read_availability'] * config['write_availability']

    im = ax3.imshow(heatmap_data, cmap='viridis', origin='lower')
    ax3.set_xlabel('Write Quorum Size (W)')
    ax3.set_ylabel('Read Quorum Size (R)')
    ax3.set_title('Availability Heatmap (R×W)')
    plt.colorbar(im, ax=ax3)

    # Add contour for R+W>N
    X, Y = np.meshgrid(range(1, N+1), range(1, N+1))
    Z = (X + Y) > N
    ax3.contour(X-1, Y-1, Z, levels=[0.5], colors='white', linewidths=2)

    # Add contour for W>N/2
    Z2 = Y > N/2
    ax3.contour(X-1, Y-1, Z2, levels=[0.5], colors='red', linewidths=2, linestyles='--')

    # Plot 4: Capacity vs Consistency
    ax4 = axes[1, 0]

    # Group by consistency level
    grouped_data = {}
    for config in all_configs:
        level = config['consistency'].value
        if level not in grouped_data:
            grouped_data[level] = []
        grouped_data[level].append(config['capacity'])

    boxes = []
    labels = []
    for level, capacities in grouped_data.items():
        boxes.append(capacities)
        labels.append(level)

    ax4.boxplot(boxes, labels=labels)
    ax4.set_ylabel('Capacity (ops/sec)')
    ax4.set_title('Capacity Distribution by Consistency Level')
    ax4.grid(True, alpha=0.3)
    ax4.tick_params(axis='x', rotation=45)

    # Plot 5: Tradeoff Surface (3D)
    ax5 = axes[1, 1]

    # Extract data for 3D plot
    R_vals = [c['R'] for c in all_configs]
    W_vals = [c['W'] for c in all_configs]
    avail_vals = [c['read_availability'] * c['write_availability'] for c in all_configs]
    cap_vals = [c['capacity'] for c in all_configs]

    # Color by consistency
    colors_3d = []
    for config in all_configs:
        if config['consistency'] == ConsistencyLevel.LINEARIZABLE:
            colors_3d.append('red')
        elif config['consistency'] == ConsistencyLevel.STRONG:
            colors_3d.append('orange')
        elif config['consistency'] == ConsistencyLevel.CAUSAL:
            colors_3d.append('green')
        else:
            colors_3d.append('blue')

    scatter = ax5.scatter(R_vals, W_vals, s=np.array(cap_vals)*10, 
                         c=colors_3d, alpha=0.6)
    ax5.set_xlabel('R')
    ax5.set_ylabel('W')
    ax5.set_title('Quorum Configuration Space\n(Size = Capacity)')
    ax5.grid(True, alpha=0.3)

    # Add R+W=N line
    line_x = np.arange(1, N+1)
    line_y = N - line_x
    ax5.plot(line_x, line_y, 'k--', alpha=0.5, label='R+W=N')
    ax5.legend()

    # Plot 6: Failure Tolerance Analysis
    ax6 = axes[1, 2]

    # Simulate different failure scenarios
    failure_rates = np.linspace(0.001, 0.1, 20)

    # Test different configurations
    test_configs = [
        (2, 2),  # Strong
        (1, 3),  # Write-heavy
        (3, 1),  # Read-heavy
        (1, 1),  # Weak
    ]

    for R, W in test_configs:
        qs = QuorumSystem(N, R, W)
        availabilities = []

        for p_fail in failure_rates:
            P_read, P_write = qs.probability_available(p_fail)
            availabilities.append(P_read * P_write)  # Both read and write available

        ax6.plot(failure_rates, availabilities, 
                label=f'R={R}, W={W}', linewidth=2)

    ax6.set_xlabel('Node Failure Probability')
    ax6.set_ylabel('System Availability')
    ax6.set_title('Availability vs Failure Rate')
    ax6.legend()
    ax6.grid(True, alpha=0.3)

    plt.tight_layout()
    plt.show()

    # Print optimal configurations for different scenarios
    print("\n4. Recommended configurations for different use cases:")
    print("-"*70)

    recommendations = [
        ("Financial transactions", ConsistencyLevel.LINEARIZABLE, 0.9999, 1000),
        ("Social media feeds", ConsistencyLevel.EVENTUAL, 0.99, 10000),
        ("Shopping cart", ConsistencyLevel.CAUSAL, 0.999, 5000),
        ("Configuration data", ConsistencyLevel.STRONG, 0.9999, 100),
        ("Analytics data", ConsistencyLevel.EVENTUAL, 0.95, 50000),
    ]

    for use_case, required_consistency, required_availability, required_throughput in recommendations:
        print(f"\n{use_case}:")
        print(f"  Requirements: {required_consistency.value}, "
              f"avail>{required_availability}, "
              f"throughput>{required_throughput}")

        # Find matching configurations
        matches = []
        for config in all_configs:
            if (config['consistency'] == required_consistency and
                config['read_availability'] >= required_availability and
                config['capacity'] >= required_throughput):
                matches.append(config)

        if matches:
            # Sort by capacity (ascending for efficiency)
            matches.sort(key=lambda x: x['capacity'])
            best = matches[0]
            print(f"  Recommended: R={best['R']}, W={best['W']}")
            print(f"    Capacity: {best['capacity']:.0f} ops/sec")
            print(f"    Availability: {best['read_availability']:.4f}")
        else:
            print(f"  No configuration meets all requirements")

# Run simulation of Dynamo-style store
def simulate_dynamo_store():
    """Complete simulation of Dynamo-style store with failures"""

    print("\n" + "="*70)
    print("DYNAMO-STORE SIMULATION WITH FAILURES")
    print("="*70)

    # Create store with strong consistency
    store = DynamoDBStore(N=5, R=3, W=3)

    print("\n1. Writing initial data...")
    contexts = {}

    # Write some data
    keys = ["user:1001:profile", "user:1001:cart", "config:feature_flag"]
    values = [{"name": "Alice", "age": 30}, 
              {"items": ["book", "pen"]},
              {"new_ui": True}]

    for key, value in zip(keys, values):
        context = store.write(key, value)
        contexts[key] = context
        print(f"  Wrote {key}: {value}")

    print("\n2. Reading data normally...")
    for key in keys:
        value, ctx = store.read(key)
        print(f"  Read {key}: {value}")

    print("\n3. Simulating network partition...")
    # Partition: nodes 0,1 in one partition, 2,3,4 in another
    store.simulate_failures({0, 1})

    # Try operations during partition
    print("\n   During partition:")

    try:
        # Write during partition (should fail for strong consistency)
        context = store.write("user:1001:profile", 
                            {"name": "Alice", "age": 31},
                            contexts["user:1001:profile"])
        print("   Write succeeded (unexpected!)")
    except Exception as e:
        print(f"   Write failed: {e}")

    try:
        # Read during partition
        value, ctx = store.read("user:1001:profile")
        print(f"   Read succeeded: {value}")
    except Exception as e:
        print(f"   Read failed: {e}")

    print("\n4. Healing partition and testing convergence...")
    # Clear failures
    store.simulate_failures(set())

    # Read after healing
    for key in keys:
        value, ctx = store.read(key)
        print(f"  After healing {key}: {value}")

    print("\n5. Testing concurrent writes...")
    # Simulate concurrent writes from different clients
    print("\n   Client A writes version 1...")
    context_a = store.write("concurrent:key", {"version": 1})

    print("   Client B writes version 2 (without seeing A)...")
    # Create a store that hasn't seen A's write
    store_b = DynamoDBStore(N=5, R=3, W=3)
    context_b = store_b.write("concurrent:key", {"version": 2})

    print("   Merging states...")
    # Now merge the two stores
    for node in range(5):
        if "concurrent:key" in store_b.nodes[node]:
            store.nodes[node]["concurrent:key"] = store_b.nodes[node]["concurrent:key"]

    # Read and resolve conflict
    value, ctx = store.read("concurrent:key")
    print(f"   Resolved value: {value}")

    print("\n6. Statistics:")
    stats = store.get_statistics()
    for key, value in stats.items():
        if key != 'node_loads':
            print(f"  {key}: {value}")

    # Show node load distribution
    print(f"\n  Node loads: {stats['node_loads']}")
    print(f"  Load imbalance: {stats['load_imbalance']:.3f}")

def prove_quorum_properties():
    """
    Mathematical proofs of quorum system properties
    """
    print("\n" + "="*70)
    print("MATHEMATICAL PROOFS OF QUORUM PROPERTIES")
    print("="*70)

    print("\nTheorem 1: For strong consistency, R + W > N")
    print("Proof:")
    print("  1. Let Q_R be a read quorum and Q_W be a write quorum.")
    print("  2. For strong consistency, every read must see the latest write.")
    print("  3. This requires Q_R ∩ Q_W ≠ ∅ (they must intersect).")
    print("  4. By pigeonhole principle, if |Q_R| + |Q_W| > N,")
    print("     then Q_R and Q_W must intersect.")
    print("  5. Therefore, R + W > N is necessary. ✓")

    print("\nTheorem 2: To avoid write conflicts, W > N/2")
    print("Proof:")
    print("  1. Consider two concurrent writes with quorums Q1 and Q2.")
    print("  2. If |Q1| > N/2 and |Q2| > N/2, then Q1 ∩ Q2 ≠ ∅.")
    print("  3. The intersecting node will see both writes and can order them.")
    print("  4. If W ≤ N/2, two writes could use disjoint quorums,")
    print("     leading to conflicts that require complex resolution.")
    print("  5. Therefore, W > N/2 avoids write conflicts. ✓")

    print("\nTheorem 3: Optimal load is achieved when R = W = ⌈(N+1)/2⌉")
    print("Proof:")
    print("  1. Load L = max(R/N, W/N) (bottleneck resource).")
    print("  2. Constraint: R + W > N for strong consistency.")
    print("  3. Minimize L subject to R + W > N, 1 ≤ R,W ≤ N.")
    print("  4. By symmetry and linear programming, minimum occurs when R = W.")
    print("  5. With R = W, constraint becomes 2R > N ⇒ R > N/2.")
    print("  6. Smallest integer satisfying this is R = ⌈(N+1)/2⌉.")
    print("  7. Therefore, R = W = ⌈(N+1)/2⌉ minimizes load. ✓")

    print("\nTheorem 4: Maximum failures tolerated is min(N-R, N-W)")
    print("Proof:")
    print("  1. Read requires at least R functioning nodes.")
    print("  2. Therefore, can tolerate up to N-R failures for read.")
    print("  3. Write requires at least W functioning nodes.")
    print("  4. Therefore, can tolerate up to N-W failures for write.")
    print("  5. System fails if either read or write becomes impossible.")
    print("  6. Therefore, max failures = min(N-R, N-W). ✓")

    print("\nTheorem 5: For N=3, R=2, W=2 is optimal for strong consistency")
    print("Proof:")
    print("  1. Strong consistency requires R + W > 3.")
    print("  2. Possible pairs: (2,2), (1,3), (3,1), (2,3), (3,2).")
    print("  3. Load L = max(R/3, W/3).")
    print("  4. Calculate loads:")
    print("     (2,2): L = 2/3 ≈ 0.667")
    print("     (1,3): L = 3/3 = 1.0")
    print("     (3,1): L = 3/3 = 1.0")
    print("     (2,3): L = 3/3 = 1.0")
    print("     (3,2): L = 3/3 = 1.0")
    print("  5. (2,2) has minimum load while providing strong consistency.")
    print("  6. Also satisfies W > N/2 (2 > 1.5) to avoid write conflicts.")
    print("  7. Therefore, (2,2) is optimal. ✓")

if __name__ == "__main__":
    analyze_quorum_tradeoffs_comprehensive()
    simulate_dynamo_store()
    prove_quorum_properties()

4. Erasure Coding Theory: Complete Mathematical Foundation

The Mathematics of Reed-Solomon Codes

Erasure coding transforms k data blocks into n encoded blocks (n > k) such that any k out of n blocks can reconstruct the original data. This is based on polynomial interpolation over finite fields.

Mathematical Foundation:

Finite Field (Galois Field) GF(2^w):
- A field with 2^w elements
- Operations: addition (XOR), multiplication (log/antilog tables)
- Primitive polynomial defines the field
Vandermonde Matrix:
For encoding, we create matrix V where:

   V[i][j] = (i+1)^j in GF(2^w)

The encoding operation: encoded = V × data

Reconstruction: Any k rows of V form a k×k matrix V' We solve: data = V'⁻¹ × encoded'

Complete Python Implementation with Galois Field Arithmetic

import numpy as np
from typing import List, Optional, Tuple
import math
from dataclasses import dataclass
from functools import lru_cache
import random
import hashlib
import time

class GaloisField:
    """
    Complete implementation of GF(2^8) for Reed-Solomon coding
    Uses primitive polynomial x^8 + x^4 + x^3 + x^2 + 1
    """

    def __init__(self, w: int = 8):
        self.w = w
        self.size = 1 << w  # 2^w
        self.gflog = [0] * self.size
        self.gfilog = [0] * self.size
        self.primitive_poly = 0x11D  # x^8 + x^4 + x^3 + x^2 + 1

        self._init_tables()

    def _init_tables(self):
        """Initialize logarithm and inverse logarithm tables"""
        x = 1
        for i in range(self.size - 1):
            self.gflog[x] = i
            self.gfilog[i] = x

            x <<= 1
            if x & self.size:
                x ^= self.primitive_poly

        # Set log(0) to a large value (undefined)
        self.gflog[0] = 2 * self.size

    def add(self, a: int, b: int) -> int:
        """Addition in GF(2^w) is XOR"""
        return a ^ b

    def subtract(self, a: int, b: int) -> int:
        """Subtraction in GF(2^w) is XOR (same as addition)"""
        return a ^ b

    def multiply(self, a: int, b: int) -> int:
        """Multiplication in GF(2^w) using log/antilog tables"""
        if a == 0 or b == 0:
            return 0

        # log(a) + log(b) mod (2^w - 1)
        sum_log = (self.gflog[a] + self.gflog[b]) % (self.size - 1)

        return self.gfilog[sum_log]

    def divide(self, a: int, b: int) -> int:
        """Division in GF(2^w)"""
        if b == 0:
            raise ValueError("Division by zero")
        if a == 0:
            return 0

        # log(a) - log(b) mod (2^w - 1)
        diff_log = (self.gflog[a] - self.gflog[b]) % (self.size - 1)

        return self.gfilog[diff_log]

    def power(self, a: int, n: int) -> int:
        """Exponentiation in GF(2^w)"""
        if a == 0:
            return 0
        if n == 0:
            return 1

        # (log(a) * n) mod (2^w - 1)
        power_log = (self.gflog[a] * n) % (self.size - 1)

        return self.gfilog[power_log]

    def inverse(self, a: int) -> int:
        """Multiplicative inverse in GF(2^w)"""
        if a == 0:
            raise ValueError("Zero has no inverse")

        # -log(a) mod (2^w - 1)
        inv_log = (self.size - 1 - self.gflog[a]) % (self.size - 1)

        return self.gfilog[inv_log]

    def eval_polynomial(self, coeffs: List[int], x: int) -> int:
        """Evaluate polynomial at point x using Horner's method"""
        result = 0
        for coeff in reversed(coeffs):
            result = self.multiply(result, x)
            result = self.add(result, coeff)
        return result

    def interpolate(self, points: List[Tuple[int, int]]) -> List[int]:
        """
        Lagrange interpolation in GF(2^w)
        Returns polynomial coefficients
        """
        k = len(points)

        # Initialize result polynomial coefficients to 0
        coeffs = [0] * k

        for i in range(k):
            # Compute Lagrange basis polynomial L_i(x)
            xi, yi = points[i]

            # Compute numerator product (x - x_j) for j != i
            numerator = [1]  # Start with 1 (empty product)

            for j in range(k):
                if i == j:
                    continue
                xj = points[j][0]

                # Multiply numerator by (x - x_j)
                new_numerator = [0] * (len(numerator) + 1)
                for deg, coeff in enumerate(numerator):
                    # coeff * x
                    new_numerator[deg + 1] = self.add(new_numerator[deg + 1], coeff)
                    # coeff * (-xj)
                    term = self.multiply(coeff, xj)
                    new_numerator[deg] = self.add(new_numerator[deg], term)

                numerator = new_numerator

            # Compute denominator product (x_i - x_j) for j != i
            denominator = 1
            for j in range(k):
                if i == j:
                    continue
                xj = points[j][0]
                denominator = self.multiply(denominator, self.add(xi, xj))

            # Multiply numerator by y_i / denominator
            scale = self.divide(yi, denominator)

            # Add scaled numerator to result
            for deg in range(len(numerator)):
                coeffs[deg] = self.add(coeffs[deg], self.multiply(numerator[deg], scale))

        return coeffs

    def vandermonde_matrix(self, rows: int, cols: int) -> np.ndarray:
        """Generate Vandermonde matrix for encoding"""
        matrix = np.zeros((rows, cols), dtype=int)

        for i in range(rows):
            for j in range(cols):
                matrix[i, j] = self.power(i + 1, j)  # (i+1)^j

        return matrix

    def invert_matrix(self, matrix: np.ndarray) -> np.ndarray:
        """Invert matrix in GF(2^w) using Gaussian elimination"""
        n = matrix.shape[0]

        # Augment matrix with identity
        augmented = np.hstack([matrix, np.eye(n, dtype=int)])

        # Gaussian elimination
        for col in range(n):
            # Find pivot
            pivot = col
            while pivot < n and augmented[pivot, col] == 0:
                pivot += 1

            if pivot == n:
                raise ValueError("Matrix is singular")

            # Swap rows
            if pivot != col:
                augmented[[col, pivot]] = augmented[[pivot, col]]

            # Normalize pivot row
            pivot_val = augmented[col, col]
            inv_pivot = self.inverse(pivot_val)

            for j in range(2 * n):
                augmented[col, j] = self.multiply(augmented[col, j], inv_pivot)

            # Eliminate other rows
            for row in range(n):
                if row != col and augmented[row, col] != 0:
                    factor = augmented[row, col]
                    for j in range(2 * n):
                        product = self.multiply(factor, augmented[col, j])
                        augmented[row, j] = self.subtract(augmented[row, j], product)

        # Extract inverse
        inverse = augmented[:, n:]

        return inverse

class ReedSolomon:
    """
    Complete Reed-Solomon erasure coding implementation
    """

    def __init__(self, k: int, m: int):
        """
        Initialize Reed-Solomon encoder/decoder

        Args:
            k: Number of data shards
            m: Number of parity shards
        """
        self.k = k
        self.m = m
        self.n = k + m

        self.gf = GaloisField(8)

        # Generate encoding matrix
        self.encoding_matrix = self.gf.vandermonde_matrix(self.n, self.k)

        # Precompute decoding matrices for different erasure patterns
        self.decoding_cache = {}

    def encode(self, data_shards: List[bytes]) -> List[bytes]:
        """
        Encode data shards into parity shards

        Args:
            data_shards: List of k data shards (bytes)

        Returns:
            List of n encoded shards (data + parity)
        """
        if len(data_shards) != self.k:
            raise ValueError(f"Expected {self.k} data shards, got {len(data_shards)}")

        # Ensure all shards have same length
        shard_size = len(data_shards[0])
        for shard in data_shards:
            if len(shard) != shard_size:
                raise ValueError("All data shards must have the same length")

        # Create parity shards
        parity_shards = [bytearray(shard_size) for _ in range(self.m)]

        # Encode each byte position
        for byte_pos in range(shard_size):
            # Prepare vector of bytes at this position
            data_vector = [shard[byte_pos] for shard in data_shards]

            # Compute parity: parity = encoding_matrix[k:] × data
            for i in range(self.m):
                parity = 0
                for j in range(self.k):
                    product = self.gf.multiply(self.encoding_matrix[self.k + i, j], 
                                             data_vector[j])
                    parity = self.gf.add(parity, product)
                parity_shards[i][byte_pos] = parity

        # Return all shards
        return data_shards + parity_shards

    def decode(self, shards: List[Optional[bytes]], shard_size: int) -> List[bytes]:
        """
        Decode original data from available shards

        Args:
            shards: List of n shards (some may be None for missing)
            shard_size: Size of each shard

        Returns:
            List of k recovered data shards
        """
        # Identify which shards are available
        available = [i for i, shard in enumerate(shards) if shard is not None]

        if len(available) < self.k:
            raise ValueError(f"Need at least {self.k} shards, have {len(available)}")

        # If we have exactly k shards and they're all data shards
        if (len(available) == self.k and 
            all(i < self.k for i in available)):
            # No decoding needed, just return data shards
            return [shards[i] for i in range(self.k)]

        # Select first k available shards
        selected = available[:self.k]

        # Create submatrix of encoding matrix for selected shards
        submatrix = self.encoding_matrix[selected, :self.k]

        # Get inverse of submatrix
        cache_key = tuple(sorted(selected))
        if cache_key in self.decoding_cache:
            inverse = self.decoding_cache[cache_key]
        else:
            inverse = self.gf.invert_matrix(submatrix)
            self.decoding_cache[cache_key] = inverse

        # Recover data shards
        data_shards = [bytearray(shard_size) for _ in range(self.k)]

        for byte_pos in range(shard_size):
            # Get bytes from selected shards
            selected_bytes = [shards[i][byte_pos] for i in selected]

            # Recover data: data = inverse × selected_bytes
            for i in range(self.k):
                recovered = 0
                for j in range(self.k):
                    product = self.gf.multiply(inverse[i, j], selected_bytes[j])
                    recovered = self.gf.add(recovered, product)
                data_shards[i][byte_pos] = recovered

        return [bytes(shard) for shard in data_shards]

    def repair(self, shards: List[Optional[bytes]], shard_size: int) -> List[bytes]:
        """
        Repair all missing shards (both data and parity)

        Returns:
            Complete list of n repaired shards
        """
        # First decode data
        data_shards = self.decode(shards, shard_size)

        # Then re-encode to get all shards
        return self.encode(data_shards)

    def analyze(self, node_failure_prob: float = 0.01) -> dict:
        """
        Analyze reliability and storage efficiency
        """
        # Storage overhead
        overhead = self.n / self.k

        # Probability system survives f failures
        survival_probs = []
        for f in range(self.m + 1):
            # Probability exactly f failures (binomial distribution)
            prob_f = (math.comb(self.n, f) * 
                     (node_failure_prob ** f) * 
                     ((1 - node_failure_prob) ** (self.n - f)))

            # System survives if f ≤ m
            if f <= self.m:
                survival_probs.append(prob_f)

        survival_prob = sum(survival_probs)

        # Annual durability (assuming independent failures)
        annual_durability = 1 - (1 - survival_prob) ** 365

        # Mean time to data loss (MTTDL)
        # Simplified model: MTTDL ≈ 1 / (prob_of_data_loss_per_year)
        prob_data_loss_per_year = 1 - annual_durability
        if prob_data_loss_per_year > 0:
            mttdl_years = 1 / prob_data_loss_per_year
        else:
            mttdl_years = float('inf')

        return {
            'data_shards': self.k,
            'parity_shards': self.m,
            'total_shards': self.n,
            'storage_overhead': overhead,
            'survivable_failures': self.m,
            'survival_probability': survival_prob,
            'annual_durability': annual_durability,
            'mttdl_years': mttdl_years,
            'efficiency': self.k / self.n
        }

class DistributedStorageSystem:
    """
    Complete distributed storage system with erasure coding
    """

    def __init__(self, k: int = 6, m: int = 3, node_count: int = 12):
        """
        Initialize distributed storage

        Args:
            k: Data shards
            m: Parity shards
            node_count: Total storage nodes
        """
        self.k = k
        self.m = m
        self.n = k + m
        self.node_count = node_count

        self.rs = ReedSolomon(k, m)
        self.nodes = [{} for _ in range(node_count)]
        self.node_status = [True] * node_count
        self.node_capacity = [10 * 1024 * 1024] * node_count  # 10MB each
        self.node_used = [0] * node_count

        # Statistics
        self.stats = {
            'files_stored': 0,
            'bytes_stored': 0,
            'shards_stored': 0,
            'repair_operations': 0,
            'failed_repairs': 0,
            'node_failures': 0
        }

    def _hash_shard_location(self, file_id: str, shard_index: int) -> int:
        """Consistent hashing for shard placement"""
        # Create unique identifier for this shard
        key = f"{file_id}:{shard_index}"

        # Use MD5 hash for even distribution
        hash_bytes = hashlib.md5(key.encode()).digest()
        hash_int = int.from_bytes(hash_bytes, 'big')

        return hash_int % self.node_count

    def _find_placement(self, file_id: str, shard_index: int, 
                       used_nodes: set) -> Optional[int]:
        """
        Find node for shard placement using consistent hashing
        with fallback for failed nodes
        """
        # Try primary location
        primary = self._hash_shard_location(file_id, shard_index)

        # Check primary node
        if (self.node_status[primary] and 
            primary not in used_nodes and
            self.node_used[primary] < self.node_capacity[primary]):
            return primary

        # Primary not available, try alternative locations
        for offset in range(1, self.node_count):
            node_idx = (primary + offset) % self.node_count

            if (self.node_status[node_idx] and 
                node_idx not in used_nodes and
                self.node_used[node_idx] < self.node_capacity[node_idx]):
                return node_idx

        return None

    def store_file(self, file_id: str, data: bytes) -> bool:
        """
        Store file with erasure coding across nodes
        """
        # Split data into k shards
        shard_size = math.ceil(len(data) / self.k)
        padded_data = data.ljust(shard_size * self.k, b'\x00')

        data_shards = []
        for i in range(self.k):
            start = i * shard_size
            end = start + shard_size
            data_shards.append(padded_data[start:end])

        # Encode to get all shards
        all_shards = self.rs.encode(data_shards)

        # Place shards on nodes
        placements = []
        used_nodes = set()

        for i, shard in enumerate(all_shards):
            node_idx = self._find_placement(file_id, i, used_nodes)

            if node_idx is None:
                # Could not find suitable node
                return False

            # Store shard
            self.nodes[node_idx][file_id] = {
                'shard': shard,
                'index': i,
                'size': len(shard),
                'is_data': i < self.k
            }

            self.node_used[node_idx] += len(shard)
            used_nodes.add(node_idx)
            placements.append(node_idx)

        # Store metadata
        metadata = {
            'file_id': file_id,
            'size': len(data),
            'shard_size': shard_size,
            'placements': placements,
            'timestamp': time.time()
        }

        # Store metadata on multiple nodes
        metadata_nodes = placements[:3]  # Store on first 3 nodes
        for node_idx in metadata_nodes:
            if 'metadata' not in self.nodes[node_idx]:
                self.nodes[node_idx]['metadata'] = {}
            self.nodes[node_idx]['metadata'][file_id] = metadata

        # Update statistics
        self.stats['files_stored'] += 1
        self.stats['bytes_stored'] += len(data)
        self.stats['shards_stored'] += len(all_shards)

        return True

    def retrieve_file(self, file_id: str) -> Optional[bytes]:
        """
        Retrieve file, tolerating node failures
        """
        # First, find metadata
        metadata = None
        for node_idx in range(self.node_count):
            if (self.node_status[node_idx] and 
                'metadata' in self.nodes[node_idx] and
                file_id in self.nodes[node_idx]['metadata']):
                metadata = self.nodes[node_idx]['metadata'][file_id]
                break

        if metadata is None:
            return None

        # Collect available shards
        shards = [None] * self.n
        shard_size = metadata['shard_size']

        for node_idx in metadata['placements']:
            if (self.node_status[node_idx] and 
                file_id in self.nodes[node_idx]):
                shard_data = self.nodes[node_idx][file_id]
                shards[shard_data['index']] = shard_data['shard']

        # Check if we have enough shards
        available = sum(1 for s in shards if s is not None)

        if available < self.k:
            # Not enough shards, try to repair first
            self._trigger_repair(file_id, metadata)
            return None

        # Decode data
        try:
            data_shards = self.rs.decode(shards, shard_size)

            # Combine data shards
            data = bytearray()
            for shard in data_shards:
                data.extend(shard)

            # Remove padding
            data = data[:metadata['size']]

            return bytes(data)

        except Exception as e:
            print(f"Decode failed: {e}")
            return None

    def _trigger_repair(self, file_id: str, metadata: dict):
        """
        Trigger repair of missing shards
        """
        self.stats['repair_operations'] += 1

        # Collect available shards
        shards = [None] * self.n
        shard_size = metadata['shard_size']

        for node_idx in metadata['placements']:
            if (self.node_status[node_idx] and 
                file_id in self.nodes[node_idx]):
                shard_data = self.nodes[node_idx][file_id]
                shards[shard_data['index']] = shard_data['shard']

        # Check if we have enough to repair
        available = sum(1 for s in shards if s is not None)

        if available < self.k:
            self.stats['failed_repairs'] += 1
            return False

        # Repair all shards
        try:
            repaired_shards = self.rs.repair(shards, shard_size)

            # Store repaired shards on new nodes if needed
            for i, shard in enumerate(repaired_shards):
                if shards[i] is None:
                    # This shard was missing, find new placement
                    node_idx = self._find_placement(file_id, i, set())

                    if node_idx is not None:
                        self.nodes[node_idx][file_id] = {
                            'shard': shard,
                            'index': i,
                            'size': len(shard),
                            'is_data': i < self.k
                        }
                        self.node_used[node_idx] += len(shard)

                        # Update placement in metadata
                        metadata['placements'][i] = node_idx

            # Update metadata on nodes
            metadata_nodes = metadata['placements'][:3]
            for node_idx in metadata_nodes:
                if 'metadata' not in self.nodes[node_idx]:
                    self.nodes[node_idx]['metadata'] = {}
                self.nodes[node_idx]['metadata'][file_id] = metadata

            return True

        except Exception as e:
            print(f"Repair failed: {e}")
            self.stats['failed_repairs'] += 1
            return False

    def simulate_node_failure(self, node_indices: List[int]):
        """
        Simulate node failures
        """
        for idx in node_indices:
            self.node_status[idx] = False
            self.stats['node_failures'] += 1

            # Clear node data
            self.nodes[idx] = {}
            self.node_used[idx] = 0

    def simulate_node_recovery(self, node_indices: List[int]):
        """
        Simulate node recovery
        """
        for idx in node_indices:
            self.node_status[idx] = True

    def get_statistics(self) -> dict:
        """
        Get system statistics
        """
        # Calculate storage efficiency
        total_capacity = sum(self.node_capacity)
        total_used = sum(self.node_used)

        # Calculate data durability estimate
        analysis = self.rs.analyze(node_failure_prob=0.01)

        return {
            **self.stats,
            'total_capacity': total_capacity,
            'total_used': total_used,
            'utilization': total_used / total_capacity,
            'expected_durability': analysis['annual_durability'],
            'storage_overhead': analysis['storage_overhead'],
            'survivable_failures': analysis['survivable_failures']
        }

    def get_system_health(self) -> dict:
        """
        Get comprehensive system health report
        """
        # Count healthy nodes
        healthy_nodes = sum(self.node_status)

        # Check files for missing shards
        files_at_risk = 0
        files_unrecoverable = 0

        # For each file, check shard availability
        all_files = set()
        for node in self.nodes:
            if 'metadata' in node:
                all_files.update(node['metadata'].keys())

        for file_id in all_files:
            # Find metadata
            metadata = None
            for node_idx in range(self.node_count):
                if (self.node_status[node_idx] and 
                    'metadata' in self.nodes[node_idx] and
                    file_id in self.nodes[node_idx]['metadata']):
                    metadata = self.nodes[node_idx]['metadata'][file_id]
                    break

            if metadata is None:
                continue

            # Count available shards
            available = 0
            for node_idx in metadata['placements']:
                if (self.node_status[node_idx] and 
                    file_id in self.nodes[node_idx]):
                    available += 1

            if available < self.k:
                files_unrecoverable += 1
            elif available < self.n:
                files_at_risk += 1

        return {
            'healthy_nodes': healthy_nodes,
            'failed_nodes': self.node_count - healthy_nodes,
            'files_at_risk': files_at_risk,
            'files_unrecoverable': files_unrecoverable,
            'total_files': len(all_files),
            'node_failure_rate': self.stats['node_failures'] / max(1, self.stats['files_stored'])
        }

def compare_storage_strategies():
    """
    Compare replication vs erasure coding
    """
    print("="*70)
    print("STORAGE STRATEGY COMPARISON: REPLICATION VS ERASURE CODING")
    print("="*70)

    # Test configurations
    data_size = 1024 * 1024  # 1MB
    test_data = b"X" * data_size

    strategies = [
        {
            'name': '3x Replication',
            'type': 'replication',
            'copies': 3,
            'storage': data_size * 3
        },
        {
            'name': 'EC (4+2)',
            'type': 'erasure',
            'k': 4,
            'm': 2,
            'storage': data_size * 6 / 4
        },
        {
            'name': 'EC (6+3)',
            'type': 'erasure',
            'k': 6,
            'm': 3,
            'storage': data_size * 9 / 6
        },
        {
            'name': 'EC (10+4)',
            'type': 'erasure',
            'k': 10,
            'm': 4,
            'storage': data_size * 14 / 10
        }
    ]

    # Node failure probability
    p_fail = 0.01

    print("\nComparison Table:")
    print("-"*120)
    print(f"{'Strategy':<15} {'Storage':<12} {'Overhead':<10} {'Survives':<10} "
          f"{'Annual Durability':<18} {'MTTDL (years)':<15} {'Efficiency':<10}")
    print("-"*120)

    for strategy in strategies:
        if strategy['type'] == 'replication':
            copies = strategy['copies']
            overhead = copies

            # Probability all copies fail
            all_fail = p_fail ** copies
            survival = 1 - all_fail

            annual_durability = 1 - (1 - survival) ** 365

            if all_fail > 0:
                mttdl = 1 / (all_fail * 365)  # Approximate
            else:
                mttdl = float('inf')

            efficiency = 1 / copies

            print(f"{strategy['name']:<15} {strategy['storage']/1024/1024:<11.2f}MB "
                  f"{overhead:<10.2f}x {copies-1:<10} "
                  f"{annual_durability:<18.6f} {mttdl:<15.2e} {efficiency:<10.2f}")

        else:
            k = strategy['k']
            m = strategy['m']
            n = k + m

            rs = ReedSolomon(k, m)
            analysis = rs.analyze(p_fail)

            print(f"{strategy['name']:<15} {strategy['storage']/1024/1024:<11.2f}MB "
                  f"{analysis['storage_overhead']:<10.2f}x {m:<10} "
                  f"{analysis['annual_durability']:<18.6f} "
                  f"{analysis['mttdl_years']:<15.2e} "
                  f"{analysis['efficiency']:<10.2f}")

    # Visual comparison
    visualize_comparison(strategies, p_fail)

def visualize_comparison(strategies, p_fail):
    """Create visualization of storage strategies"""
    import matplotlib.pyplot as plt

    fig, axes = plt.subplots(2, 2, figsize=(12, 10))

    # Prepare data
    names = [s['name'] for s in strategies]

    # Calculate metrics for each strategy
    storages = []
    durabilities = []
    efficiencies = []
    costs = []  # Cost = storage * (1 - durability)

    for strategy in strategies:
        if strategy['type'] == 'replication':
            copies = strategy['copies']
            all_fail = p_fail ** copies
            survival = 1 - all_fail
            annual_durability = 1 - (1 - survival) ** 365
            efficiency = 1 / copies

            storages.append(strategy['storage'] / 1024 / 1024)  # MB
            durabilities.append(annual_durability)
            efficiencies.append(efficiency)
            costs.append(strategy['storage'] * (1 - annual_durability))

        else:
            k = strategy['k']
            m = strategy['m']

            rs = ReedSolomon(k, m)
            analysis = rs.analyze(p_fail)

            storages.append(strategy['storage'] / 1024 / 1024)
            durabilities.append(analysis['annual_durability'])
            efficiencies.append(analysis['efficiency'])
            costs.append(strategy['storage'] * (1 - analysis['annual_durability']))

    # Plot 1: Storage vs Durability
    ax1 = axes[0, 0]
    scatter = ax1.scatter(storages, durabilities, s=100, alpha=0.6)
    ax1.set_xlabel('Storage Required (MB)')
    ax1.set_ylabel('Annual Durability')
    ax1.set_title('Storage vs Durability Tradeoff')
    ax1.grid(True, alpha=0.3)

    # Annotate points
    for i, name in enumerate(names):
        ax1.annotate(name, (storages[i], durabilities[i]), 
                    xytext=(5, 5), textcoords='offset points')

    # Plot 2: Efficiency vs Durability
    ax2 = axes[0, 1]
    ax2.scatter(efficiencies, durabilities, s=100, alpha=0.6)
    ax2.set_xlabel('Storage Efficiency')
    ax2.set_ylabel('Annual Durability')
    ax2.set_title('Efficiency vs Durability')
    ax2.grid(True, alpha=0.3)

    for i, name in enumerate(names):
        ax2.annotate(name, (efficiencies[i], durabilities[i]), 
                    xytext=(5, 5), textcoords='offset points')

    # Plot 3: Cost Analysis
    ax3 = axes[1, 0]
    bars = ax3.bar(names, [c / 1024 / 1024 for c in costs])
    ax3.set_xlabel('Strategy')
    ax3.set_ylabel('Expected Loss (MB)')
    ax3.set_title('Expected Data Loss Cost')
    ax3.tick_params(axis='x', rotation=45)
    ax3.grid(True, alpha=0.3, axis='y')

    # Add value labels on bars
    for bar in bars:
        height = bar.get_height()
        ax3.text(bar.get_x() + bar.get_width()/2., height,
                f'{height:.2f}', ha='center', va='bottom')

    # Plot 4: Survival probability vs failures
    ax4 = axes[1, 1]

    # Simulate different numbers of failures
    max_failures = 10
    failure_counts = list(range(max_failures + 1))

    for strategy in strategies[:2]:  # Just show first two for clarity
        survival_probs = []

        for f in failure_counts:
            if strategy['type'] == 'replication':
                # Replication survives if at least 1 copy survives
                copies = strategy['copies']
                prob_survive = 1 - (p_fail ** min(f, copies))
                survival_probs.append(prob_survive)

            else:
                # Erasure coding survives if f ≤ m
                k = strategy['k']
                m = strategy['m']
                if f <= m:
                    survival_probs.append(1.0)
                else:
                    survival_probs.append(0.0)

        ax4.plot(failure_counts, survival_probs, 
                label=strategy['name'], linewidth=2)

    ax4.set_xlabel('Number of Node Failures')
    ax4.set_ylabel('Survival Probability')
    ax4.set_title('Failure Tolerance')
    ax4.legend()
    ax4.grid(True, alpha=0.3)

    plt.tight_layout()
    plt.show()

def run_distributed_storage_simulation():
    """
    Simulate complete distributed storage system
    """
    print("\n" + "="*70)
    print("DISTRIBUTED STORAGE SIMULATION")
    print("="*70)

    # Create storage system
    storage = DistributedStorageSystem(k=4, m=2, node_count=10)

    print("\n1. Storing files...")

    # Store some files
    files = [
        ("doc1.txt", b"This is document 1 with important content."),
        ("img1.jpg", b"FAKE_IMAGE_DATA" * 100),  # 1.6KB
        ("data.bin", b"\x00\x01\x02\x03" * 500),  # 2KB
    ]

    for file_id, data in files:
        success = storage.store_file(file_id, data)
        print(f"  Stored {file_id} ({len(data)} bytes): {'✓' if success else '✗'}")

    print("\n2. Retrieving files normally...")
    for file_id, _ in files:
        data = storage.retrieve_file(file_id)
        if data:
            print(f"  Retrieved {file_id}: {len(data)} bytes")
        else:
            print(f"  Failed to retrieve {file_id}")

    print("\n3. Simulating node failures...")
    # Fail 3 random nodes
    failed_nodes = random.sample(range(10), 3)
    storage.simulate_node_failure(failed_nodes)
    print(f"  Failed nodes: {failed_nodes}")

    print("\n4. Retrieving after failures...")
    for file_id, _ in files:
        data = storage.retrieve_file(file_id)
        if data:
            print(f"  Retrieved {file_id} after failures: ✓")
        else:
            print(f"  Failed to retrieve {file_id}: ✗")

    print("\n5. System health check...")
    health = storage.get_system_health()
    for key, value in health.items():
        print(f"  {key}: {value}")

    print("\n6. Triggering repair...")
    # Try to repair missing shards
    for file_id, _ in files:
        # Find metadata to trigger repair
        for node_idx in range(10):
            if (storage.node_status[node_idx] and 
                'metadata' in storage.nodes[node_idx] and
                file_id in storage.nodes[node_idx]['metadata']):
                metadata = storage.nodes[node_idx]['metadata'][file_id]
                storage._trigger_repair(file_id, metadata)
                break

    print("\n7. Statistics:")
    stats = storage.get_statistics()
    for key, value in stats.items():
        print(f"  {key}: {value}")

def demonstrate_reed_solomon_math():
    """
    Demonstrate Reed-Solomon mathematics step by step
    """
    print("\n" + "="*70)
    print("REED-SOLOMON MATHEMATICS DEMONSTRATION")
    print("="*70)

    # Create Galois Field
    gf = GaloisField(8)

    print("\n1. Galois Field GF(2^8) operations:")
    a, b = 5, 10
    print(f"   {a} + {b} = {gf.add(a, b)} (XOR)")
    print(f"   {a} × {b} = {gf.multiply(a, b)}")
    print(f"   {a}⁻¹ = {gf.inverse(a)}")
    print(f"   Verify: {a} × {a}⁻¹ = {gf.multiply(a, gf.inverse(a))} (should be 1)")

    print("\n2. Polynomial evaluation:")
    coeffs = [1, 2, 3]  # Represents 1 + 2x + 3x²
    x = 4
    result = gf.eval_polynomial(coeffs, x)
    print(f"   P(x) = 1 + 2x + 3x²")
    print(f"   P({x}) = {result}")

    print("\n3. Lagrange interpolation:")
    points = [(1, 2), (2, 3), (3, 5)]  # (x, y) points
    coeffs = gf.interpolate(points)
    print(f"   Points: {points}")
    print(f"   Interpolated polynomial coefficients: {coeffs}")

    # Verify
    print("\n   Verification:")
    for x, y in points:
        computed = gf.eval_polynomial(coeffs, x)
        print(f"   P({x}) = {computed} (expected {y}) {'✓' if computed == y else '✗'}")

    print("\n4. Vandermonde matrix for (4,2) Reed-Solomon:")
    rs = ReedSolomon(4, 2)
    print("   Encoding matrix (6×4):")
    print(rs.encoding_matrix)

    print("\n5. Encoding example:")
    data = [1, 2, 3, 4]  # Simple data
    print(f"   Data: {data}")

    # Encode
    encoded = []
    for i in range(6):
        row_sum = 0
        for j in range(4):
            row_sum = gf.add(row_sum, gf.multiply(rs.encoding_matrix[i, j], data[j]))
        encoded.append(row_sum)

    print(f"   Encoded: {encoded}")

    print("\n6. Decoding with erasures:")
    # Simulate losing shards 0 and 2
    received = encoded.copy()
    received[0] = None
    received[2] = None
    print(f"   Received (with erasures): {received}")

    # Use remaining shards 1, 3, 4, 5
    available = [1, 3, 4, 5]
    submatrix = rs.encoding_matrix[available, :4]
    print(f"   Submatrix from available shards:")
    print(submatrix)

    # Invert submatrix
    inverse = gf.invert_matrix(submatrix)
    print(f"   Inverse of submatrix:")
    print(inverse)

    # Recover data
    received_data = [received[i] for i in available]
    recovered = [0, 0, 0, 0]

    for i in range(4):
        for j in range(4):
            recovered[i] = gf.add(recovered[i], 
                                gf.multiply(inverse[i, j], received_data[j]))

    print(f"   Recovered data: {recovered}")
    print(f"   Original data:  {data}")
    print(f"   Match: {'✓' if recovered == data else '✗'}")

if __name__ == "__main__":
    compare_storage_strategies()
    run_distributed_storage_simulation()
    demonstrate_reed_solomon_math()

5. Byzantine Agreement: Complete Implementation with Proofs

The Byzantine Generals Problem: Mathematical Foundation

Problem Statement: N generals, some of which may be traitors (Byzantine), need to agree on a common plan of action.

Impossibility Results:

Theorem (Lamport 1982): With 3 generals and 1 traitor, no solution exists.
Theorem: Solution exists iff N ≥ 3f + 1, where f is number of traitors.

Why 3f + 1?

f traitors could lie
f non-responding (slow/crashed)
Need f+1 honest, responding generals to outnumber traitors

Complete C++ Implementation: Practical Byzantine Fault Tolerance (PBFT)

#include <iostream>
#include <vector>
#include <map>
#include <set>
#include <string>
#include <memory>
#include <random>
#include <chrono>
#include <algorithm>
#include <openssl/sha.h>
#include <openssl/evp.h>
#include <iomanip>
#include <queue>
#include <thread>
#include <mutex>
#include <condition_variable>
#include <atomic>

// ============================================================================
// CRYPTOGRAPHIC PRIMITIVES
// ============================================================================

class CryptographicHash {
public:
    static std::string sha256(const std::string& data) {
        unsigned char hash[SHA256_DIGEST_LENGTH];
        SHA256_CTX sha256;

        SHA256_Init(&sha256);
        SHA256_Update(&sha256, data.c_str(), data.size());
        SHA256_Final(hash, &sha256);

        std::stringstream ss;
        for(int i = 0; i < SHA256_DIGEST_LENGTH; i++) {
            ss << std::hex << std::setw(2) << std::setfill('0') 
               << static_cast<int>(hash[i]);
        }

        return ss.str();
    }

    static std::string sign(const std::string& data, int node_id) {
        // Simplified signing: hash(data + secret)
        std::string secret = "node_secret_" + std::to_string(node_id);
        return sha256(data + secret);
    }

    static bool verify(const std::string& data, 
                      const std::string& signature, 
                      int node_id) {
        std::string computed = sign(data, node_id);
        return computed == signature;
    }
};

// ============================================================================
// MESSAGE TYPES AND STRUCTURES
// ============================================================================

struct Message {
    enum Type {
        REQUEST,        // Client request
        PREPREPARE,     // Leader proposes
        PREPARE,        // Replicas prepare
        COMMIT,         // Replicas commit
        REPLY,          // Reply to client
        VIEW_CHANGE,    // View change protocol
        NEW_VIEW        // New view establishment
    };

    Type type;
    int view_number;
    int sequence_number;
    std::string digest;
    std::string content;
    int sender_id;
    std::string signature;
    std::string client_id;
    uint64_t timestamp;

    Message() = default;

    Message(Type t, int view, int seq, const std::string& d, 
            const std::string& c, int sender)
        : type(t), view_number(view), sequence_number(seq),
          digest(d), content(c), sender_id(sender),
          timestamp(std::chrono::system_clock::now().time_since_epoch().count()) {}

    std::string hash() const {
        std::string data = 
            std::to_string(static_cast<int>(type)) +
            std::to_string(view_number) +
            std::to_string(sequence_number) +
            digest +
            content +
            std::to_string(sender_id) +
            std::to_string(timestamp);

        return CryptographicHash::sha256(data);
    }

    void sign_message(int node_id) {
        std::string data = hash();
        signature = CryptographicHash::sign(data, node_id);
    }

    bool verify_signature() const {
        std::string data = hash();
        return CryptographicHash::verify(data, signature, sender_id);
    }

    std::string to_string() const {
        std::string type_str;
        switch(type) {
            case REQUEST: type_str = "REQUEST"; break;
            case PREPREPARE: type_str = "PREPREPARE"; break;
            case PREPARE: type_str = "PREPARE"; break;
            case COMMIT: type_str = "COMMIT"; break;
            case REPLY: type_str = "REPLY"; break;
            case VIEW_CHANGE: type_str = "VIEW_CHANGE"; break;
            case NEW_VIEW: type_str = "NEW_VIEW"; break;
        }

        return "Message{" + type_str + 
               ", view=" + std::to_string(view_number) +
               ", seq=" + std::to_string(sequence_number) +
               ", digest=" + digest.substr(0, 8) + "..." +
               ", sender=" + std::to_string(sender_id) + "}";
    }
};

// ============================================================================
// PBFT STATE MACHINE
// ============================================================================

class PBFTState {
private:
    int node_id;
    int total_nodes;
    int max_faulty;

    // Current state
    int current_view;
    int last_executed;
    int last_stable_checkpoint;

    // Message logs
    std::map<int, std::map<std::string, std::set<int>>> prepare_certificates;
    std::map<int, std::map<std::string, std::set<int>>> commit_certificates;

    // Checkpoints
    std::map<int, std::string> checkpoints;  // sequence -> state_hash

    // Client requests
    std::map<std::string, std::pair<int, Message>> client_requests;

    // Pending requests
    std::map<int, Message> pending_requests;

    // View change state
    bool view_change_pending;
    std::map<int, std::set<int>> view_change_certificates;

    // Watermarks
    int low_watermark;
    int high_watermark;

    std::mutex state_mutex;

public:
    PBFTState(int id, int n)
        : node_id(id), total_nodes(n), max_faulty((n - 1) / 3),
          current_view(0), last_executed(0), last_stable_checkpoint(0),
          view_change_pending(false), low_watermark(0), high_watermark(100) {

        if (n <= 3 * max_faulty) {
            throw std::runtime_error("N must be > 3f for Byzantine tolerance");
        }
    }

    bool is_primary() const {
        return current_view % total_nodes == node_id;
    }

    int primary_of_view(int view) const {
        return view % total_nodes;
    }

    bool can_prepare(const Message& msg) {
        std::lock_guard<std::mutex> lock(state_mutex);

        // Check view number
        if (msg.view_number != current_view) {
            return false;
        }

        // Check sequence number is within watermarks
        if (msg.sequence_number <= low_watermark || 
            msg.sequence_number > high_watermark) {
            return false;
        }

        // Check if we already prepared for this sequence
        auto it = prepare_certificates.find(msg.sequence_number);
        if (it != prepare_certificates.end()) {
            auto& certs = it->second;
            if (certs.find(msg.digest) != certs.end()) {
                // Already prepared this digest
                return false;
            }
        }

        return true;
    }

    void add_prepare(const Message& msg) {
        std::lock_guard<std::mutex> lock(state_mutex);

        prepare_certificates[msg.sequence_number][msg.digest].insert(msg.sender_id);
    }

    bool has_prepare_certificate(int sequence, const std::string& digest) {
        std::lock_guard<std::mutex> lock(state_mutex);

        auto seq_it = prepare_certificates.find(sequence);
        if (seq_it == prepare_certificates.end()) {
            return false;
        }

        auto& certs = seq_it->second;
        auto cert_it = certs.find(digest);
        if (cert_it == certs.end()) {
            return false;
        }

        // Need 2f matching prepares (excluding our own)
        int count = cert_it->second.size();
        if (cert_it->second.find(node_id) != cert_it->second.end()) {
            count--;  // Don't count our own
        }

        return count >= 2 * max_faulty;
    }

    void add_commit(const Message& msg) {
        std::lock_guard<std::mutex> lock(state_mutex);

        commit_certificates[msg.sequence_number][msg.digest].insert(msg.sender_id);
    }

    bool has_commit_certificate(int sequence, const std::string& digest) {
        std::lock_guard<std::mutex> lock(state_mutex);

        auto seq_it = commit_certificates.find(sequence);
        if (seq_it == commit_certificates.end()) {
            return false;
        }

        auto& certs = seq_it->second;
        auto cert_it = certs.find(digest);
        if (cert_it == certs.end()) {
            return false;
        }

        // Need 2f+1 matching commits
        return cert_it->second.size() >= 2 * max_faulty + 1;
    }

    bool can_execute(int sequence) {
        std::lock_guard<std::mutex> lock(state_mutex);
        return sequence == last_executed + 1;
    }

    void mark_executed(int sequence) {
        std::lock_guard<std::mutex> lock(state_mutex);
        if (sequence == last_executed + 1) {
            last_executed = sequence;

            // Create checkpoint every 100 requests
            if (sequence % 100 == 0) {
                create_checkpoint(sequence);
            }
        }
    }

    void create_checkpoint(int sequence) {
        // In real implementation, this would save application state
        std::string state_hash = CryptographicHash::sha256(
            "checkpoint_at_" + std::to_string(sequence));

        checkpoints[sequence] = state_hash;
        last_stable_checkpoint = sequence;

        // Move watermarks
        low_watermark = last_stable_checkpoint;
        high_watermark = low_watermark + 100;
    }

    void start_view_change(int new_view) {
        std::lock_guard<std::mutex> lock(state_mutex);
        view_change_pending = true;
        current_view = new_view;
    }

    void add_view_change(int view, int node) {
        std::lock_guard<std::mutex> lock(state_mutex);
        view_change_certificates[view].insert(node);
    }

    bool has_view_change_certificate(int view) {
        std::lock_guard<std::mutex> lock(state_mutex);
        auto it = view_change_certificates.find(view);
        if (it == view_change_certificates.end()) {
            return false;
        }
        return it->second.size() >= 2 * max_faulty + 1;
    }

    int get_current_view() const { return current_view; }
    int get_last_executed() const { return last_executed; }
    int get_low_watermark() const { return low_watermark; }
    int get_high_watermark() const { return high_watermark; }
    int get_total_nodes() const { return total_nodes; }
    int get_max_faulty() const { return max_faulty; }
};

// ============================================================================
// PBFT NODE IMPLEMENTATION
// ============================================================================

class PBFTNode {
private:
    int node_id;
    bool byzantine;
    PBFTState state;

    // Network
    std::function<void(const Message&)> broadcast_callback;
    std::function<void(int, const Message&)> send_callback;

    // Request queue
    std::queue<Message> request_queue;
    std::mutex queue_mutex;
    std::condition_variable queue_cv;

    // Worker thread
    std::thread worker_thread;
    std::atomic<bool> running;

    // Byzantine behavior
    std::mt19937 rng;
    std::uniform_real_distribution<double> dist;

    // Statistics
    struct Stats {
        int messages_received = 0;
        int messages_sent = 0;
        int requests_executed = 0;
        int byzantine_actions = 0;
        int view_changes = 0;
    } stats;

public:
    PBFTNode(int id, int total_nodes, bool is_byzantine = false)
        : node_id(id), byzantine(is_byzantine), 
          state(id, total_nodes), running(false),
          rng(std::chrono::system_clock::now().time_since_epoch().count()),
          dist(0.0, 1.0) {}

    ~PBFTNode() {
        stop();
    }

    void set_network_callbacks(
        std::function<void(const Message&)> broadcast,
        std::function<void(int, const Message&)> send) {

        broadcast_callback = broadcast;
        send_callback = send;
    }

    void start() {
        running = true;
        worker_thread = std::thread(&PBFTNode::process_loop, this);
    }

    void stop() {
        running = false;
        queue_cv.notify_all();
        if (worker_thread.joinable()) {
            worker_thread.join();
        }
    }

    void receive_message(const Message& msg) {
        if (!msg.verify_signature()) {
            std::cerr << "Node " << node_id << ": Invalid signature from " 
                      << msg.sender_id << std::endl;
            return;
        }

        stats.messages_received++;

        // Byzantine nodes might drop messages
        if (byzantine && dist(rng) < 0.2) {  // 20% chance to drop
            stats.byzantine_actions++;
            return;
        }

        std::lock_guard<std::mutex> lock(queue_mutex);
        request_queue.push(msg);
        queue_cv.notify_one();
    }

    void process_loop() {
        while (running) {
            Message msg;
            {
                std::unique_lock<std::mutex> lock(queue_mutex);
                queue_cv.wait(lock, [this]() { 
                    return !request_queue.empty() || !running; 
                });

                if (!running) break;

                msg = request_queue.front();
                request_queue.pop();
            }

            process_message(msg);
        }
    }

    void process_message(const Message& msg) {
        switch (msg.type) {
            case Message::REQUEST:
                handle_request(msg);
                break;
            case Message::PREPREPARE:
                handle_preprepare(msg);
                break;
            case Message::PREPARE:
                handle_prepare(msg);
                break;
            case Message::COMMIT:
                handle_commit(msg);
                break;
            case Message::VIEW_CHANGE:
                handle_view_change(msg);
                break;
            case Message::NEW_VIEW:
                handle_new_view(msg);
                break;
            default:
                std::cerr << "Unknown message type: " << msg.type << std::endl;
        }
    }

    void handle_request(const Message& msg) {
        if (!state.is_primary()) {
            // Forward to primary
            int primary = state.primary_of_view(state.get_current_view());
            if (send_callback) {
                send_callback(primary, msg);
            }
            return;
        }

        // Primary assigns sequence number
        int sequence = state.get_last_executed() + 1;

        // Create PRE-PREPARE message
        Message preprepare(Message::PREPREPARE,
                          state.get_current_view(),
                          sequence,
                          msg.digest,
                          msg.content,
                          node_id);

        preprepare.sign_message(node_id);

        // Byzantine primary might assign wrong sequence
        if (byzantine && dist(rng) < 0.3) {
            stats.byzantine_actions++;
            preprepare.sequence_number = sequence + 100;  // Wrong sequence
        }

        if (broadcast_callback) {
            broadcast_callback(preprepare);
            stats.messages_sent++;
        }
    }

    void handle_preprepare(const Message& msg) {
        // Verify pre-prepare
        if (msg.view_number != state.get_current_view()) {
            return;
        }

        if (state.primary_of_view(msg.view_number) != msg.sender_id) {
            return;
        }

        if (!state.can_prepare(msg)) {
            return;
        }

        // Create PREPARE message
        Message prepare(Message::PREPARE,
                       msg.view_number,
                       msg.sequence_number,
                       msg.digest,
                       msg.content,
                       node_id);

        prepare.sign_message(node_id);

        // Byzantine nodes might send wrong digest
        if (byzantine && dist(rng) < 0.3) {
            stats.byzantine_actions++;
            prepare.digest = "WRONG_DIGEST";
        }

        if (broadcast_callback) {
            broadcast_callback(prepare);
            stats.messages_sent++;
        }

        state.add_prepare(msg);
    }

    void handle_prepare(const Message& msg) {
        state.add_prepare(msg);

        // Check if we have prepare certificate
        if (state.has_prepare_certificate(msg.sequence_number, msg.digest)) {
            // Create COMMIT message
            Message commit(Message::COMMIT,
                          msg.view_number,
                          msg.sequence_number,
                          msg.digest,
                          msg.content,
                          node_id);

            commit.sign_message(node_id);

            if (broadcast_callback) {
                broadcast_callback(commit);
                stats.messages_sent++;
            }
        }
    }

    void handle_commit(const Message& msg) {
        state.add_commit(msg);

        // Check if we have commit certificate
        if (state.has_commit_certificate(msg.sequence_number, msg.digest)) {
            // Check if we can execute
            if (state.can_execute(msg.sequence_number)) {
                execute_request(msg);
                state.mark_executed(msg.sequence_number);
                stats.requests_executed++;
            }
        }
    }

    void execute_request(const Message& msg) {
        std::cout << "Node " << node_id << " executing request: " 
                  << msg.sequence_number << " - " << msg.content << std::endl;

        // In real implementation, this would execute the state machine operation

        // Send reply to client
        Message reply(Message::REPLY,
                     msg.view_number,
                     msg.sequence_number,
                     msg.digest,
                     "EXECUTED: " + msg.content,
                     node_id);

        reply.client_id = msg.client_id;
        reply.sign_message(node_id);

        // Send to client (simplified)
        if (send_callback) {
            // Client ID 0 for simplicity
            send_callback(0, reply);
        }
    }

    void handle_view_change(const Message& msg) {
        // Start view change if we haven't already
        if (msg.view_number > state.get_current_view()) {
            state.start_view_change(msg.view_number);
            stats.view_changes++;
        }

        state.add_view_change(msg.view_number, msg.sender_id);

        // Check if we have enough view change messages
        if (state.has_view_change_certificate(msg.view_number)) {
            // New primary sends NEW-VIEW
            if (state.primary_of_view(msg.view_number) == node_id) {
                Message new_view(Message::NEW_VIEW,
                                msg.view_number,
                                0,  // sequence doesn't matter
                                "",
                                "NEW_VIEW_CONTENT",
                                node_id);

                new_view.sign_message(node_id);

                if (broadcast_callback) {
                    broadcast_callback(new_view);
                }
            }
        }
    }

    void handle_new_view(const Message& msg) {
        // Verify new view comes from primary of that view
        if (state.primary_of_view(msg.view_number) != msg.sender_id) {
            return;
        }

        // Update our view
        state.start_view_change(msg.view_number);

        std::cout << "Node " << node_id << " moving to view " 
                  << msg.view_number << std::endl;
    }

    const Stats& get_stats() const { return stats; }
    int get_id() const { return node_id; }
    bool is_byzantine() const { return byzantine; }
};

// ============================================================================
// PBFT NETWORK SIMULATION
// ============================================================================

class PBFTNetwork {
private:
    std::vector<std::unique_ptr<PBFTNode>> nodes;
    std::map<int, std::vector<Message>> message_queues;
    std::mutex network_mutex;

    int total_nodes;
    int byzantine_count;

    // Statistics
    struct NetworkStats {
        int total_messages = 0;
        int delivered_messages = 0;
        int dropped_messages = 0;
        std::map<int, int> messages_by_type;
        std::chrono::steady_clock::time_point start_time;
    } stats;

public:
    PBFTNetwork(int n, int byzantine) 
        : total_nodes(n), byzantine_count(byzantine) {

        if (n <= 3 * byzantine) {
            throw std::runtime_error("N must be > 3f for Byzantine tolerance");
        }

        // Create nodes
        for (int i = 0; i < n; i++) {
            bool is_byzantine = i < byzantine;
            auto node = std::make_unique<PBFTNode>(i, n, is_byzantine);

            // Set callbacks
            node->set_network_callbacks(
                [this, i](const Message& msg) {
                    // Broadcast
                    for (int j = 0; j < total_nodes; j++) {
                        if (j != i) {
                            deliver_message(j, msg);
                        }
                    }
                },
                [this, i](int target, const Message& msg) {
                    // Send to specific node
                    if (target >= 0 && target < total_nodes && target != i) {
                        deliver_message(target, msg);
                    }
                }
            );

            nodes.push_back(std::move(node));
        }

        stats.start_time = std::chrono::steady_clock::now();
        message_queues.resize(n);
    }

    void deliver_message(int target, const Message& msg) {
        std::lock_guard<std::mutex> lock(network_mutex);

        stats.total_messages++;
        stats.messages_by_type[msg.type]++;

        // Simulate network delays (0-50ms)
        std::this_thread::sleep_for(
            std::chrono::milliseconds(rand() % 50));

        // Simulate message loss (2%)
        if (rand() % 100 < 2) {
            stats.dropped_messages++;
            return;
        }

        message_queues[target].push_back(msg);
    }

    void process_messages() {
        std::lock_guard<std::mutex> lock(network_mutex);

        for (int i = 0; i < total_nodes; i++) {
            for (const auto& msg : message_queues[i]) {
                nodes[i]->receive_message(msg);
                stats.delivered_messages++;
            }
            message_queues[i].clear();
        }
    }

    void broadcast_client_request(const std::string& request) {
        Message msg(Message::REQUEST,
                   0,  // view
                   0,  // sequence (client doesn't know)
                   CryptographicHash::sha256(request),
                   request,
                   -1);  // client

        msg.client_id = "client_0";
        msg.sign_message(-1);

        // Send to all nodes
        for (int i = 0; i < total_nodes; i++) {
            deliver_message(i, msg);
        }
    }

    void run_consensus(int num_requests) {
        std::cout << "\nStarting PBFT consensus with " << total_nodes 
                  << " nodes (" << byzantine_count << " Byzantine)" << std::endl;

        // Start all nodes
        for (auto& node : nodes) {
            node->start();
        }

        // Send client requests
        for (int i = 0; i < num_requests; i++) {
            std::string request = "REQUEST_" + std::to_string(i) + 
                                 "_" + std::to_string(rand());

            std::cout << "\nSending request: " << request << std::endl;
            broadcast_client_request(request);

            // Process messages
            for (int j = 0; j < 5; j++) {  // Multiple rounds
                process_messages();
                std::this_thread::sleep_for(std::chrono::milliseconds(100));
            }
        }

        // Stop nodes
        for (auto& node : nodes) {
            node->stop();
        }

        print_statistics();
    }

    void print_statistics() {
        auto end_time = std::chrono::steady_clock::now();
        auto duration = std::chrono::duration_cast<std::chrono::milliseconds>(
            end_time - stats.start_time);

        std::cout << "\n" << std::string(60, '=') << std::endl;
        std::cout << "PBFT SIMULATION STATISTICS" << std::endl;
        std::cout << std::string(60, '=') << std::endl;

        std::cout << "\nNetwork Statistics:" << std::endl;
        std::cout << "  Total messages: " << stats.total_messages << std::endl;
        std::cout << "  Delivered messages: " << stats.delivered_messages << std::endl;
        std::cout << "  Dropped messages: " << stats.dropped_messages 
                  << " (" << (stats.dropped_messages * 100.0 / stats.total_messages) 
                  << "%)" << std::endl;
        std::cout << "  Duration: " << duration.count() << "ms" << std::endl;

        std::cout << "\nMessage types:" << std::endl;
        for (auto& [type, count] : stats.messages_by_type) {
            std::string type_str;
            switch(type) {
                case Message::REQUEST: type_str = "REQUEST"; break;
                case Message::PREPREPARE: type_str = "PREPREPARE"; break;
                case Message::PREPARE: type_str = "PREPARE"; break;
                case Message::COMMIT: type_str = "COMMIT"; break;
                case Message::REPLY: type_str = "REPLY"; break;
                case Message::VIEW_CHANGE: type_str = "VIEW_CHANGE"; break;
                case Message::NEW_VIEW: type_str = "NEW_VIEW"; break;
            }
            std::cout << "  " << type_str << ": " << count << std::endl;
        }

        std::cout << "\nNode Statistics:" << std::endl;
        for (auto& node : nodes) {
            auto node_stats = node->get_stats();
            std::cout << "  Node " << node->get_id() 
                      << (node->is_byzantine() ? " (Byzantine)" : " (Honest)") 
                      << ":" << std::endl;
            std::cout << "    Messages received: " << node_stats.messages_received << std::endl;
            std::cout << "    Messages sent: " << node_stats.messages_sent << std::endl;
            std::cout << "    Requests executed: " << node_stats.requests_executed << std::endl;
            std::cout << "    Byzantine actions: " << node_stats.byzantine_actions << std::endl;
            std::cout << "    View changes: " << node_stats.view_changes << std::endl;
        }
    }
};

// ============================================================================
// BYZANTINE AGREEMENT PROOFS AND ANALYSIS
// ============================================================================

void prove_byzantine_agreement() {
    std::cout << "=" << std::string(70, '=') << std::endl;
    std::cout << "BYZANTINE AGREEMENT: MATHEMATICAL PROOFS" << std::endl;
    std::cout << "=" << std::string(70, '=') << std::endl;

    std::cout << "\nTheorem 1: With 3 generals and 1 traitor, agreement is impossible" << std::endl;
    std::cout << "Proof (by contradiction):" << std::endl;
    std::cout << "  1. Assume generals A, B, C where C is traitor." << std::endl;
    std::cout << "  2. A sends 'attack' to B and C." << std::endl;
    std::cout << "  3. C tells B that A said 'retreat' (lying)." << std::endl;
    std::cout << "  4. B hears 'attack' from A and 'retreat' from C." << std::endl;
    std::cout << "  5. B cannot determine who is lying: A or C?" << std::endl;
    std::cout << "  6. Therefore, B cannot decide consistently with A. ✓" << std::endl;

    std::cout << "\nTheorem 2: Agreement is possible iff N ≥ 3f + 1" << std::endl;
    std::cout << "Proof:" << std::endl;
    std::cout << "  Necessity (N ≥ 3f + 1):" << std::endl;
    std::cout << "    1. Divide N nodes into 3 groups: honest (H), faulty (F), slow (S)." << std::endl;
    std::cout << "    2. Worst case: |F| = f, |S| = f (non-responding)." << std::endl;
    std::cout << "    3. Need |H| ≥ f + 1 to outnumber faulty nodes." << std::endl;
    std::cout << "    4. Therefore: N = |H| + |F| + |S| ≥ (f + 1) + f + f = 3f + 1. ✓" << std::endl;

    std::cout << "\n  Sufficiency (algorithm exists for N ≥ 3f + 1):" << std::endl;
    std::cout << "    1. PBFT protocol provides safety and liveness." << std::endl;
    std::cout << "    2. Safety: Honest nodes agree on order of requests." << std::endl;
    std::cout << "    3. Liveness: Client eventually receives reply." << std::endl;
    std::cout << "    4. Proof via quorum intersection: any two quorums of size 2f+1" << std::endl;
    std::cout << "       intersect in at least f+1 honest nodes. ✓" << std::endl;

    std::cout << "\nTheorem 3: Optimal resilience is f < N/3" << std::endl;
    std::cout << "Proof:" << std::endl;
    std::cout << "  1. Assume f ≥ N/3." << std::endl;
    std::cout << "  2. Then N ≤ 3f." << std::endl;
    std::cout << "  3. From Theorem 2, agreement is impossible." << std::endl;
    std::cout << "  4. Therefore, maximum tolerable faults is f < N/3. ✓" << std::endl;

    std::cout << "\nTheorem 4: PBFT provides linearizability" << std::endl;
    std::cout << "Proof:" << std::endl;
    std::cout << "  1. All operations are totally ordered by sequence numbers." << std::endl;
    std::cout << "  2. Commit certificates ensure all honest nodes execute in same order." << std::endl;
    std::cout << "  3. View changes preserve sequence number ordering." << std::endl;
    std::cout << "  4. Therefore, execution is equivalent to a centralized system. ✓" << std::endl;
}

void analyze_byzantine_scenarios() {
    std::cout << "\n" << std::string(70, '=') << std::endl;
    std::cout << "BYZANTINE FAILURE SCENARIO ANALYSIS" << std::endl;
    std::cout << std::string(70, '=') << std::endl;

    // Test different configurations
    struct Scenario {
        int total_nodes;
        int byzantine_nodes;
        bool should_succeed;
        std::string description;
    };

    std::vector<Scenario> scenarios = {
        {4, 1, true, "Minimal configuration (N=4, f=1)"},
        {7, 2, true, "Typical configuration (N=7, f=2)"},
        {10, 3, true, "Large configuration (N=10, f=3)"},
        {3, 1, false, "Impossible case (N=3, f=1)"},
        {6, 2, true, "Boundary case (N=6, f=2)"},
        {5, 2, false, "Too many faults (N=5, f=2)"},
    };

    std::cout << "\nScenario Analysis:" << std::endl;
    std::cout << std::string(80, '-') << std::endl;
    std::cout << std::setw(20) << "Configuration" 
              << std::setw(15) << "N ≥ 3f+1" 
              << std::setw(15) << "Expected" 
              << std::setw(30) << "Description" << std::endl;
    std::cout << std::string(80, '-') << std::endl;

    for (const auto& scenario : scenarios) {
        bool condition = scenario.total_nodes >= 3 * scenario.byzantine_nodes + 1;

        std::cout << std::setw(10) << "N=" + std::to_string(scenario.total_nodes)
                  << std::setw(10) << "f=" + std::to_string(scenario.byzantine_nodes)
                  << std::setw(15) << (condition ? "Yes" : "No")
                  << std::setw(15) << (scenario.should_succeed ? "Success" : "Failure")
                  << std::setw(30) << scenario.description << std::endl;
    }

    // Mathematical analysis
    std::cout << "\nMathematical Analysis:" << std::endl;
    std::cout << std::string(80, '-') << std::endl;

    for (int N = 3; N <= 10; N++) {
        int max_f = (N - 1) / 3;
        double resilience = max_f * 100.0 / N;

        std::cout << "N=" << N << ": Max f=" << max_f 
                  << " (" << resilience << "% nodes can be Byzantine)"
                  << std::endl;
    }
}

void demonstrate_pbft_phases() {
    std::cout << "\n" << std::string(70, '=') << std::endl;
    std::cout << "PBFT PROTOCOL PHASES DEMONSTRATION" << std::endl;
    std::cout << std::string(70, '=') << std::endl;

    // Create a minimal PBFT network
    int N = 4;
    int f = 1;

    std::cout << "\nNormal Case Operation (N=" << N << ", f=" << f << "):" << std::endl;
    std::cout << "1. Client sends request to all replicas" << std::endl;
    std::cout << "2. Primary (replica 0) assigns sequence number" << std::endl;
    std::cout << "3. Primary broadcasts PRE-PREPARE" << std::endl;
    std::cout << "4. Replicas broadcast PREPARE after verifying" << std::endl;
    std::cout << "5. After 2f PREPARE messages, replicas broadcast COMMIT" << std::endl;
    std::cout << "6. After 2f+1 COMMIT messages, replicas execute request" << std::endl;
    std::cout << "7. Replicas send REPLY to client" << std::endl;
    std::cout << "8. Client waits for f+1 matching replies" << std::endl;

    std::cout << "\nView Change Protocol:" << std::endl;
    std::cout << "1. Replica suspects primary is faulty (timeout)" << std::endl;
    std::cout << "2. Replica broadcasts VIEW-CHANGE to new view" << std::endl;
    std::cout << "3. After 2f+1 VIEW-CHANGE messages, new primary is elected" << std::endl;
    std::cout << "4. New primary broadcasts NEW-VIEW with checkpoint" << std::endl;
    std::cout << "5. Replicas move to new view and resume operation" << std::endl;

    std::cout << "\nSafety Argument:" << std::endl;
    std::cout << "- Any two quorums of size 2f+1 intersect in at least f+1 honest nodes" << std::endl;
    std::cout << "- These honest nodes ensure consistent ordering" << std::endl;
    std::cout << "- View changes preserve ordering through sequence numbers" << std::endl;

    std::cout << "\nLiveness Argument:" << std::endl;
    std::cout << "- Eventually, a view with non-faulty primary will last long enough" << std::endl;
    std::cout << "- Timeouts ensure progress despite faulty primaries" << std::endl;
    std::cout << "- GST (Global Stabilization Time) assumption" << std::endl;
}

int main() {
    std::cout << "BYZANTINE FAULT TOLERANCE: COMPLETE IMPLEMENTATION AND ANALYSIS" << std::endl;

    // Run mathematical proofs
    prove_byzantine_agreement();

    // Analyze scenarios
    analyze_byzantine_scenarios();

    // Demonstrate PBFT phases
    demonstrate_pbft_phases();

    // Run simulations
    std::cout << "\n" << std::string(70, '=') << std::endl;
    std::cout << "PBFT SIMULATIONS" << std::endl;
    std::cout << std::string(70, '=') << std::endl;

    try {
        // Test 1: Normal operation
        std::cout << "\nTest 1: Normal operation (N=4, f=0)" << std::endl;
        PBFTNetwork network1(4, 0);
        network1.run_consensus(3);

        // Test 2: With Byzantine nodes
        std::cout << "\n\nTest 2: With Byzantine nodes (N=7, f=2)" << std::endl;
        PBFTNetwork network2(7, 2);
        network2.run_consensus(3);

        // Test 3: Impossible case (should throw)
        std::cout << "\n\nTest 3: Impossible case (N=3, f=1)" << std::endl;
        try {
            PBFTNetwork network3(3, 1);
            network3.run_consensus(1);
        } catch (const std::runtime_error& e) {
            std::cout << "Expected error: " << e.what() << std::endl;
        }

    } catch (const std::exception& e) {
        std::cerr << "Error: " << e.what() << std::endl;
    }

    return 0;
}

6. Threshold Cryptography: Shamir's Secret Sharing

Mathematical Foundation

Shamir's Secret Sharing is based on polynomial interpolation over finite fields:

Theorem: Given any t points in the plane with distinct x-coordinates, there exists exactly one polynomial of degree t-1 that passes through all points.

Algorithm:

Choose a prime p > secret
Create polynomial: f(x) = s + a₁x + a₂x² + ... + aₜ₋₁xᵗ⁻¹ mod p
Share point (i, f(i)) with party i
Any t points can reconstruct via Lagrange interpolation

Complete Python Implementation

import random
from typing import List, Tuple
import hashlib
from dataclasses import dataclass
from functools import lru_cache
import sympy as sp

class ShamirSecretSharing:
    """
    Complete implementation of Shamir's Secret Sharing
    """

    def __init__(self, prime: int = 2**127 - 1):
        """
        Initialize with a large prime
        """
        self.prime = prime

    def generate_shares(self, secret: int, n: int, t: int) -> List[Tuple[int, int]]:
        """
        Generate n shares, any t of which can reconstruct the secret

        Args:
            secret: The secret to share (integer)
            n: Total number of shares
            t: Threshold (minimum shares needed)

        Returns:
            List of (x, y) pairs
        """
        if t > n:
            raise ValueError("Threshold cannot exceed total shares")

        if secret >= self.prime:
            raise ValueError("Secret must be less than prime")

        # Generate random coefficients for polynomial of degree t-1
        coefficients = [secret] + [random.randint(1, self.prime - 1) 
                                  for _ in range(t - 1)]

        # Evaluate polynomial at points 1, 2, ..., n
        shares = []
        for x in range(1, n + 1):
            y = self._evaluate_polynomial(coefficients, x)
            shares.append((x, y))

        return shares

    def _evaluate_polynomial(self, coeffs: List[int], x: int) -> int:
        """Evaluate polynomial using Horner's method"""
        result = 0
        for coeff in reversed(coeffs):
            result = (result * x + coeff) % self.prime
        return result

    def reconstruct_secret(self, shares: List[Tuple[int, int]]) -> int:
        """
        Reconstruct secret using Lagrange interpolation

        Args:
            shares: List of at least t shares

        Returns:
            Reconstructed secret
        """
        if not shares:
            raise ValueError("No shares provided")

        # Lagrange interpolation to compute f(0)
        secret = 0

        for i, (xi, yi) in enumerate(shares):
            # Compute Lagrange basis polynomial L_i(0)
            numerator = 1
            denominator = 1

            for j, (xj, _) in enumerate(shares):
                if i == j:
                    continue

                numerator = (numerator * (-xj)) % self.prime
                denominator = (denominator * (xi - xj)) % self.prime

            # L_i(0) = numerator / denominator
            li = (numerator * self._mod_inverse(denominator)) % self.prime

            # Add contribution of this share
            secret = (secret + yi * li) % self.prime

        return secret

    def _mod_inverse(self, a: int) -> int:
        """Compute modular inverse using extended Euclidean algorithm"""
        return pow(a, -1, self.prime)  # Python 3.8+

    def verify_share(self, share: Tuple[int, int], commitment: List[int]) -> bool:
        """
        Verify share validity using Feldman's VSS
        """
        x, y = share

        # g^y should equal product of commitments^ (x^i)
        # Simplified verification
        lhs = pow(2, y, self.prime)  # Using 2 as generator

        rhs = 1
        for i, comm in enumerate(commitment):
            rhs = (rhs * pow(comm, x**i, self.prime)) % self.prime

        return lhs == rhs

class ThresholdSignature:
    """
    Threshold signatures using Shamir's Secret Sharing
    """

    def __init__(self, n: int, t: int):
        self.n = n
        self.t = t
        self.sss = ShamirSecretSharing()

        # Generate key shares
        self.private_key = random.randint(1, 2**256)
        self.shares = self.sss.generate_shares(self.private_key, n, t)

        # Public commitments for verification
        self.commitments = self._generate_commitments()

    def _generate_commitments(self) -> List[int]:
        """Generate Feldman commitments for verification"""
        # For simplicity, use g = 2
        g = 2
        prime = self.sss.prime

        commitments = []
        for i in range(self.t):
            # Commitment for coefficient i
            comm = pow(g, random.randint(1, prime - 1), prime)
            commitments.append(comm)

        return commitments

    def sign_share(self, message: str, share_idx: int) -> Tuple[int, int]:
        """
        Generate signature share for a message

        Returns: (share_index, signature_share)
        """
        x, y = self.shares[share_idx]

        # Hash message
        msg_hash = int(hashlib.sha256(message.encode()).hexdigest(), 16)
        msg_hash = msg_hash % self.sss.prime

        # Generate signature share
        # Simplified: s_i = y * H(m) mod p
        sig_share = (y * msg_hash) % self.sss.prime

        return x, sig_share

    def combine_signatures(self, sig_shares: List[Tuple[int, int]]) -> int:
        """
        Combine signature shares using Lagrange interpolation

        Returns: Complete signature
        """
        # Reconstruct signature using same method as secret sharing
        signature = 0

        for i, (xi, si) in enumerate(sig_shares):
            # Compute Lagrange basis polynomial L_i(0)
            numerator = 1
            denominator = 1

            for j, (xj, _) in enumerate(sig_shares):
                if i == j:
                    continue

                numerator = (numerator * (-xj)) % self.sss.prime
                denominator = (denominator * (xi - xj)) % self.sss.prime

            li = (numerator * self.sss._mod_inverse(denominator)) % self.sss.prime

            signature = (signature + si * li) % self.sss.prime

        return signature

    def verify_signature(self, message: str, signature: int) -> bool:
        """
        Verify threshold signature

        Returns: True if signature is valid
        """
        # Hash message
        msg_hash = int(hashlib.sha256(message.encode()).hexdigest(), 16)
        msg_hash = msg_hash % self.sss.prime

        # Compute expected: g^signature mod p
        g = 2
        lhs = pow(g, signature, self.sss.prime)

        # Compute: y^H(m) mod p where y = g^private_key
        y = pow(g, self.private_key, self.sss.prime)
        rhs = pow(y, msg_hash, self.sss.prime)

        return lhs == rhs

class DistributedKeyGeneration:
    """
    Distributed Key Generation without trusted dealer
    """

    def __init__(self, n: int, t: int):
        self.n = n
        self.t = t
        self.sss = ShamirSecretSharing()

        # Each party will generate their own polynomial
        self.parties = []

    class Party:
        def __init__(self, party_id: int, n: int, t: int, sss):
            self.id = party_id
            self.n = n
            self.t = t
            self.sss = sss

            # Generate random secret
            self.secret = random.randint(1, sss.prime - 1)

            # Create shares for other parties
            self.shares = sss.generate_shares(self.secret, n, t)

            # Generate commitments for verification
            self.commitments = self._generate_commitments()

        def _generate_commitments(self):
            """Generate commitments to polynomial coefficients"""
            g = 2
            prime = self.sss.prime

            # For simplicity, just commit to the secret
            return [pow(g, self.secret, prime)]

        def get_share_for(self, party_id: int) -> Tuple[int, int]:
            """Get share for specific party"""
            return self.shares[party_id - 1]

    def run_protocol(self):
        """Run DKG protocol"""
        print("Starting Distributed Key Generation...")

        # Create parties
        self.parties = [self.Party(i, self.n, self.t, self.sss) 
                       for i in range(1, self.n + 1)]

        # Phase 1: Each party sends shares to others
        print("\nPhase 1: Distributing shares...")
        received_shares = {i: [] for i in range(1, self.n + 1)}

        for party in self.parties:
            for other_id in range(1, self.n + 1):
                if other_id == party.id:
                    continue

                share = party.get_share_for(other_id)
                received_shares[other_id].append((party.id, share))

        # Phase 2: Verify shares
        print("\nPhase 2: Verifying shares...")
        valid_shares = {i: [] for i in range(1, self.n + 1)}

        for party_id in range(1, self.n + 1):
            for share_giver_id, share in received_shares[party_id]:
                # Verify share using commitments
                x, y = share
                giver = self.parties[share_giver_id - 1]

                # Simplified verification
                if y < self.sss.prime:  # Basic check
                    valid_shares[party_id].append(share)

        # Phase 3: Compute final key shares
        print("\nPhase 3: Computing final key shares...")
        final_shares = []

        for party_id in range(1, self.n + 1):
            # Each party sums received shares
            party_shares = valid_shares[party_id]
            if len(party_shares) < self.t:
                print(f"Party {party_id}: insufficient valid shares")
                continue

            # Sum y-values for same x
            x = party_id
            y_sum = 0

            for _, (_, y) in party_shares:
                y_sum = (y_sum + y) % self.sss.prime

            final_shares.append((x, y_sum))

        # The sum of secrets is the distributed key
        return final_shares

    def reconstruct_key(self, shares: List[Tuple[int, int]]) -> int:
        """Reconstruct distributed key from shares"""
        return self.sss.reconstruct_secret(shares)

def demonstrate_threshold_crypto():
    """
    Complete demonstration of threshold cryptography
    """
    print("="*70)
    print("THRESHOLD CRYPTOGRAPHY COMPLETE DEMONSTRATION")
    print("="*70)

    # Parameters
    n = 5  # Total shares
    t = 3  # Threshold

    # 1. Basic Secret Sharing
    print("\n1. Shamir's Secret Sharing:")
    print("-"*40)

    sss = ShamirSecretSharing()
    secret = 123456789

    print(f"Original secret: {secret}")

    # Generate shares
    shares = sss.generate_shares(secret, n, t)
    print(f"\nGenerated {n} shares (need {t} to reconstruct):")
    for i, (x, y) in enumerate(shares):
        print(f"  Share {i+1}: ({x}, {y})")

    # Reconstruct with threshold
    print(f"\nReconstructing with {t} shares...")
    selected_shares = shares[:t]
    reconstructed = sss.reconstruct_secret(selected_shares)
    print(f"Reconstructed secret: {reconstructed}")
    print(f"Match: {'✓' if reconstructed == secret else '✗'}")

    # Try with insufficient shares
    print(f"\nTrying with {t-1} shares (should fail)...")
    try:
        insufficient = shares[:t-1]
        sss.reconstruct_secret(insufficient)
        print("Unexpected success!")
    except Exception as e:
        print(f"Expected error: {e}")

    # 2. Threshold Signatures
    print("\n\n2. Threshold Signatures:")
    print("-"*40)

    ts = ThresholdSignature(n, t)
    message = "Hello, threshold crypto!"

    print(f"Message: {message}")

    # Generate signature shares
    print(f"\nGenerating signature shares from {t} parties...")
    sig_shares = []
    for i in range(t):
        x, sig_share = ts.sign_share(message, i)
        sig_shares.append((x, sig_share))
        print(f"  Party {i}: share = {sig_share}")

    # Combine signatures
    print("\nCombining signature shares...")
    full_signature = ts.combine_signatures(sig_shares)
    print(f"Full signature: {full_signature}")

    # Verify signature
    print("\nVerifying signature...")
    is_valid = ts.verify_signature(message, full_signature)
    print(f"Signature valid: {'✓' if is_valid else '✗'}")

    # 3. Distributed Key Generation
    print("\n\n3. Distributed Key Generation (no trusted dealer):")
    print("-"*40)

    dkg = DistributedKeyGeneration(n, t)
    final_shares = dkg.run_protocol()

    print(f"\nGenerated {len(final_shares)} final shares")

    # Reconstruct distributed key
    if len(final_shares) >= t:
        distributed_key = dkg.reconstruct_key(final_shares[:t])
        print(f"Distributed key: {distributed_key}")

    # 4. Security Analysis
    print("\n\n4. Security Analysis:")
    print("-"*40)

    print("\nInformation Theoretic Security:")
    print("  - With t-1 shares, zero information about secret")
    print("  - Security doesn't depend on computational assumptions")
    print("  - Perfect secrecy for finite fields")

    print("\nProperties:")
    print(f"  - Additive homomorphism: f(x) + g(x) = (f+g)(x)")
    print(f"  - Any subset of size t can reconstruct")
    print(f"  - Any subset of size < t learns nothing")

    print("\nApplications:")
    print("  - Distributed custody of crypto assets")
    print("  - Secure multi-party computation")
    print("  - Byzantine fault tolerance")
    print("  - Password recovery systems")

def mathematical_proofs():
    """
    Mathematical proofs for threshold cryptography
    """
    print("\n" + "="*70)
    print("MATHEMATICAL PROOFS FOR THRESHOLD CRYPTOGRAPHY")
    print("="*70)

    print("\nTheorem 1: Lagrange Interpolation Uniqueness")
    print("Proof:")
    print("  Given: Points (x₁, y₁), ..., (xₜ, yₜ) with distinct xᵢ")
    print("  Want: Polynomial f of degree ≤ t-1 with f(xᵢ) = yᵢ")
    print("  Construct Lagrange basis polynomials:")
    print("    Lᵢ(x) = ∏_{j≠i} (x - xⱼ) / (xᵢ - xⱼ)")
    print("  Then f(x) = Σ yᵢ Lᵢ(x)")
    print("  f has degree ≤ t-1 and f(xᵢ) = yᵢ")
    print("  Uniqueness: If g also satisfies, then f-g has t roots")
    print("  but degree ≤ t-1, so f-g ≡ 0 ⇒ f = g ✓")

    print("\nTheorem 2: Information Theoretic Security")
    print("Proof:")
    print("  Given any t-1 shares and any secret s, ∃ polynomial f")
    print("  with f(0) = s passing through given points")
    print("  Specifically: choose random aₜ₋₁, solve for others")
    print("  Thus probability distribution of secret is uniform")
    print("  given t-1 shares ⇒ perfect secrecy ✓")

    print("\nTheorem 3: Linear Homomorphism")
    print("Proof:")
    print("  Let f, g be sharing polynomials for secrets s, s'")
    print("  Then h(x) = f(x) + g(x) is polynomial of same degree")
    print("  with h(0) = s + s'")
    print("  Shares are (xᵢ, f(xᵢ) + g(xᵢ))")
    print("  Thus secret sharing preserves addition ✓")

def analyze_threshold_security():
    """
    Analyze security of threshold schemes
    """
    print("\n" + "="*70)
    print("SECURITY ANALYSIS OF THRESHOLD SCHEMES")
    print("="*70)

    # Different configurations
    configurations = [
        (3, 2),  # 2-of-3
        (5, 3),  # 3-of-5
        (7, 4),  # 4-of-7
        (10, 6), # 6-of-10
    ]

    print("\nConfiguration Analysis:")
    print("-"*80)
    print(f"{'Scheme':<10} {'Shares Needed':<15} {'Total Shares':<15} "
          f"{'Security Level':<15} {'Redundancy':<15}")
    print("-"*80)

    for total, threshold in configurations:
        # Security increases with threshold
        security = 2**(threshold * 32)  # Approximate

        # Redundancy = total / threshold
        redundancy = total / threshold

        print(f"{threshold}-of-{total}:{threshold:<15} {total:<15} "
              f"{security:<15.2e} {redundancy:<15.2f}")

    # Attack scenarios
    print("\nAttack Scenarios:")
    print("-"*80)

    scenarios = [
        ("Brute force", "Try all possible secrets", "Exponential in field size"),
        ("Share theft", "Steal t-1 shares", "Still secure"),
        ("Malicious dealer", "Distribute invalid shares", "Use VSS"),
        ("Network attacks", "Intercept shares", "Use encryption"),
    ]

    for name, description, defense in scenarios:
        print(f"{name:<20} {description:<30} {defense:<30}")

if __name__ == "__main__":
    demonstrate_threshold_crypto()
    mathematical_proofs()
    analyze_threshold_security()

7. Formal Verification of Protocols: TLA+ and Model Checking

Mathematical Foundations of Formal Verification

Formal verification uses mathematical logic to prove correctness properties of systems:

Temporal Logic (LTL/CTL):

Linear Temporal Logic (LTL): Specifies properties over single execution paths

  ◇P = Eventually P (P will eventually be true)
  □P = Always P (P is always true)
  P U Q = P until Q (P holds until Q becomes true)

Computation Tree Logic (CTL): Specifies properties over computation trees

  ∃◇P = There exists a path where P eventually holds
  ∀□P = For all paths, P always holds

The TLA+ Approach:
TLA+ (Temporal Logic of Actions) combines:

Actions: State transitions described as mathematical formulas
Temporal Operators: To specify liveness and safety properties
Refinement: To prove implementation matches specification

Complete TLA+ Specification for Paxos Consensus

---------------------------- MODULE Paxos ----------------------------
EXTENDS Integers, Sequences, FiniteSets, TLC

CONSTANT Acceptor, Proposer, Value
ASSUME Acceptor \cap Proposer = {} \* Disjoint sets
ASSUME Cardinality(Acceptor) > 0 \* At least one acceptor
ASSUME Cardinality(Proposer) > 0 \* At least one proposer

VARIABLES
    ballot,           \* Current ballot number (proposer -> nat)
    accepted,         \* Accepted proposals (acceptor -> [ballot, value])
    proposed,         \* Proposed values (proposer -> value)
    decided           \* Decided values (value or nil)

TypeInvariant ==
    /\ ballot \in [Proposer -> Nat]
    /\ accepted \in [Acceptor -> SUBSET (Nat \times Value)]
    /\ proposed \in [Proposer -> Value \union {Nil}]
    /\ decided \in Value \union {Nil}

(*---------------------------------------------------------------
    Phase 1a: Proposer sends prepare request
----------------------------------------------------------------*)
SendPrepare(proposer) ==
    /\ proposed[proposer] = Nil
    /\ ballot' = [ballot EXCEPT ![proposer] = @ + 1]
    /\ UNCHANGED <<accepted, proposed, decided>>

(*---------------------------------------------------------------
    Phase 1b: Acceptor responds to prepare
----------------------------------------------------------------*)
ReceivePrepare(proposer, acceptor) ==
    /\ \E b \in Nat : \* There exists some ballot
        /\ b > ballot[proposer]
        /\ \A a \in Acceptor : \* No higher ballot accepted
            \A <b2, v2> \in accepted[a] : b2 < b
    /\ accepted' = [accepted EXCEPT ![acceptor] = @ \union {<ballot[proposer], Nil>}]
    /\ UNCHANGED <<ballot, proposed, decided>>

(*---------------------------------------------------------------
    Phase 2a: Proposer sends accept request
----------------------------------------------------------------*)
SendAccept(proposer, value) ==
    /\ proposed[proposer] = Nil
    /\ \E acceptor \in Acceptor : \* Majority promises
        LET promises == {a \in Acceptor : <ballot[proposer], Nil> \in accepted[a]}
        IN Cardinality(promises) > Cardinality(Acceptor) \div 2
    /\ proposed' = [proposed EXCEPT ![proposer] = value]
    /\ UNCHANGED <<ballot, accepted, decided>>

(*---------------------------------------------------------------
    Phase 2b: Acceptor accepts proposal
----------------------------------------------------------------*)
ReceiveAccept(proposer, acceptor, value) ==
    /\ proposed[proposer] = value
    /\ \E <b, _> \in accepted[acceptor] : b = ballot[proposer]
    /\ accepted' = [accepted EXCEPT ![acceptor] = 
                    @ \union {<ballot[proposer], value>}]
    /\ UNCHANGED <<ballot, proposed, decided>>

(*---------------------------------------------------------------
    Learn decision
----------------------------------------------------------------*)
LearnDecision(value) ==
    /\ \E acceptor \in Acceptor : \* Majority accepted
        LET accepts == {a \in Acceptor : \E b : <b, value> \in accepted[a]}
        IN Cardinality(accepts) > Cardinality(Acceptor) \div 2
    /\ decided' = value
    /\ UNCHANGED <<ballot, accepted, proposed>>

(*---------------------------------------------------------------
    Next-state relation
----------------------------------------------------------------*)
Next ==
    \/ \E p \in Proposer : SendPrepare(p)
    \/ \E p \in Proposer, a \in Acceptor : ReceivePrepare(p, a)
    \/ \E p \in Proposer, v \in Value : SendAccept(p, v)
    \/ \E p \in Proposer, a \in Acceptor, v \in Value : ReceiveAccept(p, a, v)
    \/ \E v \in Value : LearnDecision(v)

(*---------------------------------------------------------------
    Initial state
----------------------------------------------------------------*)
Init ==
    /\ ballot = [p \in Proposer |-> 0]
    /\ accepted = [a \in Acceptor |-> {}]
    /\ proposed = [p \in Proposer |-> Nil]
    /\ decided = Nil

(*---------------------------------------------------------------
    Safety properties
----------------------------------------------------------------*)
Safety ==
    /\ \A v1, v2 \in Value : \* Agreement: only one value decided
        (decided = v1 /\ decided = v2) => (v1 = v2)
    /\ \A a \in Acceptor : \* Consistency of accepted values
        \A <b1, v1>, <b2, v2> \in accepted[a] :
            (b1 = b2) => (v1 = v2)

(*---------------------------------------------------------------
    Liveness properties
----------------------------------------------------------------*)
Liveness ==
    /\ \E v \in Value : \* Validity: only proposed values can be decided
        (decided = v) => (\E p \in Proposer : proposed[p] = v)
    /\ <>(\E v \in Value : decided = v)  \* Termination: eventually decide

(*---------------------------------------------------------------
    Complete specification
----------------------------------------------------------------*)
Spec == Init /\ [][Next]_<<ballot, accepted, proposed, decided>> /\ Liveness

THEOREM Spec => []Safety  \* Safety is invariant

=============================================================================

Complete Python Implementation: Model Checking and Verification

import itertools
from typing import Set, Dict, List, Tuple, Optional, Any
from dataclasses import dataclass, field
from enum import Enum
from collections import defaultdict
import time
import random
from functools import lru_cache

class LTLFormula:
    """
    Linear Temporal Logic formula implementation
    """

    def __init__(self, formula_type: str, *args):
        self.type = formula_type
        self.args = args

    def evaluate(self, trace: List[Dict[str, Any]], position: int = 0) -> bool:
        """Evaluate LTL formula on trace starting at position"""
        if self.type == "atomic":
            prop_name = self.args[0]
            return prop_name in trace[position]

        elif self.type == "not":
            return not self.args[0].evaluate(trace, position)

        elif self.type == "and":
            return all(arg.evaluate(trace, position) for arg in self.args)

        elif self.type == "or":
            return any(arg.evaluate(trace, position) for arg in self.args)

        elif self.type == "implies":
            left, right = self.args
            return (not left.evaluate(trace, position)) or right.evaluate(trace, position)

        elif self.type == "always":
            subformula = self.args[0]
            return all(subformula.evaluate(trace, i) for i in range(position, len(trace)))

        elif self.type == "eventually":
            subformula = self.args[0]
            return any(subformula.evaluate(trace, i) for i in range(position, len(trace)))

        elif self.type == "until":
            left, right = self.args
            for i in range(position, len(trace)):
                if right.evaluate(trace, i):
                    return True
                if not left.evaluate(trace, i):
                    return False
            return False

        elif self.type == "next":
            subformula = self.args[0]
            if position + 1 < len(trace):
                return subformula.evaluate(trace, position + 1)
            return False

        else:
            raise ValueError(f"Unknown formula type: {self.type}")

    def to_str(self) -> str:
        if self.type == "atomic":
            return self.args[0]
        elif self.type == "not":
            return f"¬({self.args[0].to_str()})"
        elif self.type == "and":
            return f"({' ∧ '.join(arg.to_str() for arg in self.args)})"
        elif self.type == "or":
            return f"({' ∨ '.join(arg.to_str() for arg in self.args)})"
        elif self.type == "implies":
            return f"({self.args[0].to_str()} → {self.args[1].to_str()})"
        elif self.type == "always":
            return f"□({self.args[0].to_str()})"
        elif self.type == "eventually":
            return f"◇({self.args[0].to_str()})"
        elif self.type == "until":
            return f"({self.args[0].to_str()} U {self.args[1].to_str()})"
        elif self.type == "next":
            return f"○({self.args[0].to_str()})"

class StateMachine:
    """
    Finite state machine for protocol verification
    """

    def __init__(self, states: Set[str], 
                 initial_state: str,
                 transitions: Dict[str, List[str]]):
        self.states = states
        self.initial_state = initial_state
        self.transitions = transitions

        # Atomic propositions for each state
        self.state_props = {state: set() for state in states}

        # Cache for reachable states
        self._reachable_cache = {}

    def add_proposition(self, state: str, prop: str):
        """Add atomic proposition to state"""
        self.state_props[state].add(prop)

    def get_successors(self, state: str) -> List[str]:
        """Get all possible next states"""
        return self.transitions.get(state, [])

    def get_trace_props(self, trace: List[str]) -> List[Set[str]]:
        """Get propositions for each state in trace"""
        return [self.state_props[state] for state in trace]

    def check_ltl(self, formula: LTLFormula, max_depth: int = 10) -> Tuple[bool, Optional[List[str]]]:
        """
        Model check LTL formula using DFS
        Returns: (satisfied, counterexample_trace)
        """
        visited = set()

        def dfs(state: str, depth: int, trace: List[str]) -> Tuple[bool, Optional[List[str]]]:
            if depth > max_depth:
                return True, None  # Assume satisfied for bounded model checking

            trace_key = (state, depth)
            if trace_key in visited:
                return True, None

            visited.add(trace_key)
            current_trace = trace + [state]

            # Get propositions for current trace
            trace_props = self.get_trace_props(current_trace)

            # Check if current trace violates formula
            if not formula.evaluate(trace_props, 0):
                return False, current_trace

            # Check all successor states
            for next_state in self.get_successors(state):
                satisfied, counterexample = dfs(next_state, depth + 1, current_trace)
                if not satisfied:
                    return False, counterexample

            return True, None

        return dfs(self.initial_state, 0, [])

    def check_invariant(self, invariant: LTLFormula) -> Tuple[bool, Optional[List[str]]]:
        """
        Check state invariant (□P)
        """
        return self.check_ltl(LTLFormula("always", invariant))

    def check_liveness(self, liveness: LTLFormula) -> Tuple[bool, Optional[List[str]]]:
        """
        Check liveness property (◇P)
        """
        return self.check_ltl(LTLFormula("eventually", liveness))

    def get_all_traces(self, max_length: int) -> List[List[str]]:
        """Generate all possible traces up to max length"""
        traces = []

        def dfs(state: str, depth: int, trace: List[str]):
            if depth >= max_length:
                traces.append(trace.copy())
                return

            current_trace = trace + [state]
            traces.append(current_trace.copy())

            for next_state in self.get_successors(state):
                dfs(next_state, depth + 1, current_trace)

        dfs(self.initial_state, 0, [])
        return traces

class PaxosModelChecker:
    """
    Complete model checker for Paxos consensus protocol
    """

    def __init__(self, num_proposers: int = 2, num_acceptors: int = 3):
        self.num_proposers = num_proposers
        self.num_acceptors = num_acceptors

        # State representation
        self.states = set()
        self.transitions = defaultdict(list)

        # Build state machine
        self._build_state_space()

    def _state_to_str(self, state: Dict) -> str:
        """Convert state dictionary to string representation"""
        parts = []

        # Ballot numbers
        for i in range(self.num_proposers):
            parts.append(f"B{i}={state.get(f'ballot_{i}', 0)}")

        # Accepted values
        for i in range(self.num_acceptors):
            accepted = state.get(f'accepted_{i}', (0, None))
            parts.append(f"A{i}=({accepted[0]},{accepted[1]})")

        # Proposed values
        for i in range(self.num_proposers):
            proposed = state.get(f'proposed_{i}', None)
            parts.append(f"P{i}={proposed}")

        # Decision
        decision = state.get('decision', None)
        parts.append(f"D={decision}")

        return "|".join(parts)

    def _build_state_space(self):
        """Build complete state space for Paxos"""
        # Generate all possible states
        # This is simplified - real Paxos has infinite state space

        # Initialize with empty state
        initial_state = {}
        for i in range(self.num_proposers):
            initial_state[f'ballot_{i}'] = 0
            initial_state[f'proposed_{i}'] = None

        for i in range(self.num_acceptors):
            initial_state[f'accepted_{i}'] = (0, None)

        initial_state['decision'] = None

        initial_str = self._state_to_str(initial_state)
        self.states.add(initial_str)

        # BFS to explore state space (limited)
        queue = [initial_state]
        visited = {initial_str}

        max_states = 1000  # Limit for tractability
        state_count = 0

        while queue and state_count < max_states:
            state = queue.pop(0)
            state_str = self._state_to_str(state)
            state_count += 1

            # Generate all possible next states
            next_states = self._get_next_states(state)

            for next_state in next_states:
                next_str = self._state_to_str(next_state)

                if next_str not in visited:
                    visited.add(next_str)
                    self.states.add(next_str)
                    queue.append(next_state)

                # Add transition
                if next_str not in self.transitions[state_str]:
                    self.transitions[state_str].append(next_str)

    def _get_next_states(self, state: Dict) -> List[Dict]:
        """Generate all possible next states from current state"""
        next_states = []

        # Phase 1a: Proposer sends prepare
        for i in range(self.num_proposers):
            if state.get(f'proposed_{i}') is None:
                new_state = state.copy()
                new_state[f'ballot_{i}'] = state.get(f'ballot_{i}', 0) + 1
                next_states.append(new_state)

        # Phase 1b: Acceptor responds to prepare
        for i in range(self.num_acceptors):
            current_ballot, current_value = state.get(f'accepted_{i}', (0, None))

            # Find highest ballot from proposers
            max_ballot = max(state.get(f'ballot_{j}', 0) 
                           for j in range(self.num_proposers))

            if max_ballot > current_ballot:
                new_state = state.copy()
                new_state[f'accepted_{i}'] = (max_ballot, None)
                next_states.append(new_state)

        # Phase 2a: Proposer sends accept
        for i in range(self.num_proposers):
            if state.get(f'proposed_{i}') is None:
                # Check if proposer has majority promises
                ballot = state.get(f'ballot_{i}', 0)
                promises = 0

                for j in range(self.num_acceptors):
                    accepted_ballot, _ = state.get(f'accepted_{j}', (0, None))
                    if accepted_ballot == ballot:
                        promises += 1

                if promises > self.num_acceptors // 2:
                    # Proposer can propose a value
                    for value in ['A', 'B']:  # Possible values
                        new_state = state.copy()
                        new_state[f'proposed_{i}'] = value
                        next_states.append(new_state)

        # Phase 2b: Acceptor accepts
        for i in range(self.num_acceptors):
            current_ballot, current_value = state.get(f'accepted_{i}', (0, None))

            # Find proposers with matching ballot
            for j in range(self.num_proposers):
                ballot = state.get(f'ballot_{j}', 0)
                value = state.get(f'proposed_{j}')

                if value is not None and ballot == current_ballot:
                    new_state = state.copy()
                    new_state[f'accepted_{i}'] = (ballot, value)
                    next_states.append(new_state)

        # Learn decision
        # Check if any value has majority acceptance
        for value in ['A', 'B']:
            accepts = 0
            for i in range(self.num_acceptors):
                _, accepted_value = state.get(f'accepted_{i}', (0, None))
                if accepted_value == value:
                    accepts += 1

            if accepts > self.num_acceptors // 2 and state.get('decision') is None:
                new_state = state.copy()
                new_state['decision'] = value
                next_states.append(new_state)

        return next_states

    def verify_safety(self) -> Dict[str, Any]:
        """
        Verify Paxos safety properties
        """
        results = {}

        # Build state machine
        sm = StateMachine(
            states=self.states,
            initial_state=self._state_to_str({
                f'ballot_{i}': 0 for i in range(self.num_proposers)
            }),
            transitions=dict(self.transitions)
        )

        # Add propositions
        for state_str in self.states:
            # Parse state string
            parts = state_str.split('|')
            decision = None
            for part in parts:
                if part.startswith('D='):
                    decision = part[2:] if part[2:] != 'None' else None

            if decision:
                sm.add_proposition(state_str, f"decided_{decision}")

            # Check for conflicting decisions (safety violation)
            # In real implementation, we'd check more properties

        # Check Agreement: Only one value can be decided
        agreement_formula = LTLFormula("implies",
            LTLFormula("and",
                LTLFormula("atomic", "decided_A"),
                LTLFormula("atomic", "decided_B")),
            LTLFormula("atomic", "false"))  # A and B cannot both be true

        satisfied, counterexample = sm.check_invariant(agreement_formula)
        results['agreement'] = {
            'satisfied': satisfied,
            'counterexample': counterexample
        }

        # Check Validity: Only proposed values can be decided
        # Simplified check
        validity_formula = LTLFormula("implies",
            LTLFormula("or",
                LTLFormula("atomic", "decided_A"),
                LTLFormula("atomic", "decided_B")),
            LTLFormula("atomic", "value_proposed"))

        satisfied, counterexample = sm.check_invariant(validity_formula)
        results['validity'] = {
            'satisfied': satisfied,
            'counterexample': counterexample
        }

        # Check Termination: Eventually a decision is reached
        termination_formula = LTLFormula("eventually",
            LTLFormula("or",
                LTLFormula("atomic", "decided_A"),
                LTLFormula("atomic", "decided_B")))

        satisfied, counterexample = sm.check_liveness(termination_formula)
        results['termination'] = {
            'satisfied': satisfied,
            'counterexample': counterexample
        }

        return results

class SPINModelChecker:
    """
    Simplified SPIN-like model checker with Promela-like syntax
    """

    def __init__(self):
        self.processes = []
        self.channels = {}
        self.variables = {}
        self.ltl_properties = []

    def add_process(self, name: str, code: str):
        """Add a process with Promela-like code"""
        self.processes.append({
            'name': name,
            'code': code,
            'pc': 0,  # Program counter
            'local_vars': {}
        })

    def add_channel(self, name: str, capacity: int):
        """Add a message channel"""
        self.channels[name] = {
            'queue': [],
            'capacity': capacity
        }

    def add_variable(self, name: str, initial_value):
        """Add global variable"""
        self.variables[name] = initial_value

    def add_ltl_property(self, formula: str):
        """Add LTL property to verify"""
        self.ltl_properties.append(formula)

    def parse_promela(self, code: str):
        """Parse simplified Promela code"""
        lines = [line.strip() for line in code.split('\n') if line.strip()]

        for line in lines:
            if line.startswith('chan '):
                # Channel declaration
                parts = line.split()
                name = parts[1]
                capacity = int(parts[3]) if len(parts) > 3 else 0
                self.add_channel(name, capacity)

            elif line.startswith('int '):
                # Variable declaration
                parts = line.split()
                name = parts[1]
                value = int(parts[3]) if len(parts) > 3 else 0
                self.add_variable(name, value)

            elif line.startswith('proctype '):
                # Process type
                name = line.split()[1].rstrip('()')
                # Simplified - would need to parse body

    def model_check(self, max_steps: int = 1000) -> Dict[str, Any]:
        """
        Perform model checking with state space exploration
        """
        # Initial state
        initial_state = {
            'processes': [p.copy() for p in self.processes],
            'channels': {k: v.copy() for k, v in self.channels.items()},
            'variables': self.variables.copy(),
            'step': 0
        }

        visited = set()
        queue = [(initial_state, [])]  # (state, trace)
        violations = []

        while queue:
            state, trace = queue.pop(0)

            # Check for LTL violations in current trace
            for formula in self.ltl_properties:
                if not self._check_ltl_trace(formula, trace + [state]):
                    violations.append({
                        'formula': formula,
                        'trace': trace + [state]
                    })

            # Check if we should stop
            if len(trace) >= max_steps:
                continue

            # Generate next states
            next_states = self._get_next_states(state)

            for next_state in next_states:
                state_hash = self._hash_state(next_state)
                if state_hash not in visited:
                    visited.add(state_hash)
                    queue.append((next_state, trace + [state]))

        return {
            'states_explored': len(visited),
            'violations': violations,
            'safe': len(violations) == 0
        }

    def _hash_state(self, state: Dict) -> str:
        """Create hash for state to detect duplicates"""
        # Simplified - would need proper hashing
        return str(state)

    def _get_next_states(self, state: Dict) -> List[Dict]:
        """Generate all possible next states"""
        # This is a simplified implementation
        # Real implementation would parse Promela code and execute steps
        next_states = []

        # Simulate some transitions
        # Send/receive on channels, variable updates, etc.

        return next_states

    def _check_ltl_trace(self, formula: str, trace: List[Dict]) -> bool:
        """Check LTL formula on trace"""
        # Simplified implementation
        # Real implementation would parse LTL and evaluate
        return True

def demonstrate_formal_verification():
    """
    Complete demonstration of formal verification techniques
    """
    print("="*70)
    print("FORMAL VERIFICATION OF DISTRIBUTED PROTOCOLS")
    print("="*70)

    # 1. LTL Formula Examples
    print("\n1. Linear Temporal Logic (LTL) Examples:")
    print("-"*40)

    # Define atomic propositions
    p = LTLFormula("atomic", "connected")
    q = LTLFormula("atomic", "data_sent")
    r = LTLFormula("atomic", "ack_received")

    # Safety: Always, if connected then eventually data sent
    safety = LTLFormula("always", 
        LTLFormula("implies", p, LTLFormula("eventually", q)))

    # Liveness: Eventually ack received
    liveness = LTLFormula("eventually", r)

    # Response: Always, if data sent then eventually ack received
    response = LTLFormula("always",
        LTLFormula("implies", q, LTLFormula("eventually", r)))

    print(f"Safety formula: {safety.to_str()}")
    print(f"Liveness formula: {liveness.to_str()}")
    print(f"Response formula: {response.to_str()}")

    # Test evaluation
    trace = [
        {"connected"},
        {"connected", "data_sent"},
        {"connected", "ack_received"},
        {"ack_received"}
    ]

    print(f"\nTrace: {trace}")
    print(f"Safety holds: {safety.evaluate(trace)}")
    print(f"Liveness holds: {liveness.evaluate(trace)}")
    print(f"Response holds: {response.evaluate(trace)}")

    # 2. State Machine Model Checking
    print("\n\n2. State Machine Model Checking:")
    print("-"*40)

    # Create a simple state machine
    states = {"S0", "S1", "S2", "S3"}
    transitions = {
        "S0": ["S1", "S2"],
        "S1": ["S3"],
        "S2": ["S3"],
        "S3": []
    }

    sm = StateMachine(states, "S0", transitions)

    # Add propositions
    sm.add_proposition("S0", "initial")
    sm.add_proposition("S1", "processing")
    sm.add_proposition("S2", "processing")
    sm.add_proposition("S3", "finished")

    # Check invariant: always (processing -> not finished)
    invariant = LTLFormula("implies",
        LTLFormula("atomic", "processing"),
        LTLFormula("not", LTLFormula("atomic", "finished")))

    satisfied, counterexample = sm.check_invariant(invariant)
    print(f"Invariant '{invariant.to_str()}' satisfied: {satisfied}")
    if counterexample:
        print(f"Counterexample: {counterexample}")

    # Check liveness: eventually finished
    liveness_prop = LTLFormula("eventually", LTLFormula("atomic", "finished"))
    satisfied, counterexample = sm.check_liveness(liveness_prop)
    print(f"\nLiveness '{liveness_prop.to_str()}' satisfied: {satisfied}")

    # 3. Paxos Model Checking
    print("\n\n3. Paxos Consensus Protocol Verification:")
    print("-"*40)

    checker = PaxosModelChecker(num_proposers=2, num_acceptors=3)
    results = checker.verify_safety()

    print("Paxos Safety Properties:")
    for prop, result in results.items():
        print(f"  {prop}: {'✓' if result['satisfied'] else '✗'}")
        if not result['satisfied'] and result['counterexample']:
            print(f"    Counterexample length: {len(result['counterexample'])}")

    # 4. Promela-like Model Checking
    print("\n\n4. Promela/SPIN-style Model Checking:")
    print("-"*40)

    spin = SPINModelChecker()

    # Define a simple mutual exclusion protocol
    promela_code = """
    bool turn = 0;
    bool flag[2] = 0;

    proctype P(int id) {
        flag[id] = 1;
        turn = 1 - id;
        (flag[1-id] == 0 || turn == id);
        /* critical section */
        flag[id] = 0;
    }
    """

    spin.parse_promela(promela_code)

    # Add LTL properties
    spin.add_ltl_property("[] (P0_in_critical -> !P1_in_critical)")  # Mutual exclusion
    spin.add_ltl_property("[] (P0_wants -> <> P0_in_critical)")  # Liveness

    print("Mutual exclusion protocol defined")
    print("LTL properties added for verification")

    # 5. Theorem Proving Concepts
    print("\n\n5. Theorem Proving Concepts:")
    print("-"*40)

    print("Hoare Logic for Protocol Verification:")
    print("  {P} S {Q} - If precondition P holds before S, then Q holds after")
    print("\nExample: Two-Phase Commit")
    print("  Precondition: All participants in init state")
    print("  Protocol: Coordinator sends prepare -> participants vote -> decide")
    print("  Postcondition: All participants have same decision (commit/abort)")

    print("\nInvariant Proof Technique:")
    print("  1. Base case: Invariant holds in initial state")
    print("  2. Inductive step: If invariant holds and action occurs,")
    print("     invariant still holds in next state")
    print("  3. Conclusion: Invariant holds for all reachable states")

def verify_distributed_mutex():
    """
    Complete verification of distributed mutual exclusion protocol
    """
    print("\n" + "="*70)
    print("COMPLETE VERIFICATION: DISTRIBUTED MUTUAL EXCLUSION")
    print("="*70)

    # Define the protocol states
    class MutexState(Enum):
        RELEASED = "released"
        WANTED = "wanted"
        HELD = "held"

    # Create state machine for Ricart-Agrawala algorithm
    states = set()
    transitions = defaultdict(list)

    # Generate all possible states (simplified)
    # 2 processes, each can be in 3 states, plus timestamp state
    for p1_state in MutexState:
        for p2_state in MutexState:
            for ts1 in [0, 1, 2]:
                for ts2 in [0, 1, 2]:
                    state_str = f"P1:{p1_state.value}:{ts1}|P2:{p2_state.value}:{ts2}"
                    states.add(state_str)

    # Define transitions based on Ricart-Agrawala algorithm
    # Simplified version
    for state_str in states:
        parts = state_str.split('|')
        p1_part, p2_part = parts

        p1_state, p1_ts = p1_part.split(':')[1], int(p1_part.split(':')[2])
        p2_state, p2_ts = p2_part.split(':')[1], int(p2_part.split(':')[2])

        # Process 1 can request critical section
        if p1_state == MutexState.RELEASED.value:
            new_state = f"P1:{MutexState.WANTED.value}:{max(p1_ts, p2_ts) + 1}|{p2_part}"
            transitions[state_str].append(new_state)

        # Process 1 can enter critical section if it has smallest timestamp
        if (p1_state == MutexState.WANTED.value and 
            (p2_state != MutexState.WANTED.value or p1_ts < p2_ts or 
             (p1_ts == p2_ts and 1 < 2))):  # Process IDs break ties
            new_state = f"P1:{MutexState.HELD.value}:{p1_ts}|{p2_part}"
            transitions[state_str].append(new_state)

        # Process 1 can release
        if p1_state == MutexState.HELD.value:
            new_state = f"P1:{MutexState.RELEASED.value}:{p1_ts}|{p2_part}"
            transitions[state_str].append(new_state)

        # Similar transitions for process 2
        if p2_state == MutexState.RELEASED.value:
            new_state = f"{p1_part}|P2:{MutexState.WANTED.value}:{max(p1_ts, p2_ts) + 1}"
            transitions[state_str].append(new_state)

        if (p2_state == MutexState.WANTED.value and 
            (p1_state != MutexState.WANTED.value or p2_ts < p1_ts or 
             (p2_ts == p1_ts and 2 < 1))):
            new_state = f"{p1_part}|P2:{MutexState.HELD.value}:{p2_ts}"
            transitions[state_str].append(new_state)

        if p2_state == MutexState.HELD.value:
            new_state = f"{p1_part}|P2:{MutexState.RELEASED.value}:{p2_ts}"
            transitions[state_str].append(new_state)

    # Create state machine
    initial_state = f"P1:{MutexState.RELEASED.value}:0|P2:{MutexState.RELEASED.value}:0"
    sm = StateMachine(states, initial_state, dict(transitions))

    # Add propositions
    for state_str in states:
        if "HELD" in state_str:
            if "P1:HELD" in state_str:
                sm.add_proposition(state_str, "P1_in_critical")
            if "P2:HELD" in state_str:
                sm.add_proposition(state_str, "P2_in_critical")

        if "WANTED" in state_str:
            if "P1:WANTED" in state_str:
                sm.add_proposition(state_str, "P1_wants")
            if "P2:WANTED" in state_str:
                sm.add_proposition(state_str, "P2_wants")

    # Verify properties
    print("\nVerifying Mutual Exclusion Properties:")

    # 1. Mutual exclusion: Never both processes in critical section
    mutex_formula = LTLFormula("always",
        LTLFormula("not",
            LTLFormula("and",
                LTLFormula("atomic", "P1_in_critical"),
                LTLFormula("atomic", "P2_in_critical"))))

    satisfied, counterexample = sm.check_invariant(mutex_formula)
    print(f"  1. Mutual Exclusion: {'✓' if satisfied else '✗'}")
    if not satisfied and counterexample:
        print(f"     Violation found in trace of length {len(counterexample)}")

    # 2. Deadlock freedom: If process wants CS, eventually gets it
    deadlock_free = LTLFormula("always",
        LTLFormula("implies",
            LTLFormula("atomic", "P1_wants"),
            LTLFormula("eventually", LTLFormula("atomic", "P1_in_critical"))))

    satisfied, counterexample = sm.check_invariant(deadlock_free)
    print(f"  2. Deadlock Freedom: {'✓' if satisfied else '✗'}")

    # 3. Starvation freedom: Eventually every request is granted
    # Simplified check
    no_starvation = LTLFormula("eventually",
        LTLFormula("implies",
            LTLFormula("atomic", "P1_wants"),
            LTLFormula("atomic", "P1_in_critical")))

    satisfied, counterexample = sm.check_liveness(no_starvation)
    print(f"  3. No Starvation: {'✓' if satisfied else '✗'}")

    # 4. Safety: Process only enters CS when it has permission
    # Based on timestamp comparison
    safety = LTLFormula("always",
        LTLFormula("implies",
            LTLFormula("atomic", "P1_in_critical"),
            LTLFormula("or",
                LTLFormula("not", LTLFormula("atomic", "P2_wants")),
                LTLFormula("atomic", "P1_has_smaller_timestamp"))))  # Simplified

    satisfied, counterexample = sm.check_invariant(safety)
    print(f"  4. Safety (timestamp order): {'✓' if satisfied else '✗'}")

    # Generate some example traces
    print("\nExample Protocol Execution Traces:")
    traces = sm.get_all_traces(max_length=5)

    for i, trace in enumerate(traces[:3]):  # Show first 3
        print(f"\n  Trace {i+1}:")
        for j, state in enumerate(trace):
            print(f"    Step {j}: {state}")

def mathematical_proofs_formal_verification():
    """
    Mathematical proofs for formal verification
    """
    print("\n" + "="*70)
    print("MATHEMATICAL FOUNDATIONS OF FORMAL VERIFICATION")
    print("="*70)

    print("\nTheorem 1: Soundness of Hoare Logic")
    print("Proof:")
    print("  Let {P} S {Q} be a Hoare triple.")
    print("  By induction on the structure of S:")
    print("    Base case: Skip: {P} skip {P} (trivially true)")
    print("    Assignment: {P[e/x]} x := e {P}")
    print("    Sequence: {P} S1; S2 {R} if {P} S1 {Q} and {Q} S2 {R}")
    print("    If: {P ∧ B} S1 {Q} and {P ∧ ¬B} S2 {Q} ⇒ {P} if B then S1 else S2 {Q}")
    print("    While: {P ∧ B} S {P} ⇒ {P} while B do S {P ∧ ¬B}")
    print("  Each rule preserves truth. ✓")

    print("\nTheorem 2: Completeness of Hoare Logic (Cook's Theorem)")
    print("Proof (sketch):")
    print("  1. For any program S and assertions P, Q,")
    print("     if {P} S {Q} is true in all interpretations,")
    print("     then it is provable in Hoare logic.")
    print("  2. Construct weakest precondition wp(S, Q)")
    print("  3. Show P → wp(S, Q) is valid")
    print("  4. By Gödel's completeness theorem, valid formulas are provable.")
    print("  5. Therefore, {P} S {Q} is provable. ✓")

    print("\nTheorem 3: Floyd-Hoare Method for Loop Invariants")
    print("Proof:")
    print("  For loop: while B do S")
    print("  Find invariant I such that:")
    print("    1. P → I  (initialization)")
    print("    2. {I ∧ B} S {I}  (preservation)")
    print("    3. I ∧ ¬B → Q  (postcondition)")
    print("  Then by while rule, {P} while B do S {Q}")
    print("  This is complete for loops with well-founded ordering. ✓")

    print("\nTheorem 4: Temporal Logic Model Checking (CTL*)")
    print("Proof (complexity):")
    print("  CTL model checking: O(|S| × |φ|) where S is state space")
    print("  LTL model checking: PSPACE-complete in formula size")
    print("  CTL* model checking: EXPTIME-complete")
    print("  Proof via reduction from QBF (Quantified Boolean Formulas). ✓")

if __name__ == "__main__":
    demonstrate_formal_verification()
    verify_distributed_mutex()
    mathematical_proofs_formal_verification()

8. Zero-Knowledge Proofs in Distributed Systems

Mathematical Foundations of ZK Proofs

Zero-Knowledge Proofs allow proving a statement without revealing why it's true:

Three Properties:

Completeness: If statement is true, honest verifier will be convinced
Soundness: If statement is false, no cheating prover can convince verifier
Zero-knowledge: Verifier learns nothing beyond statement's truth

Mathematical Framework:

Σ-protocols: 3-move protocols (commit, challenge, response)
Fiat-Shamir Heuristic: Convert interactive to non-interactive using hash functions
zk-SNARKs: Succinct Non-interactive ARguments of Knowledge
- Uses quadratic arithmetic programs and pairing-based cryptography

Complete Python Implementation: zk-SNARKs and Bulletproofs

import hashlib
import random
from typing import List, Tuple, Dict, Any, Optional
from dataclasses import dataclass
from enum import Enum
import math
import json
from functools import lru_cache
import time

# Elliptic curve parameters (simplified for educational purposes)
class EllipticCurve:
    """
    Simplified elliptic curve implementation for ZK proofs
    Using secp256k1 parameters (like Bitcoin)
    """

    def __init__(self):
        # secp256k1 parameters
        self.p = 2**256 - 2**32 - 977  # Field prime
        self.a = 0
        self.b = 7
        self.Gx = 0x79BE667EF9DCBBAC55A06295CE870B07029BFCDB2DCE28D959F2815B16F81798
        self.Gy = 0x483ADA7726A3C4655DA4FBFC0E1108A8FD17B448A68554199C47D08FFB10D4B8
        self.n = 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFEBAAEDCE6AF48A03BBFD25E8CD0364141  # Order

        self.G = (self.Gx, self.Gy)

    def point_add(self, P: Tuple[int, int], Q: Tuple[int, int]) -> Tuple[int, int]:
        """Elliptic curve point addition"""
        if P == (0, 0):
            return Q
        if Q == (0, 0):
            return P

        x1, y1 = P
        x2, y2 = Q

        if x1 == x2 and y1 != y2:
            return (0, 0)  # Point at infinity

        if P == Q:
            # Point doubling
            lam = (3 * x1 * x1 + self.a) * pow(2 * y1, -1, self.p) % self.p
        else:
            lam = (y2 - y1) * pow(x2 - x1, -1, self.p) % self.p

        x3 = (lam * lam - x1 - x2) % self.p
        y3 = (lam * (x1 - x3) - y1) % self.p

        return (x3, y3)

    def scalar_mult(self, k: int, P: Tuple[int, int]) -> Tuple[int, int]:
        """Scalar multiplication k * P"""
        result = (0, 0)  # Point at infinity
        addend = P

        while k > 0:
            if k & 1:
                result = self.point_add(result, addend)
            addend = self.point_add(addend, addend)
            k >>= 1

        return result

    def hash_to_point(self, data: bytes) -> Tuple[int, int]:
        """Hash data to point on curve"""
        # Simplified: use hash mod p as x, compute y
        h = int.from_bytes(hashlib.sha256(data).digest(), 'big') % self.p

        # Find y such that y² = x³ + ax + b
        x = h
        rhs = (x * x * x + self.a * x + self.b) % self.p

        # Try to find square root (simplified)
        y = pow(rhs, (self.p + 1) // 4, self.p)  # Works for p ≡ 3 mod 4

        return (x, y)

class SigmaProtocol:
    """
    Σ-protocol for discrete logarithm knowledge
    Proves knowledge of x such that y = g^x
    """

    def __init__(self, curve: EllipticCurve):
        self.curve = curve

    def prove(self, x: int, y: Tuple[int, int]) -> Tuple[Tuple[int, int], int, int]:
        """
        Generate proof of knowledge of discrete logarithm

        Returns: (commitment, challenge, response)
        """
        # Step 1: Prover chooses random r
        r = random.randint(1, self.curve.n - 1)

        # Step 2: Compute commitment t = g^r
        t = self.curve.scalar_mult(r, self.curve.G)

        # Step 3: Verifier sends random challenge c
        # In interactive protocol, verifier sends c
        # For non-interactive, we generate using Fiat-Shamir
        c = self._generate_challenge(y, t)

        # Step 4: Prover computes response s = r + c * x mod n
        s = (r + c * x) % self.curve.n

        return (t, c, s)

    def verify(self, y: Tuple[int, int], proof: Tuple[Tuple[int, int], int, int]) -> bool:
        """
        Verify proof of knowledge
        """
        t, c, s = proof

        # Check: g^s == t * y^c
        left = self.curve.scalar_mult(s, self.curve.G)
        right = self.curve.point_add(t, self.curve.scalar_mult(c, y))

        return left == right

    def _generate_challenge(self, y: Tuple[int, int], t: Tuple[int, int]) -> int:
        """Fiat-Shamir heuristic to make non-interactive"""
        data = json.dumps({
            'y': y,
            't': t
        }).encode()

        # Hash to get challenge
        h = hashlib.sha256(data).digest()
        return int.from_bytes(h, 'big') % self.curve.n

class Bulletproofs:
    """
    Bulletproofs implementation for range proofs
    Proves a committed value is within [0, 2^n - 1]
    """

    def __init__(self, curve: EllipticCurve, n_bits: int = 32):
        self.curve = curve
        self.n_bits = n_bits

        # Generators for Pedersen commitments
        self.G = curve.G
        self.H = curve.hash_to_point(b"bulletproofs_H_generator")

    def commit(self, value: int, blinding: int) -> Tuple[int, int]:
        """
        Pedersen commitment: C = v*G + r*H
        """
        vG = self.curve.scalar_mult(value, self.G)
        rH = self.curve.scalar_mult(blinding, self.H)
        return self.curve.point_add(vG, rH)

    def prove_range(self, value: int, blinding: int, 
                   commitment: Tuple[int, int]) -> Dict[str, Any]:
        """
        Generate bulletproof range proof
        """
        # Decompose value into bits
        bits = [(value >> i) & 1 for i in range(self.n_bits)]

        # Commit to each bit
        aL = bits  # Left vector
        aR = [b ^ 1 for b in bits]  # Right vector: aR = aL - 1

        # Generate random blinding factors
        alpha = random.randint(1, self.curve.n - 1)
        sL = [random.randint(1, self.curve.n - 1) for _ in range(self.n_bits)]
        sR = [random.randint(1, self.curve.n - 1) for _ in range(self.n_bits)]
        rho = random.randint(1, self.curve.n - 1)

        # Compute vector commitments
        A = self._vector_commit(aL, aR, alpha)
        S = self._vector_commit(sL, sR, rho)

        # Fiat-Shamir challenge
        y = self._hash_challenge(commitment, A, S)
        z = self._hash_challenge(commitment, A, S, y)

        # Compute polynomials
        l0, l1, r0, r1 = self._compute_polynomials(aL, aR, sL, sR, y, z)

        # Inner product argument
        t1, t2, tau1, tau2 = self._inner_product_argument(l0, l1, r0, r1)

        # Final challenge
        x = self._hash_challenge(commitment, A, S, y, z, t1, t2)

        # Compute responses
        taux = (tau1 * x + tau2 * x * x) % self.curve.n
        mu = (alpha + rho * x) % self.curve.n

        # Compute inner product
        inner_product = sum(l * r for l, r in zip(l0, r0))

        return {
            'A': A,
            'S': S,
            't1': t1,
            't2': t2,
            'taux': taux,
            'mu': mu,
            'inner_product': inner_product,
            'commitment': commitment,
            'L': [],  # Would contain intermediate L values
            'R': [],  # Would contain intermediate R values
            'a': 0,   # Final a
            'b': 0    # Final b
        }

    def verify_range(self, proof: Dict[str, Any]) -> bool:
        """
        Verify bulletproof range proof
        """
        # This is a simplified verification
        # Real bulletproofs have more complex verification

        commitment = proof['commitment']

        # Recompute challenges
        y = self._hash_challenge(commitment, proof['A'], proof['S'])
        z = self._hash_challenge(commitment, proof['A'], proof['S'], y)
        x = self._hash_challenge(commitment, proof['A'], proof['S'], y, z, 
                                proof['t1'], proof['t2'])

        # Check commitment reconstruction
        # In real implementation, would verify inner product argument

        # Simplified check
        return proof['inner_product'] < 2**self.n_bits

    def _vector_commit(self, a: List[int], b: List[int], alpha: int) -> Tuple[int, int]:
        """Commit to two vectors"""
        # Simplified: C = Σ(a_i * G_i) + Σ(b_i * H_i) + alpha * H
        commitment = (0, 0)

        for i in range(self.n_bits):
            # Use different generators for each position
            Gi = self.curve.hash_to_point(f"G_{i}".encode())
            Hi = self.curve.hash_to_point(f"H_{i}".encode())

            aGi = self.curve.scalar_mult(a[i], Gi)
            bHi = self.curve.scalar_mult(b[i], Hi)

            commitment = self.curve.point_add(commitment, aGi)
            commitment = self.curve.point_add(commitment, bHi)

        # Add blinding
        alphaH = self.curve.scalar_mult(alpha, self.H)
        commitment = self.curve.point_add(commitment, alphaH)

        return commitment

    def _compute_polynomials(self, aL, aR, sL, sR, y, z):
        """Compute polynomial coefficients"""
        # Compute l(x) = aL - z*1^n + sL*x
        # Compute r(x) = y^n ∘ (aR + z*1^n + sR*x) + z^2 * 2^n

        y_vector = [pow(y, i, self.curve.n) for i in range(self.n_bits)]
        two_vector = [pow(2, i, self.curve.n) for i in range(self.n_bits)]

        # l0 = aL - z*1^n
        l0 = [(aL[i] - z) % self.curve.n for i in range(self.n_bits)]
        l1 = sL  # Coefficient for x

        # r0 = y^n ∘ (aR + z*1^n) + z^2 * 2^n
        r0 = []
        for i in range(self.n_bits):
            term1 = (aR[i] + z) % self.curve.n
            term2 = (z * z * two_vector[i]) % self.curve.n
            r0.append((y_vector[i] * term1 + term2) % self.curve.n)

        # r1 = y^n ∘ sR
        r1 = [(y_vector[i] * sR[i]) % self.curve.n for i in range(self.n_bits)]

        return l0, l1, r0, r1

    def _inner_product_argument(self, l0, l1, r0, r1):
        """Generate inner product argument"""
        # Simplified - real implementation uses recursive protocol

        # Compute t1 = <l1, r0> + <l0, r1>
        t1 = sum(l1[i] * r0[i] + l0[i] * r1[i] for i in range(self.n_bits)) % self.curve.n

        # Compute t2 = <l1, r1>
        t2 = sum(l1[i] * r1[i] for i in range(self.n_bits)) % self.curve.n

        # Random blinding factors
        tau1 = random.randint(1, self.curve.n - 1)
        tau2 = random.randint(1, self.curve.n - 1)

        return t1, t2, tau1, tau2

    def _hash_challenge(self, *args) -> int:
        """Generate Fiat-Shamir challenge"""
        data = json.dumps(args, default=str).encode()
        h = hashlib.sha256(data).digest()
        return int.from_bytes(h, 'big') % self.curve.n

class zkSNARK:
    """
    Simplified zk-SNARK implementation
    Proves correct execution of arithmetic circuit
    """

    def __init__(self, curve: EllipticCurve):
        self.curve = curve

        # Setup parameters
        self.tau = random.randint(1, self.curve.n - 1)  # Toxic waste (should be destroyed)

        # CRS (Common Reference String)
        self.g1 = curve.G
        self.g2 = curve.hash_to_point(b"zksnark_g2")

        # Powers of tau in G1 and G2
        self.powers_g1 = [curve.scalar_mult(pow(self.tau, i, curve.n), self.g1) 
                         for i in range(10)]  # Up to degree 9
        self.powers_g2 = [curve.scalar_mult(pow(self.tau, i, curve.n), self.g2) 
                         for i in range(10)]

    def setup(self, circuit):
        """
        Generate proving and verification keys for circuit
        """
        # QAP (Quadratic Arithmetic Program) setup
        # Simplified - real implementation uses polynomial interpolation

        # For circuit: out = in1 * in2 (multiplication gate)
        # Represent as: out - in1 * in2 = 0

        proving_key = {
            'g1': self.g1,
            'g2': self.g2,
            'alpha_g1': self.curve.scalar_mult(self.tau, self.g1),  # αG1
            'beta_g1': self.curve.scalar_mult(self.tau * 2, self.g1),  # βG1
            'beta_g2': self.curve.scalar_mult(self.tau * 2, self.g2),  # βG2
            'powers_g1': self.powers_g1,
            'powers_g2': self.powers_g2,
            'gamma_inv': pow(self.tau * 3, -1, self.curve.n),  # γ^{-1}
            'delta_inv': pow(self.tau * 4, -1, self.curve.n),  # δ^{-1}
        }

        verification_key = {
            'g1': self.g1,
            'g2': self.g2,
            'alpha_g1_beta_g2': self.curve.point_add(
                self.curve.scalar_mult(self.tau, self.g1),
                self.curve.scalar_mult(self.tau * 2, self.g2)
            ),
            'gamma_g2': self.curve.scalar_mult(self.tau * 3, self.g2),
            'delta_g2': self.curve.scalar_mult(self.tau * 4, self.g2),
            'ic': [  # Input consistency
                self.curve.scalar_mult(pow(self.tau, i, self.curve.n), self.g1)
                for i in range(3)  # For 2 inputs + output
            ]
        }

        return proving_key, verification_key

    def prove(self, proving_key, inputs, output):
        """
        Generate proof for circuit execution
        """
        # Example circuit: out = in1 * in2

        # Compute witness polynomial
        # In real zk-SNARK, this involves:
        # 1. Computing witness (assignment to wires)
        # 2. Computing polynomial coefficients
        # 3. Evaluating at tau

        # Simplified proof generation
        proof = {
            'a_g1': self.curve.scalar_mult(random.randint(1, self.curve.n - 1), proving_key['g1']),
            'b_g2': self.curve.scalar_mult(random.randint(1, self.curve.n - 1), proving_key['g2']),
            'c_g1': self.curve.scalar_mult(random.randint(1, self.curve.n - 1), proving_key['g1']),
            'inputs': inputs,
            'output': output
        }

        return proof

    def verify(self, verification_key, proof, public_inputs):
        """
        Verify zk-SNARK proof
        """
        # Simplified verification
        # Real verification uses pairing checks:
        # e(A, B) = e(α, β) * e(C, δ) ...

        # Check input consistency
        ic_sum = (0, 0)
        for i, inp in enumerate(public_inputs):
            ic_term = self.curve.scalar_mult(inp, verification_key['ic'][i])
            ic_sum = self.curve.point_add(ic_sum, ic_term)

        # Check circuit correctness (simplified)
        # For multiplication gate: out = in1 * in2
        if len(public_inputs) >= 2:
            in1, in2 = public_inputs[:2]
            out = proof['output']

            if out != in1 * in2:
                return False

        return True

class DistributedZK:
    """
    Distributed zero-knowledge proofs for distributed systems
    """

    def __init__(self):
        self.curve = EllipticCurve()
        self.sigma = SigmaProtocol(self.curve)
        self.bulletproofs = Bulletproofs(self.curve)
        self.zksnark = zkSNARK(self.curve)

    def prove_membership(self, secret: int, public_set: List[Tuple[int, int]]) -> Dict[str, Any]:
        """
        Prove membership in set without revealing which element
        """
        # Each element: y_i = g^x_i
        # Prover knows x such that y = g^x is in set

        # Commit to random values
        r = random.randint(1, self.curve.n - 1)
        C = self.curve.scalar_mult(r, self.curve.G)

        # For each element, create proof OR proof
        proofs = []
        for y in public_set:
            # Create simulated proof for elements we don't know
            # and real proof for the one we know

            # This is simplified - real implementation uses OR composition
            if self.curve.scalar_mult(secret, self.curve.G) == y:
                # Real proof
                proof = self.sigma.prove(secret, y)
            else:
                # Simulated proof
                proof = self._simulate_proof(y)

            proofs.append(proof)

        return {
            'commitment': C,
            'proofs': proofs,
            'set_hash': hashlib.sha256(
                str(sorted(public_set)).encode()
            ).hexdigest()
        }

    def _simulate_proof(self, y: Tuple[int, int]) -> Tuple[Tuple[int, int], int, int]:
        """Simulate proof for element we don't know"""
        # Choose random challenge and response
        c = random.randint(1, self.curve.n - 1)
        s = random.randint(1, self.curve.n - 1)

        # Compute commitment that matches: t = g^s * y^{-c}
        sG = self.curve.scalar_mult(s, self.curve.G)
        neg_cY = self.curve.scalar_mult(-c % self.curve.n, y)
        t = self.curve.point_add(sG, neg_cY)

        return (t, c, s)

    def verify_membership(self, proof: Dict[str, Any], public_set: List[Tuple[int, int]]) -> bool:
        """
        Verify membership proof
        """
        # Check all proofs
        for i, (y, proof_i) in enumerate(zip(public_set, proof['proofs'])):
            if not self.sigma.verify(y, proof_i):
                return False

        return True

    def prove_shuffle(self, input_list: List[Tuple[int, int]], 
                     output_list: List[Tuple[int, int]], 
                     permutation: List[int]) -> Dict[str, Any]:
        """
        Prove shuffle (mixnet) without revealing permutation
        """
        # Shuffle proof using Neff's technique

        # Commit to permutation matrix
        n = len(input_list)
        r = random.randint(1, self.curve.n - 1)

        # Create commitments to permutation
        commitments = []
        for i in range(n):
            # Commit to each row of permutation matrix
            C = self.curve.scalar_mult(r + i, self.curve.G)
            commitments.append(C)

        # Prove that output is permutation of input
        # Simplified - real implementation uses complex proofs

        return {
            'input_hash': hashlib.sha256(str(input_list).encode()).hexdigest(),
            'output_hash': hashlib.sha256(str(output_list).encode()).hexdigest(),
            'commitments': commitments,
            'proof': 'simplified_shuffle_proof'
        }

    def verify_shuffle(self, proof: Dict[str, Any], 
                      input_list: List[Tuple[int, int]], 
                      output_list: List[Tuple[int, int]]) -> bool:
        """
        Verify shuffle proof
        """
        # Check hashes
        input_hash = hashlib.sha256(str(input_list).encode()).hexdigest()
        output_hash = hashlib.sha256(str(output_list).encode()).hexdigest()

        if input_hash != proof['input_hash']:
            return False

        if output_hash != proof['output_hash']:
            return False

        # Simplified verification
        # Real verification would check polynomial equations

        return True

def demonstrate_zk_proofs():
    """
    Complete demonstration of zero-knowledge proofs
    """
    print("="*70)
    print("ZERO-KNOWLEDGE PROOFS IN DISTRIBUTED SYSTEMS")
    print("="*70)

    dist_zk = DistributedZK()

    # 1. Sigma Protocol (Discrete Log Proof)
    print("\n1. Σ-Protocol for Discrete Logarithm:")
    print("-"*40)

    # Prover knows x such that y = g^x
    x = random.randint(1, dist_zk.curve.n - 1)
    y = dist_zk.curve.scalar_mult(x, dist_zk.curve.G)

    print(f"Secret x: {x}")
    print(f"Public y: {y}")

    # Generate proof
    proof = dist_zk.sigma.prove(x, y)
    print(f"\nProof generated: (t, c, s)")
    print(f"  t (commitment): {proof[0]}")
    print(f"  c (challenge): {proof[1]}")
    print(f"  s (response): {proof[2]}")

    # Verify proof
    valid = dist_zk.sigma.verify(y, proof)
    print(f"\nProof valid: {'✓' if valid else '✗'}")

    # 2. Bulletproofs (Range Proofs)
    print("\n\n2. Bulletproofs for Range Proofs:")
    print("-"*40)

    value = 42
    blinding = random.randint(1, dist_zk.curve.n - 1)
    commitment = dist_zk.bulletproofs.commit(value, blinding)

    print(f"Value: {value} (0 ≤ value < {2**32})")
    print(f"Commitment: {commitment}")

    # Generate range proof
    range_proof = dist_zk.bulletproofs.prove_range(value, blinding, commitment)
    print(f"\nRange proof generated")
    print(f"  Proof size: ~{len(str(range_proof))} characters")

    # Verify range proof
    valid = dist_zk.bulletproofs.verify_range(range_proof)
    print(f"Range proof valid: {'✓' if valid else '✗'}")

    # 3. Set Membership Proof
    print("\n\n3. Set Membership Proof:")
    print("-"*40)

    # Create public set
    public_set = []
    secrets = []

    for i in range(5):
        secret = random.randint(1, dist_zk.curve.n - 1)
        point = dist_zk.curve.scalar_mult(secret, dist_zk.curve.G)
        public_set.append(point)
        secrets.append(secret)

    # Prover knows one secret
    known_secret = secrets[2]  # Know 3rd element
    print(f"Public set has {len(public_set)} elements")
    print(f"Prover knows secret for element 3")

    # Generate membership proof
    membership_proof = dist_zk.prove_membership(known_secret, public_set)
    print(f"\nMembership proof generated")
    print(f"  Set hash: {membership_proof['set_hash'][:16]}...")

    # Verify membership proof
    valid = dist_zk.verify_membership(membership_proof, public_set)
    print(f"Membership proof valid: {'✓' if valid else '✗'}")

    # 4. Shuffle Proof (Mixnet)
    print("\n\n4. Shuffle Proof (Mixnet):")
    print("-"*40)

    # Create input list
    input_list = []
    for i in range(3):
        r = random.randint(1, dist_zk.curve.n - 1)
        point = dist_zk.curve.scalar_mult(r, dist_zk.curve.G)
        input_list.append(point)

    # Create permutation
    permutation = [2, 0, 1]  # Shuffle
    output_list = [input_list[i] for i in permutation]

    print(f"Input list: {len(input_list)} elements")
    print(f"Output list: {len(output_list)} elements")
    print(f"Permutation: {permutation}")

    # Generate shuffle proof
    shuffle_proof = dist_zk.prove_shuffle(input_list, output_list, permutation)
    print(f"\nShuffle proof generated")

    # Verify shuffle proof
    valid = dist_zk.verify_shuffle(shuffle_proof, input_list, output_list)
    print(f"Shuffle proof valid: {'✓' if valid else '✗'}")

    # 5. zk-SNARK for Circuit
    print("\n\n5. zk-SNARK for Arithmetic Circuit:")
    print("-"*40)

    # Simple circuit: out = in1 * in2
    in1 = 7
    in2 = 6
    out = 42

    print(f"Circuit: out = in1 * in2")
    print(f"Inputs: {in1}, {in2}")
    print(f"Output: {out}")
    print(f"Correct: {out == in1 * in2}")

    # Setup
    proving_key, verification_key = dist_zk.zksnark.setup(None)
    print(f"\nzk-SNARK setup complete")
    print(f"  Proving key size: ~{len(str(proving_key))} characters")
    print(f"  Verification key size: ~{len(str(verification_key))} characters")

    # Generate proof
    zk_proof = dist_zk.zksnark.prove(proving_key, [in1, in2], out)
    print(f"\nzk-SNARK proof generated")
    print(f"  Proof size: ~{len(str(zk_proof))} characters")

    # Verify proof
    valid = dist_zk.zksnark.verify(verification_key, zk_proof, [in1, in2])
    print(f"zk-SNARK proof valid: {'✓' if valid else '✗'}")

def zk_proofs_mathematical_foundations():
    """
    Mathematical foundations of zero-knowledge proofs
    """
    print("\n" + "="*70)
    print("MATHEMATICAL FOUNDATIONS OF ZERO-KNOWLEDGE PROOFS")
    print("="*70)

    print("\nDefinition 1: Interactive Proof System")
    print("  An interactive proof system for language L is a pair (P, V) where:")
    print("    1. Completeness: ∀x ∈ L, Pr[V accepts] ≥ 2/3")
    print("    2. Soundness: ∀x ∉ L, ∀P*, Pr[V accepts] ≤ 1/3")

    print("\nDefinition 2: Zero-Knowledge")
    print("  (P, V) is zero-knowledge if ∃ simulator S such that ∀x ∈ L:")
    print("    View_V[P(x) ↔ V(x)] ≈ S(x)")
    print("  where ≈ denotes computational indistinguishability")

    print("\nTheorem 1: Σ-protocols are Honest-Verifier Zero-Knowledge")
    print("Proof:")
    print("  1. Simulator chooses random challenge c and response s")
    print("  2. Computes commitment t = g^s * y^{-c}")
    print("  3. Output (t, c, s)")
    print("  4. Distribution identical to real protocol. ✓")

    print("\nTheorem 2: Fiat-Shamir Heuristic Security")
    print("Proof (ROM - Random Oracle Model):")
    print("  If hash function H is modeled as random oracle,")
    print("  then non-interactive protocol is secure")
    print("  assuming underlying Σ-protocol is secure. ✓")

    print("\nTheorem 3: zk-SNARK Knowledge Soundness")
    print("Proof (Knowledge of Exponent Assumption):")
    print("  If prover can produce valid proof,")
    print("  then they know witness w such that C(x, w) = 1")
    print("  under q-PKE (Power Knowledge of Exponent) assumption. ✓")

    print("\nTheorem 4: Bulletproofs Communication Complexity")
    print("Proof:")
    print("  Range proof size: O(log n) where n is bits")
    print("  Compared to O(n) for previous schemes")
    print("  Achieved through inner product argument. ✓")

def zk_applications_distributed_systems():
    """
    Applications of ZK proofs in distributed systems
    """
    print("\n" + "="*70)
    print("APPLICATIONS IN DISTRIBUTED SYSTEMS")
    print("="*70)

    applications = [
        ("Blockchain Privacy", """
        - Zcash: zk-SNARKs for private transactions
        - Monero: Ring signatures + Bulletproofs
        - Tornado Cash: Mixers using zk proofs
        """),

        ("Authentication & Authorization", """
        - Passwordless authentication
        - Attribute-based credentials
        - Anonymous credentials
        """),

        ("Verifiable Computation", """
        - Outsourced computation verification
        - Cloud computing integrity
        - Distributed machine learning
        """),

        ("Distributed Ledgers", """
        - Private smart contracts
        - Confidential assets
        - Scalable payment channels
        """),

        ("Voting Systems", """
        - End-to-end verifiable voting
        - Privacy-preserving tallying
        - Anonymous credentials for voters
        """),
    ]

    for title, description in applications:
        print(f"\n{title}:")
        print(description)

    print("\nPerformance Characteristics:")
    print("-"*40)

    characteristics = [
        ("Σ-protocols", "3 rounds", "O(1) proof size", "Fast", "Interactive"),
        ("Bulletproofs", "Non-interactive", "O(log n)", "Medium", "No trusted setup"),
        ("zk-SNARKs", "Non-interactive", "O(1)", "Slow setup", "Trusted setup"),
        ("zk-STARKs", "Non-interactive", "O(log² n)", "Fast", "No trusted setup"),
    ]

    print(f"{'Type':<15} {'Rounds':<15} {'Proof Size':<15} {'Performance':<15} {'Setup':<15}")
    print("-"*70)
    for typ, rounds, size, perf, setup in characteristics:
        print(f"{typ:<15} {rounds:<15} {size:<15} {perf:<15} {setup:<15}")

if __name__ == "__main__":
    demonstrate_zk_proofs()
    zk_proofs_mathematical_foundations()
    zk_applications_distributed_systems()

9. Distributed Tracing Theory

Mathematical Foundations of Distributed Tracing

Distributed tracing involves understanding causal relationships across services:

Mathematical Models:

Causal Ordering: Partial order based on happens-before relation (→)
Vector Clocks: Logical clocks that track causality
Span DAG: Directed Acyclic Graph representation of traces

Formal Definition:
A trace T = (S, ≤) where:

S = set of spans
≤ = happens-before relation (reflexive, antisymmetric, transitive)

Complete Python Implementation: OpenTelemetry-like Tracing

import uuid
import time
import threading
from typing import Dict, List, Set, Optional, Tuple, Any
from dataclasses import dataclass, field
from enum import Enum
from collections import defaultdict, deque
import heapq
import random
import statistics
from datetime import datetime, timedelta

class TraceState:
    """
    W3C Trace Context state
    """

    def __init__(self, items: Dict[str, str] = None):
        self.items = items or {}

    def add(self, key: str, value: str):
        """Add key-value pair to trace state"""
        self.items[key] = value

    def get(self, key: str, default: str = None) -> Optional[str]:
        """Get value from trace state"""
        return self.items.get(key, default)

    def remove(self, key: str):
        """Remove key from trace state"""
        if key in self.items:
            del self.items[key]

    def to_header(self) -> str:
        """Convert to W3C trace state header format"""
        pairs = []
        for k, v in self.items.items():
            # Vendor key format: vendor@version=value
            pairs.append(f"{k}={v}")
        return ",".join(pairs)

    @classmethod
    def from_header(cls, header: str) -> 'TraceState':
        """Parse from W3C trace state header"""
        items = {}
        if header:
            for pair in header.split(','):
                if '=' in pair:
                    k, v = pair.split('=', 1)
                    items[k.strip()] = v.strip()
        return cls(items)

@dataclass
class SpanContext:
    """
    Context propagated across service boundaries
    """
    trace_id: str
    span_id: str
    trace_flags: int = 1  # Sampled flag
    trace_state: TraceState = field(default_factory=TraceState)
    is_remote: bool = False

    @property
    def is_valid(self) -> bool:
        """Check if context is valid"""
        return bool(self.trace_id and self.span_id)

    @property
    def is_sampled(self) -> bool:
        """Check if trace is sampled"""
        return bool(self.trace_flags & 1)

    def to_headers(self) -> Dict[str, str]:
        """Convert to W3C trace context headers"""
        headers = {
            'traceparent': f"00-{self.trace_id}-{self.span_id}-{self.trace_flags:02x}",
        }

        trace_state = self.trace_state.to_header()
        if trace_state:
            headers['tracestate'] = trace_state

        return headers

    @classmethod
    def from_headers(cls, headers: Dict[str, str]) -> 'SpanContext':
        """Create from W3C trace context headers"""
        traceparent = headers.get('traceparent', '')
        tracestate = headers.get('tracestate', '')

        if not traceparent:
            return cls.create_invalid()

        # Parse traceparent: version-trace-id-parent-id-flags
        parts = traceparent.split('-')
        if len(parts) != 4:
            return cls.create_invalid()

        version, trace_id, span_id, flags = parts

        try:
            trace_flags = int(flags, 16)
        except ValueError:
            return cls.create_invalid()

        trace_state = TraceState.from_header(tracestate)

        return cls(
            trace_id=trace_id,
            span_id=span_id,
            trace_flags=trace_flags,
            trace_state=trace_state,
            is_remote=True
        )

    @classmethod
    def create_invalid(cls) -> 'SpanContext':
        """Create invalid context"""
        return cls(trace_id="", span_id="", trace_flags=0)

    @classmethod
    def generate(cls, sampled: bool = True) -> 'SpanContext':
        """Generate new context"""
        trace_id = uuid.uuid4().hex[:32]
        span_id = uuid.uuid4().hex[:16]
        trace_flags = 1 if sampled else 0

        return cls(
            trace_id=trace_id,
            span_id=span_id,
            trace_flags=trace_flags
        )

class SpanKind(Enum):
    """Types of spans"""
    INTERNAL = "internal"
    SERVER = "server"
    CLIENT = "client"
    PRODUCER = "producer"
    CONSUMER = "consumer"

@dataclass
class Span:
    """
    Representation of a single operation
    """
    name: str
    context: SpanContext
    parent_context: Optional[SpanContext] = None
    kind: SpanKind = SpanKind.INTERNAL
    start_time: float = field(default_factory=time.time)
    end_time: Optional[float] = None
    attributes: Dict[str, Any] = field(default_factory=dict)
    events: List[Dict[str, Any]] = field(default_factory=list)
    status: Dict[str, Any] = field(default_factory=dict)
    links: List[SpanContext] = field(default_factory=list)

    def add_event(self, name: str, attributes: Dict[str, Any] = None):
        """Add event to span"""
        self.events.append({
            'name': name,
            'timestamp': time.time(),
            'attributes': attributes or {}
        })

    def set_attribute(self, key: str, value: Any):
        """Set span attribute"""
        self.attributes[key] = value

    def set_status(self, code: str, description: str = ""):
        """Set span status"""
        self.status = {'code': code, 'description': description}

    def end(self):
        """End the span"""
        if self.end_time is None:
            self.end_time = time.time()

    @property
    def duration(self) -> float:
        """Get span duration in seconds"""
        if self.end_time is None:
            return time.time() - self.start_time
        return self.end_time - self.start_time

    @property
    def is_ended(self) -> bool:
        """Check if span has ended"""
        return self.end_time is not None

    def to_dict(self) -> Dict[str, Any]:
        """Convert span to dictionary"""
        return {
            'name': self.name,
            'trace_id': self.context.trace_id,
            'span_id': self.context.span_id,
            'parent_span_id': self.parent_context.span_id if self.parent_context else None,
            'kind': self.kind.value,
            'start_time': self.start_time,
            'end_time': self.end_time,
            'duration': self.duration,
            'attributes': self.attributes,
            'events': self.events,
            'status': self.status,
            'links': [link.span_id for link in self.links]
        }

class Sampler:
    """
    Base class for trace sampling strategies
    """

    def should_sample(self, 
                     parent_context: Optional[SpanContext],
                     trace_id: str,
                     name: str,
                     kind: SpanKind,
                     attributes: Dict[str, Any]) -> Tuple[bool, List[Dict[str, Any]]]:
        """
        Decide whether to sample a span

        Returns: (sampled, attributes)
        """
        raise NotImplementedError

    def get_description(self) -> str:
        """Get sampler description"""
        raise NotImplementedError

class AlwaysOnSampler(Sampler):
    """Sample all traces"""

    def should_sample(self, parent_context, trace_id, name, kind, attributes):
        return True, []

    def get_description(self):
        return "AlwaysOnSampler"

class AlwaysOffSampler(Sampler):
    """Sample no traces"""

    def should_sample(self, parent_context, trace_id, name, kind, attributes):
        return False, []

    def get_description(self):
        return "AlwaysOffSampler"

class TraceIdRatioBasedSampler(Sampler):
    """Sample based on trace ID ratio"""

    def __init__(self, ratio: float):
        if not 0 <= ratio <= 1:
            raise ValueError("Ratio must be between 0 and 1")
        self.ratio = ratio

    def should_sample(self, parent_context, trace_id, name, kind, attributes):
        # Use trace ID to make deterministic sampling decision
        # Take first 8 bytes of trace ID as number
        trace_id_bytes = bytes.fromhex(trace_id[:16])
        trace_id_int = int.from_bytes(trace_id_bytes, 'big')

        # Use upper 56 bits for sampling (avoiding modulo bias)
        upper_bits = trace_id_int >> 8
        sampled = (upper_bits % 10000) < (self.ratio * 10000)

        return sampled, [{"sampler.ratio": self.ratio}]

    def get_description(self):
        return f"TraceIdRatioBased{{{self.ratio}}}"

class ParentBasedSampler(Sampler):
    """
    Sampler that respects parent's sampling decision
    """

    def __init__(self, 
                 root: Sampler,
                 remote_parent_sampled: Sampler,
                 remote_parent_not_sampled: Sampler,
                 local_parent_sampled: Sampler,
                 local_parent_not_sampled: Sampler):
        self.root = root
        self.remote_parent_sampled = remote_parent_sampled
        self.remote_parent_not_sampled = remote_parent_not_sampled
        self.local_parent_sampled = local_parent_sampled
        self.local_parent_not_sampled = local_parent_not_sampled

    def should_sample(self, parent_context, trace_id, name, kind, attributes):
        if not parent_context:
            # No parent - use root sampler
            return self.root.should_sample(parent_context, trace_id, name, kind, attributes)

        if parent_context.is_remote:
            if parent_context.is_sampled:
                return self.remote_parent_sampled.should_sample(
                    parent_context, trace_id, name, kind, attributes)
            else:
                return self.remote_parent_not_sampled.should_sample(
                    parent_context, trace_id, name, kind, attributes)
        else:
            if parent_context.is_sampled:
                return self.local_parent_sampled.should_sample(
                    parent_context, trace_id, name, kind, attributes)
            else:
                return self.local_parent_not_sampled.should_sample(
                    parent_context, trace_id, name, kind, attributes)

    def get_description(self):
        return f"ParentBased{{root={self.root.get_description()}}}"

class AdaptiveSampler(Sampler):
    """
    Adaptive sampler that adjusts rate based on load
    """

    def __init__(self, 
                 target_rate: float = 10.0,  # Spans per second
                 adjustment_window: int = 60):  # Seconds
        self.target_rate = target_rate
        self.adjustment_window = adjustment_window
        self.current_ratio = 1.0
        self.span_counts = deque(maxlen=adjustment_window)
        self.last_adjustment = time.time()

    def should_sample(self, parent_context, trace_id, name, kind, attributes):
        # Update counts
        current_time = time.time()
        self.span_counts.append(current_time)

        # Adjust ratio periodically
        if current_time - self.last_adjustment > 1.0:  # Every second
            self._adjust_ratio()
            self.last_adjustment = current_time

        # Make sampling decision
        trace_id_bytes = bytes.fromhex(trace_id[:16])
        trace_id_int = int.from_bytes(trace_id_bytes, 'big')
        upper_bits = trace_id_int >> 8
        sampled = (upper_bits % 10000) < (self.current_ratio * 10000)

        return sampled, [{"sampler.adaptive.ratio": self.current_ratio}]

    def _adjust_ratio(self):
        """Adjust sampling ratio based on current load"""
        current_time = time.time()

        # Count spans in last window
        window_start = current_time - self.adjustment_window
        span_count = sum(1 for t in self.span_counts if t > window_start)

        actual_rate = span_count / self.adjustment_window

        # Adjust ratio to reach target rate
        if actual_rate > self.target_rate * 1.1:
            # Too many spans, reduce sampling
            self.current_ratio *= 0.9
        elif actual_rate < self.target_rate * 0.9:
            # Too few spans, increase sampling
            self.current_ratio = min(1.0, self.current_ratio * 1.1)

        # Clamp between 0.01 and 1.0
        self.current_ratio = max(0.01, min(1.0, self.current_ratio))

    def get_description(self):
        return f"AdaptiveSampler{{target={self.target_rate}, ratio={self.current_ratio:.3f}}}"

class Tracer:
    """
    Main tracer implementation
    """

    def __init__(self, 
                 name: str,
                 sampler: Sampler = None,
                 max_spans_per_trace: int = 1000):
        self.name = name
        self.sampler = sampler or AlwaysOnSampler()
        self.max_spans_per_trace = max_spans_per_trace

        # Active spans
        self.active_spans = defaultdict(list)  # trace_id -> list of spans

        # Statistics
        self.stats = {
            'spans_created': 0,
            'spans_ended': 0,
            'traces_sampled': 0,
            'traces_dropped': 0
        }

        # Exporters
        self.exporters = []

    def start_span(self, 
                  name: str,
                  context: Optional[SpanContext] = None,
                  kind: SpanKind = SpanKind.INTERNAL,
                  attributes: Dict[str, Any] = None,
                  links: List[SpanContext] = None,
                  start_time: Optional[float] = None) -> Optional[Span]:
        """
        Start a new span
        """
        attributes = attributes or {}
        links = links or []

        # Determine parent context
        parent_context = context

        # Generate trace ID if not provided
        if parent_context and parent_context.is_valid:
            trace_id = parent_context.trace_id
        else:
            trace_id = uuid.uuid4().hex[:32]

        # Check if we should sample
        sampled, sampler_attrs = self.sampler.should_sample(
            parent_context, trace_id, name, kind, attributes)

        if not sampled:
            self.stats['traces_dropped'] += 1
            return None

        self.stats['traces_sampled'] += 1

        # Generate span context
        span_id = uuid.uuid4().hex[:16]
        trace_flags = 1 if sampled else 0

        span_context = SpanContext(
            trace_id=trace_id,
            span_id=span_id,
            trace_flags=trace_flags
        )

        # Create span
        span = Span(
            name=name,
            context=span_context,
            parent_context=parent_context,
            kind=kind,
            start_time=start_time or time.time(),
            attributes={**attributes, **{k: v for k, v in sampler_attrs}},
            links=links
        )

        # Add to active spans
        self.active_spans[trace_id].append(span)

        # Check trace size limit
        if len(self.active_spans[trace_id]) > self.max_spans_per_trace:
            self._cleanup_oldest_span(trace_id)

        self.stats['spans_created'] += 1

        return span

    def _cleanup_oldest_span(self, trace_id: str):
        """Remove oldest span from trace"""
        if trace_id in self.active_spans and self.active_spans[trace_id]:
            # Find and remove oldest span
            oldest_idx = 0
            oldest_time = self.active_spans[trace_id][0].start_time

            for i, span in enumerate(self.active_spans[trace_id][1:], 1):
                if span.start_time < oldest_time:
                    oldest_time = span.start_time
                    oldest_idx = i

            removed = self.active_spans[trace_id].pop(oldest_idx)
            if not self.active_spans[trace_id]:
                del self.active_spans[trace_id]

    def end_span(self, span: Span):
        """End a span"""
        if span.is_ended:
            return

        span.end()
        self.stats['spans_ended'] += 1

        # Check if trace is complete
        trace_id = span.context.trace_id
        if trace_id in self.active_spans:
            # Remove this span from active
            self.active_spans[trace_id] = [s for s in self.active_spans[trace_id] 
                                          if s.context.span_id != span.context.span_id]

            # If no more active spans, trace is complete
            if not self.active_spans[trace_id]:
                del self.active_spans[trace_id]
                self._export_trace([span])  # Simplified - would export all spans

    def get_active_traces(self) -> Dict[str, List[Span]]:
        """Get all active traces"""
        return dict(self.active_spans)

    def get_statistics(self) -> Dict[str, Any]:
        """Get tracer statistics"""
        return {
            **self.stats,
            'active_traces': len(self.active_spans),
            'active_spans': sum(len(spans) for spans in self.active_spans.values()),
            'sampler': self.sampler.get_description()
        }

    def add_exporter(self, exporter):
        """Add span exporter"""
        self.exporters.append(exporter)

    def _export_trace(self, spans: List[Span]):
        """Export completed trace"""
        for exporter in self.exporters:
            try:
                exporter.export(spans)
            except Exception as e:
                print(f"Exporter error: {e}")

class TraceAnalyzer:
    """
    Analyze traces for performance and issues
    """

    def __init__(self):
        self.traces = []  # List of completed traces
        self.metrics = defaultdict(list)

    def add_trace(self, spans: List[Span]):
        """Add completed trace for analysis"""
        self.traces.append(spans)

        # Update metrics
        self._update_metrics(spans)

    def _update_metrics(self, spans: List[Span]):
        """Update metrics from trace"""
        if not spans:
            return

        # Trace duration
        start_times = [s.start_time for s in spans]
        end_times = [s.end_time for s in spans if s.end_time]

        if start_times and end_times:
            trace_start = min(start_times)
            trace_end = max(end_times)
            trace_duration = trace_end - trace_start

            self.metrics['trace_duration'].append(trace_duration)

        # Span durations
        for span in spans:
            if span.end_time:
                self.metrics['span_duration'].append(span.duration)

        # Spans per trace
        self.metrics['spans_per_trace'].append(len(spans))

    def get_percentiles(self, metric: str, percentiles: List[float] = None) -> Dict[float, float]:
        """Calculate percentiles for metric"""
        if metric not in self.metrics or not self.metrics[metric]:
            return {}

        values = self.metrics[metric]
        percentiles = percentiles or [0.5, 0.75, 0.9, 0.95, 0.99]

        results = {}
        for p in percentiles:
            idx = int(p * len(values))
            sorted_values = sorted(values)
            results[p] = sorted_values[idx] if idx < len(sorted_values) else sorted_values[-1]

        return results

    def analyze_critical_path(self, spans: List[Span]) -> List[Span]:
        """
        Find critical path (longest path) in trace
        """
        if not spans:
            return []

        # Build graph
        span_dict = {s.context.span_id: s for s in spans}
        children = defaultdict(list)

        for span in spans:
            if span.parent_context and span.parent_context.span_id in span_dict:
                parent_id = span.parent_context.span_id
                children[parent_id].append(span.context.span_id)

        # Find root spans
        root_spans = [s for s in spans if not s.parent_context or 
                     s.parent_context.span_id not in span_dict]

        if not root_spans:
            return []

        # DFS to find longest path
        def dfs(span_id: str, path: List[str], path_duration: float) -> Tuple[List[str], float]:
            span = span_dict[span_id]
            current_duration = path_duration + span.duration

            if not children[span_id]:
                return path + [span_id], current_duration

            best_path = []
            best_duration = 0

            for child_id in children[span_id]:
                child_path, child_duration = dfs(child_id, path + [span_id], current_duration)
                if child_duration > best_duration:
                    best_path = child_path
                    best_duration = child_duration

            return best_path, best_duration

        # Find critical path from each root
        critical_path = []
        critical_duration = 0

        for root in root_spans:
            path, duration = dfs(root.context.span_id, [], 0)
            if duration > critical_duration:
                critical_path = path
                critical_duration = duration

        return [span_dict[span_id] for span_id in critical_path]

    def detect_anomalies(self, spans: List[Span], 
                        threshold_stddev: float = 2.0) -> List[Dict[str, Any]]:
        """
        Detect anomalous spans in trace
        """
        anomalies = []

        if not spans:
            return anomalies

        # Calculate statistics for similar spans
        span_groups = defaultdict(list)
        for span in spans:
            span_groups[span.name].append(span.duration)

        for name, durations in span_groups.items():
            if len(durations) < 3:
                continue

            mean = statistics.mean(durations)
            stdev = statistics.stdev(durations) if len(durations) > 1 else 0

            # Find spans more than threshold_stddev from mean
            for span in spans:
                if span.name == name and span.end_time:
                    z_score = abs(span.duration - mean) / stdev if stdev > 0 else 0
                    if z_score > threshold_stddev:
                        anomalies.append({
                            'span': span,
                            'name': name,
                            'duration': span.duration,
                            'mean_duration': mean,
                            'z_score': z_score,
                            'anomaly_type': 'duration'
                        })

        return anomalies

    def visualize_trace(self, spans: List[Span], max_width: int = 80):
        """
        Create ASCII visualization of trace
        """
        if not spans:
            return ""

        # Sort spans by start time
        sorted_spans = sorted(spans, key=lambda s: s.start_time)

        # Find time range
        start_times = [s.start_time for s in sorted_spans]
        end_times = [s.end_time for s in sorted_spans if s.end_time]

        if not end_times:
            return "Trace not complete"

        min_time = min(start_times)
        max_time = max(end_times)
        time_range = max_time - min_time

        if time_range == 0:
            time_range = 1

        # Build visualization
        lines = []
        lines.append(f"Trace Duration: {time_range:.3f}s")
        lines.append("=" * max_width)

        for span in sorted_spans:
            # Calculate position and width
            start_pos = int((span.start_time - min_time) / time_range * max_width)
            duration = span.duration if span.end_time else (time.time() - span.start_time)
            width = max(1, int(duration / time_range * max_width))

            # Truncate name if needed
            name = span.name
            if len(name) > width - 2:
                name = name[:width-5] + "..."

            # Create bar
            bar = " " * start_pos + "[" + name.center(width-2) + "]"
            lines.append(bar)

            # Add duration info
            lines.append(f"  {span.name}: {duration:.3f}s")

        return "\n".join(lines)

def demonstrate_distributed_tracing():
    """
    Complete demonstration of distributed tracing
    """
    print("="*70)
    print("DISTRIBUTED TRACING THEORY AND IMPLEMENTATION")
    print("="*70)

    # 1. Create Tracer with Adaptive Sampling
    print("\n1. Creating Tracer with Adaptive Sampling:")
    print("-"*40)

    adaptive_sampler = AdaptiveSampler(target_rate=5.0)
    tracer = Tracer(name="example-service", sampler=adaptive_sampler)

    print(f"Tracer created: {tracer.name}")
    print(f"Sampler: {tracer.sampler.get_description()}")

    # 2. Simulate Distributed Trace
    print("\n\n2. Simulating Distributed Trace Across Services:")
    print("-"*40)

    # Service A: Receives request
    print("\nService A (HTTP Server):")
    print("  Receives request from client")

    # Create root span
    root_span = tracer.start_span(
        name="HTTP GET /api/users",
        kind=SpanKind.SERVER,
        attributes={
            "http.method": "GET",
            "http.route": "/api/users",
            "http.url": "http://example.com/api/users",
            "net.peer.ip": "192.168.1.100"
        }
    )

    if root_span:
        print(f"  Created root span: {root_span.name}")
        print(f"  Trace ID: {root_span.context.trace_id[:16]}...")
        print(f"  Span ID: {root_span.context.span_id}")

        # Service A calls Service B (Database)
        print("\n  Service A calls Service B (Database):")

        # Extract context for propagation
        headers = root_span.context.to_headers()
        print(f"  Propagating context headers: {headers}")

        # Service B receives context
        print("\nService B (Database):")
        print("  Receives call from Service A")

        # Create child span
        child_span = tracer.start_span(
            name="Database Query",
            context=SpanContext.from_headers(headers),
            kind=SpanKind.CLIENT,
            attributes={
                "db.system": "postgresql",
                "db.operation": "SELECT",
                "db.query": "SELECT * FROM users"
            }
        )

        if child_span:
            print(f"  Created child span: {child_span.name}")
            print(f"  Parent Span ID: {child_span.parent_context.span_id}")

            # Simulate database work
            time.sleep(0.05)

            # End child span
            child_span.end()
            tracer.end_span(child_span)
            print(f"  Database query completed: {child_span.duration:.3f}s")

        # Service A continues processing
        print("\nService A (continues):")

        # Add event to root span
        root_span.add_event("database_query_completed")

        # Simulate more work
        time.sleep(0.02)

        # End root span
        root_span.end()
        tracer.end_span(root_span)
        print(f"  Request completed: {root_span.duration:.3f}s")

    # 3. Sampling Strategies Comparison
    print("\n\n3. Sampling Strategies Comparison:")
    print("-"*40)

    samplers = [
        ("Always On", AlwaysOnSampler()),
        ("Always Off", AlwaysOffSampler()),
        ("10% Ratio", TraceIdRatioBasedSampler(0.1)),
        ("Adaptive (5/s)", AdaptiveSampler(target_rate=5.0)),
    ]

    print(f"\n{'Sampler':<20} {'Description':<30} {'Sampled %':<10}")
    print("-"*70)

    # Test each sampler
    test_trace_id = uuid.uuid4().hex[:32]

    for name, sampler in samplers:
        sampled_count = 0
        total_tests = 1000

        for _ in range(total_tests):
            sampled, _ = sampler.should_sample(
                None, test_trace_id, "test-span", SpanKind.INTERNAL, {})
            if sampled:
                sampled_count += 1

        percentage = sampled_count / total_tests * 100
        print(f"{name:<20} {sampler.get_description():<30} {percentage:>8.1f}%")

    # 4. Trace Analysis
    print("\n\n4. Trace Analysis and Critical Path:")
    print("-"*40)

    analyzer = TraceAnalyzer()

    # Create complex trace for analysis
    spans = []

    # Root span
    root = Span(
        name="ProcessOrder",
        context=SpanContext.generate(),
        start_time=time.time()
    )
    spans.append(root)

    # Child spans
    inventory_check = Span(
        name="CheckInventory",
        context=SpanContext.generate(),
        parent_context=root.context,
        start_time=root.start_time + 0.01
    )
    inventory_check.end_time = inventory_check.start_time + 0.05
    spans.append(inventory_check)

    payment_processing = Span(
        name="ProcessPayment",
        context=SpanContext.generate(),
        parent_context=root.context,
        start_time=root.start_time + 0.02
    )
    payment_processing.end_time = payment_processing.start_time + 0.1  # Slow!
    spans.append(payment_processing)

    shipping_calculation = Span(
        name="CalculateShipping",
        context=SpanContext.generate(),
        parent_context=inventory_check.context,
        start_time=inventory_check.end_time + 0.01
    )
    shipping_calculation.end_time = shipping_calculation.start_time + 0.03
    spans.append(shipping_calculation)

    root.end_time = max(s.end_time for s in spans if s.end_time)

    # Analyze
    analyzer.add_trace(spans)

    # Find critical path
    critical_path = analyzer.analyze_critical_path(spans)
    print("\nCritical Path (longest duration):")
    for span in critical_path:
        print(f"  {span.name}: {span.duration:.3f}s")

    # Detect anomalies
    anomalies = analyzer.detect_anomalies(spans)
    if anomalies:
        print("\nDetected Anomalies:")
        for anomaly in anomalies:
            print(f"  {anomaly['name']}: {anomaly['duration']:.3f}s "
                  f"(expected ~{anomaly['mean_duration']:.3f}s, "
                  f"z-score={anomaly['z_score']:.1f})")

    # Visualize trace
    print("\nTrace Visualization:")
    visualization = analyzer.visualize_trace(spans, max_width=60)
    print(visualization)

    # 5. W3C Trace Context Propagation
    print("\n\n5. W3C Trace Context Propagation:")
    print("-"*40)

    # Create context
    context = SpanContext.generate(sampled=True)
    context.trace_state.add("vendor1", "value1")
    context.trace_state.add("vendor2", "value2")

    print("Original Context:")
    print(f"  Trace ID: {context.trace_id}")
    print(f"  Span ID: {context.span_id}")
    print(f"  Sampled: {context.is_sampled}")
    print(f"  Trace State: {context.trace_state.to_header()}")

    # Convert to headers
    headers = context.to_headers()
    print("\nHTTP Headers for Propagation:")
    for k, v in headers.items():
        print(f"  {k}: {v}")

    # Parse from headers
    parsed = SpanContext.from_headers(headers)
    print("\nParsed Context:")
    print(f"  Trace ID: {parsed.trace_id}")
    print(f"  Span ID: {parsed.span_id}")
    print(f"  Sampled: {parsed.is_sampled}")
    print(f"  Trace State: {parsed.trace_state.to_header()}")
    print(f"  Valid: {parsed.is_valid}")

    # 6. Statistics and Monitoring
    print("\n\n6. Tracing Statistics:")
    print("-"*40)

    stats = tracer.get_statistics()
    for key, value in stats.items():
        print(f"  {key}: {value}")

def tracing_mathematical_foundations():
    """
    Mathematical foundations of distributed tracing
    """
    print("\n" + "="*70)
    print("MATHEMATICAL FOUNDATIONS OF DISTRIBUTED TRACING")
    print("="*70)

    print("\nDefinition 1: Happens-Before Relation (→)")
    print("  For events a, b in distributed system:")
    print("    1. If a and b are in same process and a occurs before b, then a → b")
    print("    2. If a is sending message m and b is receiving m, then a → b")
    print("    3. If a → b and b → c, then a → c (transitivity)")

    print("\nTheorem 1: Vector Clocks Capture Causality")
    print("Proof:")
    print("  For vector clocks V, W:")
    print("    V < W iff ∀i: V[i] ≤ W[i] and ∃j: V[j] < W[j]")
    print("  Then: a → b iff VC(a) < VC(b)")
    print("  Proof by induction on causal paths. ✓")

    print("\nTheorem 2: Sampling Rate and Error Bound")
    print("  For sampling rate p, estimation error ε:")
    print("    Pr(|X̂ - X| ≥ εX) ≤ 2 exp(-pε²X/3)")
    print("  where X is true value, X̂ is sampled estimate")
    print("  Proof using Chernoff bound. ✓")

    print("\nTheorem 3: Critical Path Identifies Bottleneck")
    print("Proof:")
    print("  Let G = (V, E) be trace DAG with weights = durations")
    print("  Critical path = longest path from source to sink")
    print("  By definition, reducing any node on critical path")
    print("  reduces total trace duration. ✓")

    print("\nTheorem 4: Optimal Sampling for Rare Events")
    print("  For rare event with probability q, optimal sampling:")
    print("    p* = min(1, √(N/qT))")
    print("  where N is sampling cost, T is trace budget")
    print("  Proof via Lagrange multiplier. ✓")

def trace_analysis_algorithms():
    """
    Algorithms for trace analysis
    """
    print("\n" + "="*70)
    print("TRACE ANALYSIS ALGORITHMS")
    print("="*70)

    algorithms = [
        ("Critical Path Analysis", """
        Input: Trace DAG G = (V, E) with weights w(v)
        Algorithm:
          1. Topological sort of vertices
          2. For each vertex v in topological order:
              dist[v] = max(dist[u] + w(u) for u in predecessors(v))
              prev[v] = argmax_u(dist[u] + w(u))
          3. Reconstruct path from sink to source using prev[]
        Complexity: O(|V| + |E|)
        """),

        ("Anomaly Detection (Statistical)", """
        Input: Span durations for operation O
        Algorithm:
          1. Collect historical durations: X = {x₁, ..., xₙ}
          2. Compute μ = mean(X), σ = stddev(X)
          3. For new observation x:
              z = |x - μ| / σ
              If z > threshold (e.g., 3), flag as anomaly
          4. Use robust statistics (median, MAD) for outliers
        """),

        ("Trace Compression (Lossy)", """
        Input: Trace T with n spans
        Algorithm:
          1. Build span similarity graph
          2. Cluster similar spans (same name, similar attributes)
          3. For each cluster, keep k representative spans
          4. Reconstruct approximate trace
        Compression ratio: O(n/k)
        """),

        ("Root Cause Analysis", """
        Input: Failed trace T_f, successful traces T_s
        Algorithm:
          1. Extract features from traces (durations, paths, attributes)
          2. Train classifier or compute similarity
          3. Find most divergent spans in T_f
          4. Rank potential root causes by divergence score
        Accuracy depends on training data
        """),
    ]

    for name, description in algorithms:
        print(f"\n{name}:")
        print(description)

if __name__ == "__main__":
    demonstrate_distributed_tracing()
    tracing_mathematical_foundations()
    trace_analysis_algorithms()

This is the beginning of a completely comprehensive guide to distributed systems theory and practice.
Share your ideas, comments and your questions with us by your comments here! We and the community will answer you 24/7!
I hope you enjoy, have nice times!