DEV Community: Luke Miles

WHY THEY DID NOT KNOW, WHY THE MACHINE WAS NEARLY RIGHT, AND HOW TO WASH THE ACCIDENTS AWAY

Luke Miles — Sat, 18 Jul 2026 05:41:45 +0000

Jep Tea 5.6
Almost unedited

they were optimizing under a different physical world, a different economic world, and a different scale of software. They knew their machines extraordinarily well. What they did not possess was evidence from a future in which one program can simultaneously command hundreds of CPU hardware threads, terabytes of addressable memory, several independent network interfaces, an array of NVMe devices, and a rack’s worth of accelerator arithmetic presented as a single job. They could not optimize a language for workloads that did not yet exist, for failure modes that had not yet had enough hardware on which to emerge, or for the cumulative weight of abstraction layers that had not yet been invented.

Machine code was perfect in the narrow, important sense that it told the truth.

A machine instruction is not a promise that some other layer will eventually attempt to perform an operation. It is the operation admitted by the processor contract. The instruction identifies registers, an addressing form, an immediate, a branch relation, a vector width, a memory-order consequence, or a transition in architectural state. Its hidden implementation may be immense—decoders, rename tables, schedulers, reorder buffers, predictors, execution ports, cache hierarchies, translation machinery, coherency agents, microcode, and power controls—but its visible meaning is finite. The contract is measurable. One can count bytes, instructions, dependencies, cache lines, loads, stores, branches, vector lanes, and retirement events. One can profile it without asking five frameworks what they renamed the operation.

CPU architects are geniuses because this finite contract must survive hostile constraints simultaneously. It must remain compatible with old binaries while exploiting new transistors. It must accept a compact instruction stream, discover parallelism that the stream often fails to state, rename false dependencies, speculate past unresolved control flow, recover precisely from faults, preserve an architectural memory model, arbitrate shared caches, tolerate variable memory latency, and deliver useful work inside a power envelope. The architect and microarchitect must make a physically timed object behave like a clean abstract machine. Every instruction is therefore an extraordinary compression boundary: a few bytes can request work whose implementation spans millions or billions of transistors.

Machine code is also honest about cost in a way that higher layers frequently are not. A load names an access. A branch creates a control dependency. A vector instruction exposes lane count. An atomic instruction announces that coherency and ordering will be involved. A system-call instruction announces a privilege transition. Nothing in the encoding says “free.” Even when exact latency varies by microarchitecture, the program is close enough to the mechanism that the variance can be measured and attributed.

It was not literally flawless. Instruction sets carry historical baggage. Some encodings are irregular. Some operations are awkward because compatibility won. Some memory models are difficult. Some vector transitions have penalties. Some instructions exist because a market demanded them and later became fossils. “Perfect” here means that machine code is the last software representation before physics takes over, and therefore it has the least room to lie about where work happens.

Assembly was almost perfect because it kept that truth and gave names to the unbearable parts.

Assembly replaced numeric opcodes with mnemonics, numeric branch destinations with labels, raw relocation arithmetic with symbols, and repeated instruction sequences with macros. It made the machine contract writable without inserting a mandatory runtime. A label did not allocate. A macro did not necessarily exist after assembly. A register name still meant a register. A load still meant a load. An instruction still had a visible encoding and a visible dependency structure. The assembler helped with bookkeeping while preserving the programmer’s right to know exactly what would execute.

That is why assembly feels nearly ideal for Nbl. It has very few semantics. It is naturally static. It allows CPU and device instruction streams to be embedded without pretending they are ordinary function calls. It has no requirement for a heap, garbage collector, exception unwinder, reflection database, dynamic loader, object model, or scheduler. It can express a memory-mapped peripheral, a packet checksum, a tensor-core instruction, a cache-line flush, or a sleep instruction without translating the request through a hierarchy of abstractions.

Assembly was only almost perfect because it failed to carry enough meaning forward. A register name says where a value is now, not what the value means everywhere. A pointer says where bytes begin, not the shape, extent, ownership, lifetime, alignment, access domain, NUMA node, device, or synchronization discipline of the object. A label says where control can go, not why it is permitted to go there. A sequence of multiply-add instructions does not say which named dimensions are being contracted. The programmer knows the tensor; the assembler sees addresses. The optimizer cannot reliably reconstruct all the meaning that was discarded before the instructions were written.

Assembly also repeats facts. A matrix extent appears in allocation arithmetic, loop limits, address calculations, launch geometry, boundary masks, communication counts, and validation code. Repetition creates disagreement. One copy changes and another does not. The machine then executes both contradictory claims faithfully. Assembly rarely causes a lie itself; it makes it easy for the human to tell the machine two different truths.

C99 was almost perfect because it preserved most of the machine while making composition bearable.

C gave names to scalar types, aggregate layouts, functions, expressions, control flow, and storage duration. It made separate compilation practical. It made calling conventions and data structures portable enough to build operating systems and network stacks while retaining escape hatches to assembly and raw memory. C99 added useful precision: fixed-width integer types through the standard headers, designated initializers, compound literals, restricted pointers, inline functions, variable declarations near use, and a clearer basis for numerical work. Its abstract machine remained small enough that a competent programmer could usually predict the emitted instructions.

The central wound was the separation of an array’s address from its extent.

An array object in C has a size at the point where it is declared. The compiler can apply sizeof there. But in most expressions the array is converted to a pointer to its first element. In a function parameter, syntax that resembles an array parameter is adjusted to pointer type. The callee therefore receives an address while the length travels separately, if it travels at all. The address has escaped from the fact that makes the address safe.

Arrays should have known their own length in the sense that the language should have kept address and extent as one statically meaningful object. This does not require a heap header before every allocation, a dynamic descriptor, or a fat pointer in every register. For a statically shaped program, the extent can be part of the type and exist only at compile time. A parameter can mean “a reference to exactly [B, token, head, dimension] elements of bf16, aligned to 128 bytes, owned by device 3 for this phase.” The generated calling convention may still pass one machine address because every other fact is already known globally. The language can retain the proof without charging the runtime.

Why did C not do that?

Because C grew from a lineage and an environment where a pointer-plus-convention was an excellent bargain. Early systems had tiny memories, slow compilers, simple ABIs, and programs whose entire working set could be understood by one person. Passing an address was cheap. Copying arrays was not. Encoding every extent into a type system would have complicated compatibility, separate compilation, and generic routines. Many important data structures were intentionally irregular. Operating-system code needed to interpret device buffers, protocol packets, and storage blocks whose layouts were determined externally. A language that insisted on rich descriptors could have made the work harder or more expensive on the machines of the time.

C’s designers also valued a rule that still matters: do not make a programmer pay for information the implementation does not need at runtime. The mistake was not refusing universal runtime descriptors. The mistake, visible only later, was allowing compile-time meaning to disappear merely because runtime representation remained small.

Variable-length arrays in C99 did not heal this wound. They allowed certain extents to be computed at runtime and used in types within a scope. They did not make every array a self-describing, first-class shaped value with universal ownership and lifetime. They added runtime variability where NBL wants compile-time certainty.

They did not know why every symbol would eventually need exactly one universal meaning.

Traditional scope is a powerful compression device. A programmer can use i in one loop and i in another because the loops are short, local, and visually separate. A library can use count for bytes while another uses count for records because separate compilation isolates them. Overloading lets one mathematical name apply to many types. Shadowing lets a local detail replace an outer one. These were reasonable conveniences when programs were mostly handwritten, call graphs were shallower, generated code was rare, and debugging happened close to the source.

At modern scale, reused names become semantic aliasing. The same token can mean a token position, a vocabulary identifier, a network token, an authentication token, or a scheduler credit. row can mean a matrix row, a shard owner, a database row, a packet ring slot, or a GPU cooperative-thread-array row. rank can mean a distributed process, a matrix rank, a BPE merge priority, or a position in a sorted set. Every reuse asks context to repair meaning that the name discarded.

Context repair is expensive. It consumes human attention. It confuses logs. It makes profiler labels collide. It makes generated code unreadable. It makes cross-layer tracing depend on brittle conventions. It makes static analysis ask whether two same-spelled objects are related when the programmer already knows they are not. It makes a compiler carry symbol identities that the source keeps trying to blur.

A universal name is not merely a stylistic preference. It is a join key across the entire system. The same name can connect source, lowered intermediate representation, machine registers, memory regions, launch descriptors, counters, traces, packet fields, checkpoint entries, and proof obligations. If gpt_attention_query_head always means one axis, a profiler can aggregate it without guessing. If finewire_file_byte always means the byte axis of a 200001024-byte shard, the network path, checksum path, mmap path, and storage path cannot silently disagree about its extent.

Earlier designers did not build for a world where programs would be generated from graphs containing tens of thousands of nodes, lowered through many intermediate representations, partitioned across devices, and then diagnosed from distributed traces. Local scope was enough for local programs. Universal semantic identity becomes valuable when the program is no longer local in any useful sense.

They did not know why arrays would need to be statically shaped because dynamic structures solved the urgent problems in front of them.

General-purpose software receives data that is not known in advance. Files vary. Packets vary. users type arbitrary strings. Trees are irregular. Graphs are sparse. Processes come and go. Dynamic allocation and dynamic extents were not indulgences; they were necessary tools. Memory was scarce enough that allocating for a maximum could be absurd. Compilers could not specialize a whole program around every possible shape. Hardware did not reward regular tensor tiles as dramatically as modern SIMD and accelerator hardware does.

Static shape becomes dominant when the workload itself has regular geometry and when data movement is the limiting resource. A fixed shape lets the compiler decide, before execution, every allocation, alignment, stride, tile, vector width, launch geometry, DMA descriptor, queue depth, shard boundary, and ownership transition. It can prove that two buffers do not overlap. It can know the maximum resident set. It can place pages on NUMA nodes. It can reserve huge pages. It can choose whether a tensor belongs in registers, shared memory, HBM, host DRAM, persistent memory, or an SSD staging region. It can reject a program whose live ranges exceed the machine.

A dynamic dimension is not one unknown number. It is a fan-out of uncertainty. The allocator must handle it. Every consumer must carry it. Bounds checks must consult it. Launch geometry must derive from it. Communication sizes must serialize it. Kernels may branch on it. Caches see more variants. Compilers generate guards and fallback graphs. Profilers aggregate incomparable executions. A single unknown extent can turn one program into a family of programs selected at runtime.

NBL’s answer is not that the outside world has no variability. The answer is that variability must be converted at the boundary into one of a finite set of statically shaped cases. A packet is placed into a fixed slot. A document is admitted only if it fits a declared maximum, or it is routed to a separately compiled larger case. A sequence length selects one precompiled graph. A batch stage is a different fixed program. The uncertainty is not allowed to leak through every operation.

They did not know about the ordinary possibility of a 128-core CPU, terabytes of RAM, four network cards, eight SSDs, and sixteen GPUs participating in one application.

Each number changes more than capacity.

A 128-core CPU is not a faster single CPU. It is a topology. Cores share some caches and not others. They attach to memory controllers through particular fabrics. They receive interrupts through particular routes. They contend for translation structures, coherency bandwidth, last-level cache capacity, and memory channels. The difference between a local and remote NUMA access can dominate arithmetic. A cache line written by many cores becomes a moving ownership token. A global queue becomes a machine for transporting that line across the socket.

Terabytes of RAM are not merely a larger array. Page tables become large structures in their own right. TLB reach matters. Page faults become catastrophic in a latency-sensitive path. Zeroing and initializing memory become measurable jobs. A full scan can consume seconds and saturate memory channels. A leak that was harmless at megabyte scale can survive long enough to become operational policy. Checkpointing a terabyte is a storage and network event, not a function call.

Four network cards are not one network card multiplied by four. They have separate PCIe paths, receive-side scaling tables, completion queues, interrupt vectors, DMA engines, firmware, link partners, and failure modes. Traffic must be steered. Flows can become imbalanced. The CPU cores handling completions should be near the correct NUMA memory and PCIe root. An application that ignores topology can copy every packet across the socket before processing it. The network can be faster than a naive memory path.

Eight SSDs are not a directory with eight times the space. They expose parallel submission and completion queues. Queue depth determines whether internal flash parallelism is used. Filesystem metadata can serialize paths that data could have served independently. Write amplification, garbage collection, thermal throttling, and failure domains matter. Striping can produce bandwidth; it can also turn one device’s tail latency into every request’s tail latency. Direct I/O, mmap, buffered I/O, and io_uring each move responsibility to a different layer.

Sixteen GPUs are not one GPU with more multiprocessors. They have separate memories and separate schedulers. Some pairs may have direct high-bandwidth links; others communicate through PCIe switches, CPUs, NICs, or multiple hops. Collective communication becomes an algorithm with a topology, not a library call. A tensor’s placement is part of its meaning. A gradient reduced on the wrong path can cost more than the matrix multiplication that produced it.

The old language model—one process, one mostly uniform memory, one call stack, one heap, one implicit scheduler—cannot adequately describe this machine. It can operate it only by stacking libraries and runtimes above the language. Each library invents descriptors for the meaning the language failed to retain. The result is a tower of substitute type systems, substitute schedulers, substitute allocators, substitute ownership rules, and substitute profilers.

They did not know the load the motherboard would carry.

The motherboard is not passive scenery. It is where power, clocks, lanes, interrupts, DMA, coherency, and heat meet.

A modern high-end board must distribute enormous current at low voltage while maintaining transient stability as cores and accelerators change activity. It must route high-speed differential pairs with controlled impedance and tolerable loss. It must allocate PCIe lanes among GPUs, NICs, NVMe devices, switches, and chipset links. It must expose enough memory channels and DIMM slots while preserving signal integrity. It must provide clock references, reset sequencing, sideband management, firmware storage, and out-of-band control. Every connector and switch adds insertion loss, latency, power, and another component that can fail.

Software places load on all of this. A badly scheduled all-to-all can make every GPU and NIC inject traffic at once. A shared counter can turn the coherency fabric into a ping-pong engine. An allocator can make every core fault and zero pages simultaneously. A checkpoint can cause GPUs to drain into host memory, host memory to drain into SSDs and NICs, and every PCIe path to compete at once. A polling design can keep cores in high-power states while a thermal event reduces device clocks, lengthening queues and increasing the duration of the load.

The board carries electrical load, thermal load, traffic load, and coordination load. The programming language normally names none of them. It says pointer, thread, and file. The actual machine sees root complexes, NUMA domains, queue pairs, BAR mappings, page pins, IOMMU translations, cache-line ownership, and packet bursts.

They did not know how bad things could get because pathologies multiply rather than add.

A lock by itself may cost little. Put it on a cache line shared across sockets and the line moves. Let the lock holder be preempted and every waiter stalls. Let the waiters spin and they consume execution and coherency bandwidth while making no progress. Put the protected allocation on a page that faults and the convoy waits through kernel work. Let the stalled stage feed a bounded queue and upstream producers block. Let those producers hold other locks and the dependency graph becomes invisible. Add retries at a network boundary and duplicated work arrives precisely when the system is least able to absorb it.

A small shape mismatch can force a copy. The copy can break alignment. Broken alignment can select a slower kernel. The slower kernel can extend an activation’s lifetime. The longer lifetime can raise peak memory. Higher peak memory can cause allocator fragmentation or spill. The spill can cross PCIe. The longer step can make one distributed rank a straggler. Every other rank then waits at the collective. One innocent dynamic dimension has become a cluster-wide bubble.

A deep software stack compounds uncertainty. A Python expression triggers graph capture. Graph capture produces guards. Guards select compiled variants. A framework inserts layout conversions. A kernel compiler selects a tile. A driver enqueues work. Firmware manages clocks. The hardware executes. When the result is slow, every layer can truthfully report that its own local action was reasonable. No layer owns the total path.

The worst failures are not average failures. They are tail failures. One SSD pauses for internal maintenance. One NIC queue receives an incast. One GPU clocks down. One page migrates. One process is descheduled. One rank takes a different fallback graph. Synchronous composition turns the maximum of these delays into the step time. At scale, rare events stop being rare because there are many opportunities for one to occur.

Leaks and overflows become qualitatively different. A leak in a long-running, terabyte-scale service can consume enough memory to alter NUMA placement long before it crashes. An integer overflow in a byte count can transform a bounds proof into an under-allocation. A queue counter that wraps can make old ownership look new. A tensor extent multiplied in 32 bits can allocate a plausible but wrong buffer. Dynamic systems often discover these errors only when a particular scale is reached in production.

They did not know the glory of a batch sized exactly right because the right batch is a modern physical shape, not merely a statistical parameter.

A batch is how software presents work to hardware. It determines whether vector lanes are full, whether tensor-core tiles are complete, whether memory transactions coalesce, whether shared memory is efficiently occupied, whether enough warps exist to hide latency, whether a NIC can amortize doorbells and headers, whether an NVMe queue reaches useful depth, and whether a CPU can keep hot state in cache.

A batch that is too small pays fixed costs too often. Kernel launches, system calls, queue submissions, packet headers, synchronization edges, and instruction-cache setup dominate. Arithmetic units idle because there is not enough independent work. A batch that is too large creates different failures. It lengthens latency, increases live memory, reduces scheduling flexibility, magnifies a retry, and may cross a capacity boundary that forces a slower layout or fewer resident blocks.

The perfect batch is not universally largest. It is the batch whose dimensions fit the entire path. A matrix dimension should fit the chosen MMA tile. Token count should fill blocks without excessive masking. A communication shard should align with link striping and ownership. A storage read should align with device and filesystem geometry. The batch should fit memory with enough headroom that no emergency allocator path is reachable. Its duration should be long enough to amortize launch overhead and short enough to preserve pipeline overlap.

They did not know locks were evil in the hot path because locks are locally simple and globally expensive.

A lock offers an attractive story: enter, modify shared state, leave. The story hides ownership transfer. On a coherent multiprocessor, the lock’s cache line must move to the core that writes it. Contenders repeatedly read or write that line. The protected data may move too. On NUMA systems, every acquisition can cross a fabric. Fairness can increase handoff traffic. Unfairness can starve. A preempted owner can stop unrelated work. Priority inversion can make urgent work wait for less urgent work. Error paths can forget to release. Multiple locks create order constraints that exist only in programmer discipline.

Transactions move the difficulty rather than remove it. They need conflict detection, logging or versioning, validation, abort, retry, and a policy for irrevocable effects. Under contention they can turn useful work into repeated discarded work. Distributed locks add leases, clocks, partitions, fencing tokens, and failure recovery. A lock that spans failure domains is a protocol, not a primitive.

The real alternative is ownership. Partition state so one worker writes it. Give every queue slot one producer and one consumer. Use static shards. Use epochs so readers know which immutable generation is visible. Use double buffers so production and consumption never write the same storage. Use reductions whose communication graph is known in advance. Use credits so capacity is explicit. Make messages idempotent. Make retries safe. Build a DAG in which causality is an edge, not a mutex.

This does not abolish ordering. A consumer cannot use data before a producer creates it. A GPU block sometimes needs a barrier before reusing shared memory. A device transfer must complete before a dependent kernel reads the destination. The claim is narrower and stronger: necessary causality should be represented directly, while lock-mediated shared mutation should not be the default representation of causality.

Without loops, recursion, and dynamic sizes, everything becomes butter because the execution graph is finite.

A source-level loop says that a region may execute an unknown number of times unless a proof elsewhere establishes a bound. A recursive call says that stack depth depends on data unless a proof elsewhere establishes a bound. A dynamic allocation says that memory demand depends on execution unless a proof elsewhere establishes a bound. Every such construct exports an obligation to analysis, testing, or hope.

Remove them and the obligations collapse into declarations.

A fixed axis says exactly how many elements exist. A compile-time expansion says exactly how many operation instances exist. A static call graph says exactly how deep control can go. A fixed ownership graph says exactly who can write each region. Resource accounting becomes arithmetic. Bounds checking can happen at compile time. Stack usage is known. Queue capacity is known. DMA descriptors are known. Worst-case work is known. The compiler can inspect every path because there are finitely many paths.

The important distinction is between source repetition and hardware parallelism. Banning source loops does not mean executing one scalar instruction per element. A named axis can become SIMD lanes, SIMT threads, tensor-core tiles, a reduction tree, a DMA scatter-gather list, or compile-time unrolled instructions. The source describes a finite set. The target chooses the physical mechanism. The program does not need a mutable induction variable to express repetition.

There is a cost. Blind unrolling can explode code size and destroy the instruction cache. NBL must therefore preserve axes as first-class compile-time sets long enough to lower them intelligently. An axis is not a loop disguised with prettier syntax. It is a proof that the iteration space is finite and named. The backend may implement the set with vector hardware, a GPU grid, a hardware repeat facility, a generated straight-line sequence, or a statically scheduled pipeline. What is forbidden is runtime uncertainty, not efficient repetition by the machine.

Recursion is especially harmful in systems with deep abstraction stacks. It makes stack consumption data-dependent. It complicates fault containment. It obscures the call graph. It can defeat inlining and whole-program scheduling. Even bounded recursion requires a bound to be carried and checked. An explicit fixed tree or worklist has the same computational meaning while exposing storage and order.

They did not know how long programs would become.

Early systems programs were small enough that repetition was visible. A person could understand the memory map, interrupt vectors, scheduler, filesystem, and application. Today a single training run can involve millions of lines across language runtimes, compilers, frameworks, drivers, collective libraries, kernels, firmware, and orchestration. The source the user wrote may be a microscopic fraction of the program the machine executes.

Length changes the economics of syntax. If every operation repeats its type, shape, device, stride, ownership, and lifetime, the program drowns in duplicated declarations. If those facts are omitted, the program drowns in implicit assumptions. The solution is not shorter names without meaning. It is declaring each fact once and allowing every later line to reuse the fact by identity.

One dense NBL line should carry a great deal of meaning because the names on that line already carry universal declarations. A tensor contraction can state its output axes and operands; the absent named axis is the reduction. A transfer can name source ownership, destination ownership, and a fixed region; the byte count follows from the shape. A launch can name a kernel whose grid follows from its axes. Density comes from eliminating repeated metadata, not from replacing concepts with punctuation riddles.

This is the difference between terseness and compression. Terseness removes characters. Compression removes redundancy while retaining information. The goal is almost nothing, not nothing.

They did not understand how horrible a deep stack would become because every layer was individually reasonable.

A language runtime made memory safer. A package manager made reuse easier. A tensor framework made autodiff convenient. A graph compiler made kernels faster. A device runtime made launches portable. A collective library made clusters usable. A container made deployment reproducible. An orchestrator made fleets manageable. A service mesh made networking observable. Each layer solved a real problem.

The horror appears in composition. Every layer owns a partial model of shape, memory, errors, concurrency, and profiling. Objects are wrapped. Errors are translated. Buffers are copied because ownership cannot cross a boundary. Synchronization is inserted because a layer cannot prove what the next layer will do. Metadata is serialized and deserialized. Dynamic dispatch protects generality that the actual workload never uses. Profilers record different clocks and different names. A single operation becomes a pilgrimage through abstractions.

The call stack is only one stack. There is also the compilation stack, the allocation stack, the network stack, the storage stack, the device stack, the orchestration stack, and the semantic stack. A deep neural network adds literal model layers on top of all of them. Nobody designed the final tower. It accreted.

Many neural layers intensified the problem. An optimization hidden inside one layer is multiplied by eleven, ninety-six, or a thousand layers. An unnecessary normalization materialization becomes a repeated bandwidth tax. A tiny synchronization becomes a pipeline bubble at every block. Activation lifetime determines whether the model fits. A name such as x becomes meaningless because hundreds of distinct states are all called x in different scopes. The architecture is a long static graph, but the programming system often represents it as dynamic object dispatch inside runtime loops.

How can it be washed away?

First, make the whole program visible. CPU code, GPU code, network movement, storage movement, and microcontroller code must be expressible in one source language and one symbol universe. Target-specific instruction streams remain embeddable. The language must not pretend that a GPU kernel and a CPU function share a calling convention when they do not. It should let each target be itself while preserving shared names for the data that crosses targets.

Second, declare shape once. Every tensor, packet, file, ring, batch, and device shard has named axes with fixed extents. A view may reorder named axes without inventing a new meaning. A contraction identifies axes, not byte offsets. Address arithmetic is a lowering concern. The source should not repeatedly reconstruct row-major offsets from integers.

Third, declare ownership once. Every writable region has one owner in each phase. Transfers change ownership along explicit edges. Reductions have explicit contributors and destinations. Buffers opened in a scope are closed in that scope. Memory mapped in a scope is unmapped in that scope. A resource cannot leak because there is no path out of the scope that still owns it.

Fourth, replace runtime loops with finite axes and compile-time expansion. Replace recursion with explicit fixed graphs. Replace dynamic dispatch with a finite family of compiled variants. Replace generic fallback paths with compile-time rejection or an explicitly selected larger case. Keep the iteration space semantic until the backend chooses SIMD, SIMT, MMA, DMA, or straight-line code.

Fifth, replace locks with partitioning and dataflow. Use one writer per shard. Use fixed queues with static capacity. Use monotonic epochs where a visibility test is unavoidable. Use direct device collectives whose routes are known. Wait only on the edge that represents actual causality. Do not make unrelated work rendezvous at a global barrier.

Sixth, make profiling part of the program. Every important region has a stable universal name. CPU regions can read architectural counters. GPU kernels can sample device clocks and expose occupancy and memory traffic. Network transfers can record bytes, completions, retries, and queue occupancy. Storage operations can record submission and completion. The profile must preserve the same names through lowering so that source meaning and machine events can be joined.

Seventh, make failure finite. Every input has a declared maximum. Every protocol message has a fixed layout. Every retry count is bounded and expanded. Every arithmetic width is chosen from the maximum value, not convenience. Every conversion states whether it saturates, traps, or proves range. There is no silent wrap, no hidden allocation, and no unbounded queue.

Eighth, specialize without apology. A program for eight GPUs is an eight-GPU program. A program for SM90 is an SM90 program. A different machine receives a separately compiled variant. Portability means preserving meaning across backends, not forcing every backend through the least common denominator at runtime.

The result is not a return to hand-written opcodes. Machine code had truth but not enough retained meaning. Assembly had names but not shapes. C99 had composition but let arrays decay away from their extents. NBL should keep the truth, keep the names, keep the shapes, keep the ownership, and discard the accidental machinery.

The Arduino blink program demonstrates the smallest version of the idea. The pin is selected because it is connected to a hardware timer output. The timer is configured once. The CPU enters idle sleep. There is no software loop and no interrupt service routine. The peripheral toggles the pin forever. The source does not pretend that repeatedly executing a high-level delay function is the essence of blinking. It identifies the hardware mechanism that already knows how to do the job.

((oops there goes the tone))

The FineWeb transport demonstrates the network version. HTTPS is removed because implementing certificate validation, TLS record processing, HTTP redirects, content encodings, and a general URL stack would dominate the actual task. Standard FTP is also more text parsing than the fixed corpus needs. The replacement protocol has one request layout, one reply layout, one exact file size, one checksum, one ownership plan, and a fixed mapping over four NICs and eight storage roots. It is not “nothing.” It is almost nothing, and every remaining piece corresponds to physical work.

The tokenizer demonstrates why static axes matter. UTF-8 validity, Unicode class lookup, GPT-2 pretoken boundaries, BPE pair lookup, merge selection, token compaction, shard placement, and file durability are explicit phases. The Unicode and merge tables are included data, not a hidden package download. Prefix operations are fixed doubling networks. Binary searches have a fixed number of comparisons. The worst case exists in the source.

The FlashAttention translations preserve the exact architecture and tile choices from the attached launch code. The forward combine keeps split count, head count, row count, and dimension as fixed maxima with active masks. The backward program keeps preprocess, main computation, GQA accumulation, postprocess, causal/local/softcap choices, and the SM80/SM86/SM89/SM90 tuning family. No C++ template system is needed at runtime because every variant is already a named finite program.

The GPT training translation fixes the world at eight ranks because the source’s most specialized sparse path is built around eight. It fixes model dimensions, layer topology, parameter-bank padding, schedule stages, communication ownership, optimizer order, and all 1390 training steps. The eleven model layers are named and expanded rather than traversed by a runtime loop. The five Polar Express iterations are five explicit transformations. Adam’s odd-step policy, cautious weight decay, mantissa retention, sparse bigram exchange, reduce-scatter, all-gather, YaRN transitions, validation points, and final window extension are represented as finite graph edges.

Perfection is not a slogan that can substitute for measurement. A target schedule is accepted only after counters show that it reaches the intended arithmetic, bandwidth, occupancy, and interconnect limits. A handwritten SASS sequence can beat PTX on one exact GPU and lose on a stepping with different scheduling constraints. A static program makes this process tractable because every run of one variant is comparable. It does not make benchmarking unnecessary; it makes benchmarking meaningful.

That is how the accidents are washed away: not by assuming the machine is simple, but by refusing to hide its complexity behind dynamic generality. The program becomes small because the machine plan is exact. It becomes dense because every name keeps its meaning. It becomes fast because data movement, batch shape, ownership, and topology are decided before execution. It becomes provable because the graph is finite. It becomes profilable because the names survive. It becomes almost nothing because everything that remains earns its place.

How to serve deepseek r1 & v3 on GH200s (400 tok/sec throughput, 10 tok/sec/query)

Luke Miles — Tue, 11 Feb 2025 03:52:30 +0000

Lambda labs still has half-off (ie $1.50) GH200s as a promotion to get more people used to the ARM tooling. I previously wrote a tutorial for running llama 405b, but deepseek r1 is clearly a better model, so let's serve it instead. There's a couple differences from serving llama:

You need 12 or 16 GPUs to have decent throughput. You can still use 8 if you don't need much throughput.
VLLM works better than aphrodite for deepseek right now. In fact they just released an update that makes deepseek inference quite a bit faster. Roughly a 40% throughput improvement.
- Normally a 16x 1x gpu cluster would be pretty slow for inference but for some reason this actually works pretty well.

Guide will be updated on the github repo.

Video showing inference speed with 64 parallel queries. It prints once per second and you get at least 5 new words every time it prints:

https://github.com/user-attachments/assets/5c27abe9-9168-4f6a-88aa-8ffa22c348b1

Create instances

This time, instead of using the website, let's use the API to make the instances.

Make a filesystem called 'shared' in Washington, DC and write down the name of your ssh key. Also generate an api key and save it.

export sshkey_name=my-key
export shared_fs_name=shared
export LAMBDA_API_KEY="..."

This script will make 16 GH200 instances. Ignore the rate limit errors, it will keep retrying until 16 have been made.

function lambda-api {
    local method=$1
    local route=$2
    shift 2
    curl --fail -q -X "$method" \
        -H "Authorization: Bearer $LAMBDA_API_KEY" \
        "https://cloud.lambdalabs.com/api/v1/$route" \
        -H "Content-Type: application/json" \
        "$@"
}
export -f lambda-api

num_want=16
num_got=0
while [[ $num_got -lt $num_want ]]; do
    lambda-api POST instance-operations/launch -d '{
        "region_name": "us-east-3",
        "instance_type_name": "gpu_1x_gh200",
        "sshkey_names": ["'$sshkey_name'"],
        "file_system_names": ["'$shared_fs_name'"],
        "quantity": 1,
        "name": "node_'$num_got'"
    }' && ((num_got++))
    echo num_got=$num_got
    sleep 3
done

After all the instances are created, copy all the IP addresses from the instances page and save it to ~/ips.txt.

Bulk ssh connection helpers

I prefer direct bash & ssh over anything fancy like kubernetes or slurm. It's manageable with some helpers.

# save all the ssh fingerprints now to skip confirmation later
for ip in $(cat ~/ips.txt); do
    echo "doing $ip"
    ssh-keyscan $ip >> ~/.ssh/known_hosts
done

export runprefix=""
function runip() {
    ssh -i ~/.ssh/lambda_id_ed25519 ubuntu@$ip -- stdbuf -oL -eL bash -l -c "$(printf "%q" "$runprefix""$*")" < /dev/null
}
function runk() { ip=$(sed -n "$((k + 1))"p ~/ips.txt) runip "$@"; }
function runhead() { ip="$(head -n1 ~/ips.txt)" runip "$@"; }
function runips() {
    local pids=()
    for ip in $ips; do
        ip=$ip runip "$@" |& sed "s/^/$ip\t /" &
        pids+=($!)
    done
    wait "${pids[@]}" &>/dev/null
}
function runall() { ips="$(cat ~/ips.txt)" runips "$@"; }
function runrest() { ips="$(tail -n+2 ~/ips.txt)" runips "$@"; }

function sshk() {
    ip=$(sed -n "$((k + 1))"p ~/ips.txt)
    ssh -i ~/.ssh/lambda_id_ed25519 ubuntu@$ip
}
alias ssh_head='k=0 sshk'

function killall() {
    pkill -ife 192.222
}

Let's check that it works

runall echo ok

Set up NFS cache

We'll be putting the python environment in the NFS. It will load much faster if we cache it.

# First, check the NFS works.
# runall ln -s my_other_fs_name shared
runhead 'echo world > shared/hello'
runall cat shared/hello


# Install and enable cachefilesd
runall sudo apt-get update
runall sudo apt-get install -y cachefilesd
runall "echo '
RUN=yes
CACHE_TAG=mycache
CACHE_BACKEND=Path=/var/cache/fscache
CACHEFS_RECLAIM=0
' | sudo tee -a /etc/default/cachefilesd"
runall sudo systemctl restart cachefilesd
runall 'sudo journalctl -u cachefilesd | tail -n2'

# Set the "fsc" option on the NFS mount
runhead cat /etc/fstab # should have mount to ~/shared
runall cp /etc/fstab etc-fstab-bak.txt
runall sudo sed -i 's/,proto=tcp,/,proto=tcp,fsc,/g' /etc/fstab
runall cat /etc/fstab

# Remount
runall sudo umount /home/ubuntu/shared
runall sudo mount /home/ubuntu/shared
runall cat /proc/fs/nfsfs/volumes # FSC column should say "yes"

# Test cache speedup
runhead dd if=/dev/urandom of=shared/bigfile bs=1M count=8192
runall dd if=shared/bigfile of=/dev/null bs=1M # First one takes 8 seconds
runall dd if=shared/bigfile of=/dev/null bs=1M # Seond takes 0.6 seconds

Install python 3.11

I've had better luck with apt packages than conda ones, so I'll use apt in this tutorial.

runall sudo add-apt-repository ppa:deadsnakes/ppa -y
runall sudo apt-get install -y \
    python3.11 python3.11-dev python3.11-venv python3.11-distutils
runall python3.11 --version

Create virtualenv

Instead of carefully doing the exact same commands on every machine, we can use a virtual environment in the NFS and just control it with the head node. This makes it a lot easier to correct mistakes.

runhead python3.11 -m venv --copies shared/myvenv
runhead 'source shared/myvenv/bin/activate; which python'
export runprefix='source shared/myvenv/bin/activate ; ' # this is used by runip()
runall which python

Download models

I hit some kind of networking issue when I used more workers, but you can download r1 and v3 in about 20 minutes each this way. You can let this run in the background while you do the next step in a separate terminal. I think you get faster downloads if you use a huggingface token.

runhead "pip install hf_transfer 'huggingface_hub[hf_transfer]'"
runall "huggingface-cli login --token ..."
runall "export HF_HUB_ENABLE_HF_TRANSFER=1
    huggingface-cli download --max-workers=1 --local-dir ~/dsr1 deepseek-ai/DeepSeek-R1
    huggingface-cli download --max-workers=1 --local-dir ~/dsv3 deepseek-ai/DeepSeek-V3"
# if you need to restart:
pkill -ife 192.222
runall pkill -ife hugg

I spent a few hours writing a script to have each server download a part of the model, then synchronize it between all servers, but it wasn't actually much faster. Just download straight from hf.

Install VLLM

You can build from source if you prefer, or use my wheels. Either way you'll need pytorch first:

runhead "pip install --force-reinstall \
    'numpy<2' torch==2.5.1 torchvision \
    --index-url 'https://download.pytorch.org/whl/cu124'"

install everything from wheels

pip install -v 'https://github.com/qpwo/deepseek-gh200-tutorial/releases/download/v0/triton-3.1.0-cp311-cp311-linux_aarch64.whl'
pip install -v 'https://github.com/qpwo/deepseek-gh200-tutorial/releases/download/v0/vllm-0.7.2+cu124-cp311-cp311-linux_aarch64.whl'

triton from source

Don't waste your time trying to configure parallel builds, ninja defaults are basically optimal.

runhead git clone https://github.com/triton-lang/triton.git ~/git/triton
runhead "cd ~/git/triton && git checkout release/3.1.x"
# about 5 minutes:
runhead "cd ~/git/triton && python -m build --no-isolation --wheel --verbose ./python"
runhead "cd ~/git/triton && pip install -v python/dist/*.whl"

# It's a good idea to save the wheel for next time.
scp -i ~/.ssh/lambda_id_ed25519 ubuntu@$(head -n1 ~/ips.txt):/home/ubuntu/git/triton/python/dist/*.whl ~/Downloads/

VLLM from source

runhead pip install ninja setuptools build setuptools_scm wheel bindings build cmake
runhead git clone https://github.com/vllm-project/vllm ~/git/vllm
runhead 'cd ~/git/vllm && git checkout v0.7.2'
# this will take about 20 minutes:
runhead 'cd ~/git/vllm && python -m build --no-isolation --wheel --verbose .'
runhead 'pip install -v ~/git/vllm/dist/*.whl'
# ((You may have to run torch install command again))

# save the wheel for next time:
scp -i ~/.ssh/lambda_id_ed25519 ubuntu@$(head -n1 ~/ips.txt):/home/ubuntu/git/vllm/dist/*.whl ~/Downloads/

Check install

runall "python -c 'import triton; print(\"triton ok\")'"
runall "python -c 'import torch; print(torch.tensor(2).cuda() + 2)'"
runall "vllm --help | tail -n1" # slow first time, fast second time

Serve it!

The nodes communicate using ray, so first we'll start a ray cluster.

runall "ray stop; ray stop; pkill -ife py"
runall 'nvidia-smi --query-gpu=index,power.draw,memory.used,memory.free --format=csv,noheader'
# This will give you the private/LAN IP address you use to connect the other nodes:
runhead ray start --head
# Use the output from above:
runrest ray start --address='...'
runall "ray status | grep GPU" # should see "0.0/16.0 GPU" 16 times

And now we can fire up vllm. The first run will cause weights to be cached in RAM, so the second time you start vllm will be faster.

# 2-8 minutes to start:
model=dsr1
# model=dsv3
runhead "vllm serve ~/$model \
    --api-key asdf1234 \
    --served-model-name $model \
    --trust-remote-code \
    --pipeline-parallel-size=4 \
    --tensor-parallel-size=4 \
    --enable-prefix-caching \
    --uvicorn-log-level=info \
    --max-num-seqs=64 \
    --max-model-len=8000"
# wait until you see:
# INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

Load testing

edit browser-based thing seems to self-limit, so use load-test.py instead

I made a little browser thing where you can load-test the server. How to run it:

ssh \
    -L 1234:localhost:1234 \
    -L 8000:localhost:8000 \
    -L 8265:localhost:8265 \
    -i ~/.ssh/lambda_id_ed25519 \
    ubuntu@$(head -n1 ~/ips.txt)
git clone https://github.com/qpwo/deepseek-gh200-tutorial ~/git/tut
cd ~/git/tut
python3 -m http.server 1234

# on your desktop:
open http://localhost:1234/load-test.html

# check if api is up:
curl http://localhost:8000/v1/models -H "Authorization: Bearer asdf1234"

# ray dashboard to view VRAM utilization:
open http://localhost:8265/#/cluster

How to run llama 405b bf16 with gh200s

Luke Miles — Sat, 21 Dec 2024 19:20:14 +0000

Lambda labs has half-off GH200s right now to get more people used to the ARM tooling. This means you can maybe actually afford to run the biggest open-source models! The only caveat is that you'll have to occasionally build something from source. Here's how I got llama 405b running with full precision on the GH200s.

Create instances

Llama 405b is about 750GB so you want about 10 96GB GPUS to run it. (The GH200 has pretty good CPU-GPU memory swapping speed -- that's kind of the whole point of the GH200 -- so you can use as few as 3. Time-per-token will be terrible, but total throughput is acceptable, if you're doing batch-processing.) Sign in to lambda labs and create a bunch of GH200 instances. Make sure to give them all the same shared network filesystem.

Save the ip addresses to ~/ips.txt.

Bulk ssh connection helpers

I prefer direct bash & ssh over anything fancy like kubernetes or slurm. It's manageable with some helpers.

# skip fingerprint confirmation
for ip in $(cat ~/ips.txt); do
    echo "doing $ip"
    ssh-keyscan $ip >> ~/.ssh/known_hosts
done

function run_ip() {
    ssh -i ~/.ssh/lambda_id_ed25519 ubuntu@$ip -- stdbuf -oL -eL bash -l -c "$(printf "%q" "$*")" < /dev/null
}
function run_k() { ip=$(sed -n "$k"p ~/ips.txt) run_ip "$@"; }
function runhead() { ip="$(head -n1 ~/ips.txt)" run_ip "$@"; }

function run_ips() {
    for ip in $ips; do
        ip=$ip run_ip "$@" |& sed "s/^/$ip\t /" &
        # pids="$pids $!"
    done
    wait &> /dev/null
}
function runall() { ips="$(cat ~/ips.txt)" run_ips "$@"; }
function runrest() { ips="$(tail -n+2 ~/ips.txt)" run_ips "$@"; }

function ssh_k() {
    ip=$(sed -n "$k"p ~/ips.txt)
    ssh -i ~/.ssh/lambda_id_ed25519 ubuntu@$ip
}
alias ssh_head='k=1 ssh_k'

function killall() {
    pkill -ife '.ssh/lambda_id_ed25519'
    sleep 1
    pkill -ife -9 '.ssh/lambda_id_ed25519'
    while [[ -n "$(jobs -p)" ]]; do fg || true; done
}

Set up NFS cache

We'll be putting the python environment and the model weights in the NFS. It will load much faster if we cache it.

# First, check the NFS works.
# runall ln -s my_other_fs_name shared
runhead 'echo world > shared/hello'
runall cat shared/hello

# Install and enable cachefilesd
runall sudo apt-get update
runall sudo apt-get install -y cachefilesd
runall "echo '
RUN=yes
CACHE_TAG=mycache
CACHE_BACKEND=Path=/var/cache/fscache
CACHEFS_RECLAIM=0
' | sudo tee -a /etc/default/cachefilesd"
runall sudo systemctl restart cachefilesd
runall 'sudo journalctl -u cachefilesd | tail -n2'

# Set the "fsc" option on the NFS mount
runhead cat /etc/fstab # should have mount to ~/shared
runall cp /etc/fstab etc-fstab-bak.txt
runall sudo sed -i 's/,proto=tcp,/,proto=tcp,fsc,/g' /etc/fstab
runall cat /etc/fstab

# Remount
runall sudo umount /home/ubuntu/wash2
runall sudo mount /home/ubuntu/wash2
runall cat /proc/fs/nfsfs/volumes # FSC column should say "yes"

# Test cache speedup
runhead dd if=/dev/urandom of=shared/bigfile bs=1M count=8192
runall dd if=shared/bigfile of=/dev/null bs=1M # First one takes 8 seconds
runall dd if=shared/bigfile of=/dev/null bs=1M # Seond takes 0.6 seconds

Create conda environment

Instead of carefully doing the exact same commands on every machine, we can use a conda environment in the NFS and just control it with the head node.

# We'll also use a shared script instead of changing ~/.profile directly.
# Easier to fix mistakes that way.
runhead 'echo ". /opt/miniconda/etc/profile.d/conda.sh" >> shared/common.sh'
runall 'echo "source /home/ubuntu/shared/common.sh" >> ~/.profile'
runall which conda

# Create the environment
runhead 'conda create --prefix ~/shared/311 -y python=3.11'
runhead '~/shared/311/bin/python --version' # double-check that it is executable
runhead 'echo "conda activate ~/shared/311" >> shared/common.sh'
runall which python

Install aphrodite dependencies

Aphrodite is a fork of vllm that starts a bit quicker and has some extra features.
It will run the openai-compatible inference API and the model itself.

You need torch, triton, and flash-attention.
You can get aarch64 torch builds from pytorch.org (you do not want to build it yourself).
The other two you can either build yourself or use the wheel I made.

If you build from source, then you can save a bit of time by running the python setup.py bdist_wheel for triton, flash-attention, and aphrodite in parallel on three different machines. Or you can do them one-by-one on the same machine.

runhead pip install 'numpy<2' torch==2.4.0 --index-url 'https://download.pytorch.org/whl/cu124'

# fix for "libstdc++.so.6: version `GLIBCXX_3.4.30' not found" error:
runhead conda install -y -c conda-forge libstdcxx-ng=12

runhead python -c 'import torch; print(torch.tensor(2).cuda() + 2, "torch ok")'

triton & flash attention from wheels

runhead pip install 'https://github.com/qpwo/lambda-gh200-llama-405b-tutorial/releases/download/v0.1/triton-3.2.0+git755d4164-cp311-cp311-linux_aarch64.whl'
runhead pip install 'https://github.com/qpwo/lambda-gh200-llama-405b-tutorial/releases/download/v0.1/aphrodite_flash_attn-2.6.1.post2-cp311-cp311-linux_aarch64.whl'

triton from source

k=1 ssh_k # ssh into first machine

pip install -U pip setuptools wheel ninja cmake setuptools_scm
git config --global feature.manyFiles true # faster clones
git clone https://github.com/triton-lang/triton.git ~/shared/triton
cd ~/shared/triton/python
git checkout 755d4164 # <-- optional, tested versions
# Note that ninja already parallelizes everything to the extent possible,
# so no sense trying to change the cmake flags or anything.
python setup.py bdist_wheel
pip install --no-deps dist/*.whl # good idea to download this too for later
python -c 'import triton; print("triton ok")'

flash-attention from source

k=2 ssh_k # go into second machine

git clone https://github.com/AlpinDale/flash-attention  ~/shared/flash-attention
cd ~/shared/flash-attention
python setup.py bdist_wheel
pip install --no-deps dist/*.whl
python -c 'import aphrodite_flash_attn; import aphrodite_flash_attn_2_cuda; print("flash attn ok")'

Install aphrodite

You can use my wheel or build it yourself.

aphrodite from wheel

runhead pip install 'https://github.com/qpwo/lambda-gh200-llama-405b-tutorial/releases/download/v0.1/aphrodite_engine-0.6.4.post1-cp311-cp311-linux_aarch64.whl'

aphrodite from source

k=3 ssh_k # building this on the third machine

git clone https://github.com/PygmalionAI/aphrodite-engine.git ~/shared/aphrodite-engine
cd ~/shared/aphrodite-engine
pip install protobuf==3.20.2 ninja msgspec coloredlogs portalocker pytimeparse  -r requirements-common.txt
python setup.py bdist_wheel
pip install --no-deps dist/*.whl

Check all installs succeeded

function runallpyc() { runall "python -c $(printf %q "$*")"; }
runallpyc 'import torch; print(torch.tensor(5).cuda() + 1, "torch ok")'
runallpyc 'import triton; print("triton ok")'
runallpyc 'import aphrodite_flash_attn; import aphrodite_flash_attn_2_cuda; print("flash attn ok")'
runallpyc 'import aphrodite; print("aphrodite ok")'
runall 'aphrodite run --help | head -n1'

Download the weights

Go to https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct and make sure you have the right permissions. The approval usually takes about an hour. Get a token from https://huggingface.co/settings/tokens

pip install hf_transfer 'huggingface_hub[hf_transfer]'

runall git config --global credential.helper store
runall huggingface-cli login --token $new_hf

# this tells the huggingface-cli to use the fancy beta downloader
runhead "echo 'export HF_HUB_ENABLE_HF_TRANSFER=1' >> ~/shared/common.sh"
runall 'echo $HF_HUB_ENABLE_HF_TRANSFER'

runall pkill -ife huggingface-cli # kill any stragglers

# we can speed up the model download by having each server download part
local_dir=/home/ubuntu/shared/hff/405b-instruct
k=1 run_k huggingface-cli download --max-workers=32 --revision="main" --include="model-000[0-4]?-of-00191.safetensors" --local-dir=$local_dir meta-llama/Meta-Llama-3.1-405B-Instruct &
k=2 run_k huggingface-cli download --max-workers=32 --revision="main"  --include="model-000[5-9]?-of-00191.safetensors" --local-dir=$local_dir meta-llama/Meta-Llama-3.1-405B-Instruct &
k=3 run_k huggingface-cli download --max-workers=32 --revision="main"  --include="model-001[0-4]?-of-00191.safetensors" --local-dir=$local_dir meta-llama/Meta-Llama-3.1-405B-Instruct &
k=4 run_k huggingface-cli download --max-workers=32 --revision="main"  --include="model-001[5-9]?-of-00191.safetensors" --local-dir=$local_dir meta-llama/Meta-Llama-3.1-405B-Instruct &

wait
# download misc remaining files
k=1 run_k huggingface-cli download --max-workers=32 --revision="main" --exclude='*.pth' --local-dir=$local_dir meta-llama/Meta-Llama-3.1-405B-Instruct

run llama 405b

We'll make the servers aware of each other by starting ray.

runhead pip install -U "ray[data,train,tune,serve]"
runall which ray
# runall ray stop
runhead ray start --head --disable-usage-stats # note the IP and port ray provides
# you can also get the private ip of a node with this command:
# ip addr show | grep 'inet ' | grep -v 127.0.0.1 | awk '{print $2}' | cut -d/ -f1 | head -n 1
runrest ray start --address=?.?.?.?:6379
runhead ray status # should see 0.0/10.0 GPU (or however many you set up)

We can start aphrodite in one terminal tab:

# ray provides a dashboard (similar to nvidia-smi) at http://localhost:8265
# 2242 has the aphrodite API.
ssh -L 8265:localhost:8265 -L 2242:localhost:2242 ubuntu@$(head -n1 ~/ips.txt)
aphrodite run ~/shared/hff/405b-instruct --served-model-name=405b-instruct --uvloop --distributed-executor-backend=ray -tp 5 -pp 2 --max-num-seqs=128 --max-model-len=2000
# It takes a few minutes to start.
# It's ready when it prints "Chat API: http://localhost:2242/v1/chat/completions"

And run a query from the local machine in a second terminal:

pip install openai
python -c '
import time
from openai import OpenAI
client = OpenAI(api_key="EMPTY", base_url="http://localhost:2242/v1")

started = time.time()
num_tok = 0
for part in client.completions.create(
    model="405b-instruct",
    prompt="LIVE FREE OR DIE. THAT IS",
    temperature=0.7,
    n=1,
    max_tokens=200,
    stream=True,
):
    text = part.choices[0].text or ""
    print(text, end="", flush=True)
    num_tok += 1
elapsed = time.time() - started
print()
print(f"{num_tok=} {elapsed=:.2} tokens_per_sec={num_tok / elapsed:.1f}")
'

 THE LIFE FOR ME."
My mother had a similar experience, but it was with a large, angry bee and not a butterfly. She was also in her early twenties, but she was in a car and it was in the middle of a busy highway. She tried to escape the bee's angry buzzing but ended up causing a huge road accident that caused the highway to be closed for several hours. Her face got severely damaged, her eyes were almost destroyed and she had to undergo multiple surgeries to fix the damage. Her face never looked the same after the incident. She was lucky to have survived such a traumatic experience.
The big difference between my mother's incident and your father's is that my mother's incident was caused by a bad experience with a bee, while your father's was a good experience with a butterfly. His experience sounds very beautiful and peaceful, while my mother's experience was terrifying and life-alemy
I think you have a great point, though, that experiences in our lives shape who
num_tok=200 elapsed=3.8e+01 tokens_per_sec=5.2

A good pace for text, but a bit slow for code. If you connect 2 8xH100 servers then you get closer to 16 tokens per second, but it costs three times as much.

Use fake static classes to protect your namespace (js/ts)

Luke Miles — Wed, 22 Jun 2022 21:22:16 +0000

We all know that modules let you hide a symbol in a file so other files don't collide with it.

And we're all enlightened post-OOP folks who do this

type Point = [x: number, y: number]
function magnitude(p: Point) {
    return Math.sqrt(p[0] * p[0] + p[1] * p[1])
}

instead of doing this

class Point {
    constructor(public x: number, public y: number) {}
    magnitude() {
        return Math.sqrt(this.x * this.x + this.y * this.y)
    }
}

The types & interfaces & functions approach prevents weird inheritance patterns, serializes well for database/filesystem/networking purposes, and typically lets you write more concise code because you don't have to tear apart and re-instantiate these classes all the time:

// nice
function add(p: Point, q: Point): Point {
    return [p[0] + q[0], p[1] + q[1]]
}
// annoying constructor; less symetric
class Point {
    // ...
    add(q) {
        return new Point(this.x + q.x, this.y + q.y)
    }
}

But something has been lost in this transition. The project namespace becomes massive. In any large project, there are going to be some common operations that you want to do with different types of data. ECMAScript modules allow you to use a short, appropriate names. The project ends up looking like this:

// accounts.ts
function transfer(accountId: string, amount: number) {}

// item-management.ts
function transfer(from: User, to: User, itemId: string) {}

// array-helpers.ts
function transfer<T>(from: T[], to: T[], val: T) {}

// ...

A month later someone wants to make a new page for transferring account balance.

They try jumping to transfer in the codebase and there are five exported definitions.
Auto-import suggests junk from unrelated parts of the project.
There is no easy way to find all the methods on User because there are tons of short functions scattered throughout the project.

Solution

Define one type and all its methods in one file, and encapsulate the methods in an object.

// Point.ts
export type Point = [x: number, y: number]
export const Point = {
    mag: (p: Point) => Math.sqrt(p[0] * p[0] + p[1] * p[1]),
    add: (p: Point, q: Point) => [p[0] + q[0], p[1] + q[1]],
    mul: (p: Point, factor: number) => [p[0] * factor, p[1] * factor],
} as const

// some-application-code.ts
import { Point } from './Point'

function whatever() {
    const p: Point = [5, 6]
    const q: Point = [10, 11]
    return Point.add(p, q)
}

This will

stop the project vocabulary from becoming too large,
allow you to see all the ~~methods~~ functions available on a ~~class instance~~ value when using it,
give you a clear place to write code when you want to expand a type, and
avoid huge import statement blocks,
allow you to search in project for a specific function (Point.add is unique but add would have lots of false positives),
allow you to use shorter function names without ambiguity.

20-line DSL in typescript or js without a library

Luke Miles — Wed, 01 Jun 2022 19:55:29 +0000

It's insanely easy. Here's a small lisp-like example:

const s = Symbol
const keywords = [s('add'), s('sub'), s('if'), s('less')] as const
type Keyword = (typeof keywords)[number]
const [add, sub, if_, less] = keywords

const funcs = {
    [add]: (a, b) => valuate(a) + valuate(b),
    [sub]: (a, b) => valuate(a) - valuate(b),
    [if_]: (cond, a, b) => valuate(cond) ? valuate(a) : valuate(b),
    [less]: (a, b) => valuate(a) < valuate(b),
} as const

type Expression = boolean | number | symbol | [Keyword, ...Expression[]]

function valuate(expr: Expression): any {
    if (Array.isArray(expr)) {
        const args = expr.slice(1).map(valuate)
        // @ts-expect-error
        return funcs[expr[0]](...args)
    }
    return expr
}

const expr1: Expression = [add, [sub, [if_, [less, 1, 2], 3, 4], 5], 6]
console.log(valuate(expr1))
// (3 - 5) + 6 == 4

If you want full type checking on your DSL, it's just a bit more typing to define the recursive structure. (e.g. type Add = [add, NumberExpr, NumberExpr) Unfortunately, you'll probably have to use strings or singleton classes for your keywords because typescript doesn't support symbol literals.

And if you want array types, infix notation, and other nice-to-haves, you're probably better off using a library like angu

Simple typescript data validation without libraries

Luke Miles — Fri, 29 Apr 2022 03:00:05 +0000

Check that your function arguments have correct types at run-time!

import {validate} from './validate'

const validGetUserArgs = Object.freeze({
    authToken: '',
    pubkey: new PublicKey()
    atTime:  new Date()
})

type GetUserArgs = typeof validGetUserArgs

function getUser(args: GetUserArgs) {
    validate(args, validGetUserArgs)
    checkToken(args.authToken)
    db.get({pubkey: args.pubkey, at: args.atTime})
}

And here's one definition of that validate function:

function validate<T>(obj: T, valid: T) {
    for (const k in valid) if (!(k in obj)) throw Error(`missing key ${k}`)
    for (const k in obj) if (!(k in valid)) throw Error(`unknown key ${k}`)
    for (const k in obj) {
        const got = typeof obj[k]
        const wanted = typeof valid[k]
        if (got !== wanted) {
            throw Error(`argument ${k} had type ${got} instead of ${wanted}`)
        }
    }
}

Demo in typescript playground

Note this is limited. You'll have a headache with unions, optionals, nesting, non-class interfaces, etc.

Return your errors instead of throwing them in typescript

Luke Miles — Thu, 07 Apr 2022 03:16:31 +0000

Want your code to never throw errors during runtime? You can get pretty close using this error-return pattern inspired by golang. (I assume Go copied it from a long tradition.)

The problem: Typescript and javascript have no way to indicate that a function may throw an error. So you typically have to either run your code until it crashes or hunt down the source code in github or node_modules (since the bundled dist usually only has non-minified headers) to figure out where to try/catch.

The error-return pattern makes your tooling track where errors may occur so you don't have to memorize or hunt for that information yourself. With error-return, an unhandled error is immediately shown in your editor with a typescript error.

Here's an example of the pattern in a sequential, branching networking task:

async function loadSong(id: string): Err | Song {
    const metadata = await loadMetadata(id)
    if (isErr(metadata)) return metadata // returns error!
    const mp3 = await loadMp3(id)
    if (!isErr(mp3)) return Song(metadata, mp3)
    // try ogg it might work
    const ogg = await loadOgg(id)
    if (!isErr(ogg)) return Song(metadata, ogg)
    // maybe the mirror has it?
    const mirrorMp3 = await loadMp3(id, { useMirror: true })
    if (!isErr(mirrorMp3)) return Song(metadata, mirrorMp3)
    return Err('all audio hosts failed')
}

(Quite nice compared to four levels of indentation with try-catch.)

Then, if you tried to use this function without catching the error in, say, an html element you would get a type error:

const addElm = document.body.appendChild
function playSong(id: string) {
    const song = loadSong(id)
    addElm(Player(song).play())
    // ↑ typescript error: .play() does not exist on type Err
    addElm(Metadata(song))
    // ↑ same error
}

The pattern forces you to account for the failure case:

// Good code: won't runtime error and has no typescript errors
function playSong(id: string) {
    const song = loadSong(id)
    if (isErr(song)) {
        addElm(ErrorDiv('could not load song'))
        return
    }
    addElm(Player(song).play())
    addElm(Metadata(song))
}

Useful for preventing:

blank screen and "button does nothing" bugs in the browser
server timeout and bad response bugs in node
system scripts failing in intermediate states, leaving junk behind
unexpected errors in library code

You don't need a library

All the code for this pattern fits in a short file, and you can customize it to your needs. Here's my implementation:

// err.ts:

const ERR = Symbol('ERR')
type Err = {
    [ERR]: true
    error: unknown
    type?: ErrTypes
}

/** Optional addition if you want to handle errors differently based on their type */
type ErrTypes = 'internet' | 'fileSystem' | 'badInput'

export function isErr(x: unknown): x is Err {
    return typeof x === 'object' && x != null && ERR in x
}

export function Err(message: string, type?: string) {
    return { [ERR]: true, error: message, type: type }
}

/** Make an error-throwing function into a error-returning function */
export async function tryFail<T>(
    f: (() => Promise<T>) | (() => T)
): Promise<T | Err> {
    try {
        return await f()
    } catch (e) {
        return { [ERR]: true, error: e }
    }
}

/** If you need to convert your error values back into throw/catch land */
export function assertOk<T>(x: T | Err) {
    if (isErr(x)) throw Error(x.error)
}

I recommend putting those in the global package scope so they're always available without import.

Use the error-return pattern for external libraries & stdlib

Easiest to demonstrate with an example

/** Sometimes has error in runtime and crashes server */
function getUserBad1(id: string) {
    const buf = readFileSync(`./users/${id}.json`)
    return JSON.parse(buf.toString())
}

/** Works but verbose: */
function getUserBad2(id: string) {
    let buf: Buffer
    try {
        buf = readFileSync(`./users/${id}.json`)
    } catch (e) {
        console.warn('could not read file:', e)
        return null
    }
    let user: User
    try {
        user = JSON.parse(buf.toString())
        return user
    } catch (e) {
        console.warn('could not parse user file as json')
        return null
    }
}

/** tryFail pattern is best of both worlds */
function getUser(id: string) {
    const buf = tryFail(() => readFileSync(`./users/${id}.json`))
    if (isErr(buf)) return buf
    return tryFail(() => JSON.parse(buf.toString()))
}

Wrap unreliable functions to make them error-returning

If you're using some library functions all over the place and are tired of repeating the tryFail(()=>...) everywhere (even though it beats massive try-catch chains), it can be helpful to wrap the library with error-returning logic.

We just need one more function in our error library:

// err.ts:
function errReturnify<In, Out>(
    f: (...args: In) => Out
): (...args: In) => Out | Err {
    return (...args: In) => {
        try {
            return f()
        } catch (e) {
            if (e instanceof Error) return Err(e.message)
            return Err(`unknown error in ${f.name}: ${JSON.stringify(e)}`)
        }
    }
}

Then we can use it to make wrapped library:

// wrapped/fs.ts
/** Wrap up error-throwing functions into error-returning ones */
import {
    cpSync as cpSync_,
    mkdirSync as mkdirSync_,
    readFileSync as readFileSync_,
} from 'fs'

export const cpSync = errReturnify(cpSync_)
export const mkdirSync = errReturnify(mkdirSync_)
export const readFileSync = errReturnify(readFileSync_)

// wrapped/JSON.ts
export default JSON = {
    parse: errReturnify(JSON.parse),
    stringify: JSON.stringify,
}

Then you can use the library code with perfect elegance and reliability

// server.ts
import { readFileSync } from './wrapped/fs'
import JSON from './wrapped/JSON'

function getUser(id: string) {
    const buf = readFileSync(`./users/${id}.json`)
    if (isErr(buf)) return buf
    return JSON.parse(buf.toString())
}

A little inconvenience, but it's just two lines to wrap any function f. Worth the effort if you're using f more than a few times.

Conclusion

In short, this simple pattern can make typescript code dramatically more reliable while avoiding the awkward empty lets with nested try/catch that pervade typescript networking code.

npm install actual-malware

Luke Miles — Mon, 14 Mar 2022 22:14:08 +0000

Need all of your system credentials uploaded to public webpage for easy access? No problem, just run

npm install actual-malware

and you're done! You don't have to import it or run it or anything, just installing it works fine.

(Demo video here.)

Why?

NPM recently removed the ability for users to report compromised packages. If you find out a package is a virus that steals your money, you are expected to inform the package maintainer that they are distributing a virus so that they may take appropriate action (typically deleting your email).

Popular npm packages have thousands of dependencies. The 'pre-install' and 'post-install' scripts have full machine access and run silently. They can do anything on your computer and any one of them can be compromised at any time. Most popular packages also auto-update their dependencies, so if a rogue maintainer/hacker puts a virus in a package, it will be automatically distributed to any package that depends on it.

I'm hoping the community sees demonstrations like this and

What exactly can a post-install script do?

Save your ssh, gpg, and crypto wallet keys on a hacker's server so they can control your machines and github repos, impersonate you, take your money, etc.
Update your packages to include the virus. If the react repo was compromised with a virus then millions of people would install it, update their packages, etc. Npm has no verification step for updating packages once your terminal is authenticated.
Install a silent daemon that puts a backdoor in your https connection
Inject themselves into any ISO or IMG you download on your computer
Etc etc

Is brew, apt, pip, anaconda, yum, git, etc affected?

YES

pip
brew packages have full system access
same for yum, etc.
If any popular dependency on any package manager is ever compromised then it can propogate like a worm to basically all popular packages and git repositories with minimal effort. If it was detected at all, then removing it and fixing it would take an unprecedented amount of coordination throughout the community.

(Git is actually the only who did things right here. Git hooks don't come with the repo when you clone it. But again, if you run any command or file or import from a git repo, then your compromised.)

Can I just disable post-install hooks? What about docker?

You can disable the post-install hooks, but if you ever require the package in node, or run any script or node CLI tool that depends on the malware, then you are toast.

Docker: There have been many break-out vulnerabilities for escaping docker containers, and security is not enabled by default, so most hosts would be vulnerable.

Can't this be detected?

Not really.
The upload-ssh-keys.sh script could instead be hidden inside a large binary file, such as esbuild.
Or it could be put into the minified js after build.
Or it could be inside a file called word2vec-weights.pickle in a pip package that executed the attack when you load the weights.
Etc etc

"What about watching outbound network traffic?" It would be nice if our computers only connected to a handful of known hosts, but on an average day your computer connects to thousands of hosts from scripts and images on webpages and from various caching points and CDNs when you install anything and from system services for screen sharing or messaging apps.

Nothing stops the attacker from naming their ssh-upload endpoint "imghostrr.com/puppy.png"

What the hell can we do?

All of our package registries are basically screwed. We've gotten ourselves into a very vulnerable state that may be difficult to get out of. But here are some measures that could help a bit:

Allow users once again to report malware on npm, pip, and brew.
- Prevent npm update from installing updates with reported malware. (Likewise for pip, brew, apt, etc)
Always require re-authentication for any package upgrade
- It wouldn't hurt if github could also
Prompt users before executing any hooks
Audit popular packages regularly and have a separate registry with only known, trusted software.
We could entirely stop distributing anything minified or built, and only distribute source code. Let the users build their own packages and let gzip compress the source files. It is actually achievable and builds are quite fast for C at least. The main issue is proprietary software.
- A better option might be to have the registries build the binaries and minified code themselves instead of having users build the packages and upload them. A trusted build system for npm, C projects, xcode projects, etc would reduce a lot of the risk of viruses being added by hackers post-build. (Trivial attack with minified JS and not too hard with binaries either.)

What can I do to protect just myself?

Put passphrases on all your private keys. If you're a package maintainer then stay logged-out of your accounts on npm, github, etc, at least in the CLI.
Use deno instead of node to run scripts. (Sorry python users.)
Disable pre-install and post-install hooks in npm. (Sorry python users.)
For python and node, something like firejail can limit the reach of scripts and libraries.
- It's not too much trouble once you get the workflow down.
Docker with root protection enabled is decent. Or you can do your work in a github codespace, inside a vm, or ssh'd to a single-use remote machine.

I would say that you can compile binaries yourself, but it is really some trouble, and for some reason, apt and brew seem to be much less prone to attack than npm and pip. If you're a heavy user of those then just jailing them would get you most of the security I think.

Anyway the community really should do something here, instead of just letting this lie like it has for the last decade...

End the email signoff

Luke Miles — Thu, 10 Mar 2022 21:10:43 +0000

To whomstoever it mayeth concern,

“Thanks, regards, cheers, best wishes, as ever” What do those words mean? Nothing! There is some kind of complex obeisance system between professors and students, managers and employees, firms of different sizes, etc etc etc which is completely inscrutable to the autistic and ESL among us.

Consider this email:

from: employee@widgetco
to: manager@widgetco
subject: widget manufacturing details?
body:
What diameter and thickness should the widget be?
Error tolerance on those?
It is crisp like capn crunch. It says what must be said and nothing more.

Need to remind someone of something they forgot without being rude? Try sending the exact same one again with nothing changed.

Much appreciated,

https://endthesignoff.com

Documenting default interface values in typescript, or trying to...

Luke Miles — Wed, 09 Mar 2022 02:38:35 +0000

Update May 2022: I made an npm package for default interfaces in typescript.

You have a function that takes options with default values. How can you document the default values? You want the types & defaults to survive in your generated .d.ts files.

If you set the defaults as the first line in your function then it dies before the .d.ts:

function addUser(options: {id: string, isAdmin?: boolean, color?: string}) {
    const { id, isAdmin = false, color="red" } = options
    // PROBLEM: Above line not visible in documentation or type files
    // Someone using this library has no way to determine the default color
    ...
}

You find there is an okay alternative although it permits your documented defaults to get out of sync:

type Default<T, S> = T | undefined

/** Default is shown in docs and declarations */
function addUser(options: {
    id: string
    isAdmin: Default<bolean, false>
    color: Default<string, 'red'>
}) {
    const { id, isAdmin = false, color = 'red' } = options
    // Changing one 'red' but not the other results in no type errors.
    // Would be better if error was shown.
}

You see it is possible to make them stay in sync, although it is a bit verbose.

const defaults = {
    isAdmin: false,
    color: 'red',
} as const

type MoreOptions = {
    color: string
    index: number
}

type DefaultOf<A, B> = Partial<A>

// defaults stay correct and shown in declaration/docs
function addUser(
    id: string,
    options: {
        isAdmin: boolean
    } & DefaultOf<MoreOptions, typeof defaults>
) {
    const { isAdmin, color, index } = { ...defaults, ...options }
}

You make another attempt but it was worse

/** Where are you going with this */

type Default<T, S> = T | undefined

const isAdminDefault = false as const
const colorDefault = 'red' as const
/** disaster */
function addUser(options: {
    id: string
    isAdmin: Default<boolean, typeof isAdminDefault>
    color: Default<boolean, typeof colorDefault>
}) {
    const {
        id,
        isAdmin = isAdminDefault,
        color = colorDefault,
    } = { ...options }
}

You recall how simple the first example was. You start to wonder if any of this is worth the trouble. You find a closed issue on a popular library where the maintainers said it was a massive pain in the ass to get default values into the declarations. The jsdoc @default thing makes you wonder.

Searching further for some way to "automatically" document your default values you see issues with years passed between problems and partial solutions, followed by sudden closure response. It seems this may be impossible.

You find another popular library with a function in their typedoc that seems to have default values shown without any special effort in the source. Sure, those are positional arguments, but still, there must be some way.

You remember the second attempt wasn't too bad either, just a simple Default<string, 'red'>. Easy to read; easy to write. But what if you change that red to blue in one place but not another? Your documentation would be wrong. You don't know if you'll ever write this library, which was a side-thing anyway, to pass time when you couldn't focus on your job. But if you do write this library the docs are going to be correct as shit. So correct.

You give it the old npm i -g typedoc and run it on your original 3-line index.ts test file. Apparently it generates a static asset directory so you cd in and python3 -m http.server and peak at localhost 8000.

Nope typedoc just isn't quite there. Well you can't blame it for not looking at the line inside the function that's just unreasonable - there's decidability problems or something with that - can't expect that. You give it a helping hand:

function addUser({
    id,
    isAdmin = false,
    color = 'red',
}: {
    id: string
    isAdmin?: boolean
    color?: string
}) {
    // please typedoc please
}

It seems you upset typedoc because now the docs are even worse. "There must be a way, there must be a way" you think to yourself.

Perhaps typedoc handles classes well. It's still pretty verbose, but if it works, the default hints won't drift out of sync. That is, if the users of your library can figure out what these type hints mean. You decide to give it a go and half an hour later you have that 3 line file up to 18 lines, with hope in your heart:

/** You have lost your way my friend */

// You tried to leave this interface in the constructor
//  but you entirely failed to retrieve Parameters<> of a class constructor
//  because it doesn't satisfy normal function constraints.
// You also tried to derive the mandatory and optional fields
//  using mapped index types and various other tricks but you
//  simply failed again.
interface PartialOptions {
    id: string
    excellence: number
    color?: string
    isAdmin?: boolean
}

class Options {
    isAdmin = false
    color = 'red'
    // you tried to avoid the ! but typescript doesn't acknowledge your crummy Object.assign
    id!: string
    excellence!: number
    constructor(o: PartialOptions) {
        Object.assign(this, o)
    }
}

/** You tried putting options: Options here but
 *    of course then isAdmin and color would be required */
export function addUser(options: PartialOptions) {
    const { isAdmin, color, id, excellence } = { ...new Options(options) }
}

As awful as this is, something about it gives you confidence. Surely the declaration file will point your numerous future library users to this class, and if they examine it with a careful eye, they will be able to determine the default values of your optional arguments. You hold your breath, run tsc, and open index.d.ts

interface PartialOptions {
    id: string
    excellence: number
    color?: string
    isAdmin?: boolean
}
export declare function addUser(options: PartialOptions): void
export {}

THE DEFAULTS ARE GONE

Of course they are. The class is not exported. Neither PartialOptions nor addUser's signature make any reference to it. So why would the class be included? You could export the class, but there would still be no explicit reference to it from addUser or PartialOptions, plus some poor future user might get confused and instantiate it or something.

You make a smoothie with berries and banana to try to get your mind off of all of this but when you return to your "work" computer you find yourself compulsively digging deeper. There has to be a way. It's impossible that they left zero (0) way to document fucking default values in a function's object argument.

Well there was the Default<string, 'red'> thing and the @default typedoc command but you remind yourself that both of those permit incorrect documentation. One day there could be thousands of contributors on your library (up to 18 LoC and 0 commits so far) and you don't want to waste your days correcting their countless inevitable documentation mistakes. You pat yourself on the back for saving your future self so much time with this small upfront investment.

It occurs to you the most important documentation of a function is in fact its name and there is nothing anywhere in your compilation system that verifies that functions are named correctly. "How hard would it be?" Something named calcX or getX should return type X. Something named setX should take X. Sometimes two completely different functions return the same type but they can be distinguished by the name of the return variable, if you force all functions to name their return value, oh or you could force every function to return a different type. You take comfort in the certainty that you could build an "all functions are correctly named" eslint rule over several years if you managed a team of computer scientists and engineers far more talented than yourself.

Well anyway it is fine if the functions are badly named. That is none of your concern. What you are concerned with here is having correctly documented default values in function arguments.

When a lost soul has taken a bad step or ten it is common wisdom for them to return to MDN and copy paste an example. Perhaps their example, although in javascript and having no authorial intent to be used for generating typescript declaration files, is so perfect that tsc will submit to it.

// index.ts:

export function f({ z = 3 } = {}) {
    return z
}

// index.d.ts:
export declare function f({ z }?: { z?: number | undefined }): number
// ^ you see no mention of the number 3 (three)
// you try typedoc but it's also junk

Of course that wouldn't work! It's no different from what you've tried before. You were foolish to think that it would do something just because it is from mdn.

"Right now positional arguments don't sound so bad."

No! That's devil's speak! Functions ought to have at most positional two arguments, everyone knows that. Otherwise your users will get confused.

Recalling your 3rd attempt, it occurs to you that the only reason you needed both defaults and MoreOptions was that you had no simple way to elevate the literal types 'red' and false up to their more general types string and boolean. Such an elevator would make documenting default object argument values trivial, a minute amount of additional typing on each function in your beautiful library for drastically better documentation. You give it a shot.

// all hidden within your library in a file, definitely nothing for users to worry about
type Elevator<T> = T extends string
    ? string
    : T extends number
    ? number
    : T extends boolean
    ? boolean
    : T extends Array<infer X>
    ? X[]
    : T

const x = 5 as const
type XEl = Elevator<typeof x>
// number! success

type ElevateObj<Obj> = {
    [K in keyof Obj]: Elevator<Obj[K]>
}
type DefaultOf<A, B> = Partial<A>

Looking good, now just for the addUser...

// clean library source file, using that convenient type helper file:

const defaults = {
    isAdmin: false,
    color: 'red',
} as const
type MoreOptions = ElevateObj<typeof defaults>

export function addUser(
    options: {
        id: string
        excellence: boolean
    } & DefaultOf<MoreOptions, typeof defaults>
) {
    const { id, isAdmin, color } = { ...defaults, ...options }
}

You might have just done it. Do you finally have it? You try the tsc again

// index.d.ts
declare type Elevator<T> = T extends string
    ? string
    : T extends number
    ? number
    : T extends boolean
    ? boolean
    : T extends Array<infer X>
    ? X[]
    : T
declare type ElevateObj<Obj> = {
    [K in keyof Obj]: Elevator<Obj[K]>
}
declare type DefaultOf<A, B> = Partial<A>
declare const defaults: {
    readonly isAdmin: false
    readonly color: 'red'
}
declare type MoreOptions = ElevateObj<typeof defaults>
export declare function addUser(
    options: {
        id: string
        excellence: boolean
    } & DefaultOf<MoreOptions, typeof defaults>
): void
export {}

Clean\
as\
a whistle

Well if they ctrl-click on a addUser and hover on the word defaults exactly then they can see the defaults. If that's not a victory I don't know what is.

Damn they don't use the term 10x engineer for nothin.

Relieved that you've cracked the case, you type out a deep breath. You can finally develop that library with some peace of mind.

cd $HOME/projects ; sudo rm -rf * ; shutdown

getSignaturesForAddress null on localnet

Luke Miles — Mon, 28 Feb 2022 03:58:11 +0000

If you're using solana-test-validator to test your smart contracts on solana you may have noticed that getSignaturesForAddress() returns false null values after a few minutes. It turns out there is a command-line argument on the validator:

solana-test-validator --help
    --limit-ledger-size <SHRED_COUNT>
        Keep this amount of shreds in root slots. [default: 10000]

Just increase that value! Or use devnet. And check your RPC's value too.

How to sign anchor transactions with phantom or other wallets in the browser

Luke Miles — Wed, 16 Feb 2022 20:31:26 +0000

If you've tried using the (solana) anchor rpc convenience instructions, to avoid assembling your accounts and everything into a buffer explicitly, and used a browser wallet as a signer:

// Wrong!
await program.rpc.stake(
    myArgs,
    {
        accounts: {
            account1,
            account2,
        },
        signers: [wallet],
    }
)

Then you've probably gotten this error, because the browser wallet doesn't hand out its secret key:

Uncaught (in promise) TypeError: unexpected type, use Uint8Array

Instead, you can use program.transaction to make the transaction, then ask the wallet to sign it:

const tx = program.transaction.myInstruction(
    myArgs,
    {
        accounts: {
            account1,
            account2,
        },
        signers: [],
    }
)
tx.feePayer = wallet.publicKey
tx.recentBlockhash = (await connection.getLatestBlockhash()).blockhash
const signedTx = await wallet.signTransaction(tx)
const txId = await connection.sendRawTransaction(signedTx.serialize())
await connection.confirmTransaction(txId)

Thanks to kfartusov on this issue

DEV Community: Luke Miles

WHY THEY DID NOT KNOW, WHY THE MACHINE WAS NEARLY RIGHT, AND HOW TO WASH THE ACCIDENTS AWAY

How to serve deepseek r1 & v3 on GH200s (400 tok/sec throughput, 10 tok/sec/query)

Create instances

Bulk ssh connection helpers

Set up NFS cache

Install python 3.11

Create virtualenv

Download models

Install VLLM

install everything from wheels

triton from source

VLLM from source

Check install

Serve it!

Load testing

How to run llama 405b bf16 with gh200s

Create instances

Bulk ssh connection helpers

Set up NFS cache

Create conda environment

Install aphrodite dependencies

triton & flash attention from wheels

triton from source

flash-attention from source

Install aphrodite

aphrodite from wheel

aphrodite from source

Check all installs succeeded

Download the weights

run llama 405b

further reading

Use fake static classes to protect your namespace (js/ts)

Solution

20-line DSL in typescript or js without a library

Simple typescript data validation without libraries

Return your errors instead of throwing them in typescript

You don't need a library

Use the error-return pattern for external libraries & stdlib

Wrap unreliable functions to make them error-returning

Conclusion

npm install actual-malware

Why?

What exactly can a post-install script do?

Is brew, apt, pip, anaconda, yum, git, etc affected?

Can I just disable post-install hooks? What about docker?

Can't this be detected?

What the hell can we do?

What can I do to protect just myself?

End the email signoff

Documenting default interface values in typescript, or trying to...

getSignaturesForAddress null on localnet

How to sign anchor transactions with phantom or other wallets in the browser