DEV Community: Nick Lucas

LLMs do know what they're going to say

Nick Lucas — Sat, 27 Sep 2025 16:39:06 +0000

The following is written to be as penetrable as possible for a semi-technical audience, and includes an app to explore the output of an LLM. As a result some technical explanations are necessarily imprecise though I've worked hard to not say anything outright wrong. It also comes in two halves, with the latter introducing more technical details and more formal terminology.

"LLMs are probabilistic models which generate a completion to a prompt, they do this one token at a time and produce non-deterministic results."

This is a well known explanation of how LLMs work, but in much the same way as Schrödinger's cat explains quantum physics to a lay-person but gives rise to misconceptions about quantum physics, this explanation gives rise to a number of misconceptions about LLMs:

LLMs have no idea what they're going to say until they've said them
LLMs are just autocomplete for the next word

Statements to this effect are only myopically true when considering the generation process, but are repeated constantly on social media as if they represent the truth of the technology. In this post I am going to focus on why the more accurate statement would be:

LLMs know every response they could provide, along with a fixed probability for each option, and during generation we make them use a single path to a complete output

LLMs are just Maths

LLMs are input->output maths machines, they have internal parameters set by their training which don't change, and if you give the same input to an equation you will always get the same output. This means LLMs themselves are highly deterministic, it's actually the human-designed harnesses we use to produce output text (often known as Decoders or Samplers) which promote varied generation by choosing tokens which an LLM may not have reported as the highest probability.

What follows, is given a prompt (for instance "The capital of France") and generating the probabilities (formally: logits, but I'll cover that later) of possible next-tokens, and then the probabilities of all tokens in the next layer, and so on, you will get an ever-exploding tree of generation paths with consistent probabilities for each future token. Additionally, if we take that prompt and suffix some generated tokens (for instance "The capital of France is Paris") and do the same thing, then we'll also get the exact same subtree of probabilities. The LLM knows statically everything it wants to say, and the harness we wrap an LLM with gets final say in the path we take through this tree structure.

When I describe a tree structure of tokens, this is a slice of what I mean:

Or maybe this will help:

A demonstration

To demonstrate this, I've built a custom sampler which generates the tree structure of future tokens+probabilities instead of selecting a single path through the tree. You can explore a number of generations I've prepared here:

https://llm-probability-tree.me-62f.workers.dev/

The pre-generated outputs are using Google's Gemma 3 1b, a small LLM which is possible to run quickly on consumer hardware (In my case a Macbook M2 Pro) and the pre-generated data you see expands the 5 highest probability tokens at each step up to 6 tokens into the future. I've also made sure to tune the LLM settings to be constant for each call, disabling anything which could inject non-determinism.

Even with a fast model like Gemma it becomes extremely slow to expand more steps into the future though, we have a branching factor of numTokens^depth, where 5^6 is already 15625 LLM calls (which takes me ~3 minutes to run) and 5^7 would be 78125 calls (15+ minutes) so the defaults I've set are all about choosing a balance which can be quickly reproduced but still demonstrates the consistency of output on subtrees.

If you're inclined towards code, this is an open source nodejs application with a react-based UI, so you can explore the setup and customise it with your own experiments. The generated trees are stored as JSON files and committed to git, which means you can easily compare differences in the output between executions.

https://github.com/Nick-Lucas/llm-probability-tree

Take for example this comparison of "The capital of France" on the left and "The capital of France is Paris" on the right. Darker shades of green are higher probabilities and purple is a completed response.

The right example goes 2 steps further into the future, but collapsing those steps you can see the printed probabilities are identical. To reference the point of this post, the LLM was always going to predict these tokens with these likelihoods even with the shorter prompt, it knows what it wants to say but it's our responsibility to plot a path to a completed response, and we choose not to always take the highest probability token at each step.

The lay-person's conclusion

So we can see that LLMs are deterministic, and understand that we wrap them with a bit of randomness to force exploration of responses you might otherwise not see. If you explore some of the output trees a bit more, you'll also see that the meaning of the responses very rarely changes significantly, it's often a choice between a short response and a more detailed one, and most paths lead to sensible responses in any case.

The final leg of this post will go into more depth on some of the low level details, it's more technically minded but if you're non-technical and interested I will still attempt to take you along on the journey!

"Latent Space"

The basic (simplified) workings of an LLM are like this:

LLMs represent tokens as a list of numbers, we call these "Embeddings" and each token has a unique list of numbers assigned. We "embed" our prompt as a list of tokens (so a list of lists of numbers) and pass it into the LLM, this is where the maths begins
The tokens are passed to a "Transformer", which uses its training to turn the tokens into a table of relationships between each token. For instance "Capital"/"France" might have a pretty strong relationship and "The"/"France" might not. This is what we call Attention, and it helps the LLM to understand the structure of an input.
The Attention table is then passed to a Neural Network which uses its training to encode learned knowledge/meaning
Steps 2+3 are repeated over many layers with their own trained parameters

aha, we've introduced "scores" and "softmax", we'll cover that in the next section

As text flows through this process, each input token carries an internal representation the model keeps refining. Those values encode a unique position in the model’s latent space. The resulting coordinates drive the next-token odds, and any of the highest probability tokens could be a valid choice to continue generation.

If you were to reach in and change one value you're now in a different position in latent space - though still quite near to your starting position given we're talking about billions of coordinates.

As demonstrated earlier, the token probabilities never change for the same input+settings, or an input suffixed with some number of next-tokens from the tree. It's just very complex maths which encodes and applies a learned understanding of the world.

Sampling: Non-determinism, Seeds, Randomness

Okay so I've avoided how we inject non-determinism for a bit now, it's definitely fair I cover where it comes from. After the forward pass through the LLM, we pass the results to our Decoder/Sampler. Sampling strategies are designed by humans so can and will vary significantly in how they work, but a historically common example uses the following process:

"temperature" - a setting typically between 0-2 which is used to reshape the probabilities of possible next tokens, bringing lower and higher probabilities nearer to each other. 0 generally results in only the highest probability token being selected, whereas higher values will bring many tokens into contention
"softmax" - not a setting, but a normalisation step. The model actually outputs raw scores (called logits), not probabilities like I've been calling them for simplicity in the first half. Softmax turns the logits into proper probabilities that add up to 1.
"top_k" - this setting dictates how many tokens should be in contention, for top_k=5 the 5 highest probability tokens are kept and remainder are discarded
"top_p" - this setting is used to produce a shortlist of tokens whose probabilities sum to >= top_p, the shortlist is constructed in probability order, and all remaining tokens are discarded
"seed" - this is used to kick off a random number generator, and the generator is used to select the next token, the probabilities still matter for weighting but there is always a probabilistic chance that a non-highest token may be selected

Using this process the sampler can produce outputs which differ each time we run the same prompt. But that doesn't mean the LLM has no idea what it wants to say, we just chose a specific path through the tree-structure which the LLM also knows its output for. Different technologies do this differently, for example OpenAI's gpt-5 has less control exposed than gpt-4 and so is harder to make deterministic.

I'll also give an honourable mention for differing GPU hardware and software, and differing compiler kernels, when deployed on large-scale infrastructure or across multiple providers. Since we're dealing with floating point maths, which can be imprecise and have differing rounding strategies, there can be subtle differences in probability output. Additionally in the extraordinarily rare case two probabilities come out identical, we may need a tie-breaking strategy.

All of this would be possible to treat as a bug and make deterministic for a single application though, assuming we wanted to. It just very often doesn't matter as we're dealing with such small differences in probability outputs.

Resources

If you want to learn more, then I would look no further than 3blue1brown's free Neural Networks course on YouTube. Grant is a master of visualising complex maths, which is a huge help when approaching LLMs at a low level. You can start from "Large Language Models explained briefly" fairly safely though starting at the beginning will give the strongest foundation.

tRPC & React Patterns: Router Factories

Nick Lucas — Sun, 26 Feb 2023 11:01:43 +0000

This post comes in 2 halves:

tRPC Router Factories
Consuming Router Factories in a React application

Preface

Let's say you have a series of tRPC routes that look something like this:

import { router, publicProcedure } from './trpc'

const appRouter = router({
  cities: router({
    list: publicProcedure./*etc*/,
    get: publicProcedure./*etc*/,
    update: publicProcedure./*etc*/,
    delete: publicProcedure./*etc*/,
  }),
  cars: router({
    list: publicProcedure./*etc*/,
    get: publicProcedure./*etc*/,
    update: publicProcedure./*etc*/,
    delete: publicProcedure./*etc*/,
  }),
  people: router({
    list: publicProcedure./*etc*/,
    get: publicProcedure./*etc*/,
    update: publicProcedure./*etc*/,
    delete: publicProcedure./*etc*/,
  }),
})

It's clear that these routers will share a lot of functionality. They'll each operate on a different entity from your ORM or external API, but otherwise the logic and inputs/outputs will be abstractly similar. In the spirit of DRY we don't want to write and test all the endpoints multiple times if we can avoid it.

This is a perfect situation to employ the age old Factory Pattern with tRPC, we want a "Router Factory".

I've personally found this an incredibly useful pattern in my work, where we have a single router factory generating over 20 different CRUD routes against an internal API, plus another factory for a common CSV import-export pattern which we use on numerous entities. It's saved us hours and made changing and testing our application much more trivial. There used to be a big pain-point in React, which is covered towards the end of this post, but I recently contributed some new APIs to tRPC which make the Router Factory pattern far more useful.

Note, this post does assume a certain level of fundamental knowledge about tRPC and React, for instance:

You understand what the Factory Pattern is in programming
You know what the trpc.ts file in your codebase is, and by extension where router and publicProcedure have come from.
You understand how the AppRouter types get from your backend to your frontend, and how your frontend creates the tRPC React Client
You understand React, props, and something of React Query
You have a vague idea of what an ORM is, preferably knowing one quite well, and so can interpret how the ~~abuse~~ use of its APIs would apply here

Router Factory basics

A router factory is fundamentally simple. The basic frame looks a little like this:

import { router, publicProcedure } from './trpc'

function createCrudRouter() {
  return router({
    /* common procedures here */
  })
}

const appRouter = router({
  cities: createCrudRouter(),
  cars: createCrudRouter(),
  people: createCrudRouter(),
})

If you're feeling confident you could probably stop reading here and just go use it, though there's one problem we'll come up against later which you might want to jump to, but let's go a step further and make this work properly with this example.

import { router, publicProcedure } from './trpc'
import type { DbEntityName } from './my-orm'

//
// DB Entities
//
// In practice these may be interfaces, classes, or
//   Zod types. It's up to you!
// The pattern does work best where they share 
//   some common type like EntityBase here.
//

interface EntityBase {
  id: number
  name: string
}

interface CityEntity extends EntityBase {
  population: number
  areaSquared: number
}

interface CarEntity extends EntityBase {
  brand: string
  colour: string
}

interface PersonEntity extends EntityBase {
  age: number
  height: number
}

//
// The CRUD Factory
// 

// We define a config type to set the factory up 
interface CrudRouterConfig<T extends EntityBase> {
  // We provide some way to select an ORM repository
  entityName: DbEntityName

  // We need an entity validator for procedure inputs
  entityType: SomeTypeValidator<T>

  // We need a filter validator for procedure inputs
  listFilter: SomeFilterFor<T>
}

function createCrudRouter<TEntity extends EntityBase>(
  config: CrudRouterConfig<T>
  ) {
  return router({
    list: publicProcedure
      .input(config.listFilter)
      .output(array(config.entityType))
      .query(opts => {
        const repo = opts.ctx.orm[config.entityName]
        return repo.query(opts.input)
      }),
    get: publicProcedure
      .input(JustId)
      .output(config.entityType)
      .query(opts => {
        const repo = opts.ctx.orm[config.entityName]
        return repo.fetchById(opts.input.id)
      })
    update: publicProcedure
      .input(config.entityType)
      .output(config.entityType)
      .mutation(opts => {
        const repo = opts.ctx.orm[config.entityName]
        return repo.update(opts.input.id, opts.input)
      }),
    delete: publicProcedure
      .input(JustId)
      .mutation(opts => {
        const repo = opts.ctx.orm[config.entityName]
        return repo.delete(opts.input.id)
      }),
  })
}

// 
// The usage
// 

const appRouter = router({
  cities: createCrudRouter<CityEntity>({
    entityName: 'city',
    entityType: tCityEntity,
    listFilter: tIdNameFilter
  }),
  cars: createCrudRouter<CarEntity>({
    entityName: 'car',
    entityType: tCarEntity,
    listFilter: tIdNameBrandFilter
  }),
  people: createCrudRouter<PersonEntity>({
    entityName: 'person',
    entityType: tPersonEntity,
    listFilter: tNameAgeHeightFilter
  }),
})

Some of the config variables (tCityEntity, tIdNameFilter, etc) in this implementation aren't defined in the example, simply because there are many approaches to input/output types in tRPC, but essentially they'll be anything you could pass to the input/output for validation.

If all you need to do is DRY out your routers, that's essentially it, but there is one extra problem this introduces in tRPC which you might need to solve...

Using Router Factories polymorphically in React

So let's say you use these routers in a React codebase, and also want to DRY out your components to be compatible with all instances of your CrudRouter. You're going to hit a problem:

import { trpc } from "./trpc-client";
import { useForm } from "./your-form-lib";

interface Props {
  // we take the type of our cities router, which is 
  //   a little awkward but we hope will work
  router: typeof trpc["cities"]
  utils: ReturnType<typeof trpc['useContext']>["cities"]
  entityFields: ReactNode
}

function CreateForm({ router, utils, entityFields }: Props) {
  const form = useForm()

  const entityCreator = router.create.useMutation({
    onSuccess() {
      utils.invalidate()
    }
  });

  return (
    <form 
      onSubmit={
        form.handleSubmit(data => {
          entityCreator.mutate(data)
        })
      }
    >
      {entityFields}

      <button type="submit">Create</button>;
    </form>
  )
}

// Great!
function CreateCityPage() {
  const utils = trpc.useContext()

  return (
    <CreateForm 
      router={trpc.cities} 
      utils={utils.cities} 
      entityFields={<CityFields />} 
    />
  )
}

function CreateCarPage() {
  const utils = trpc.useContext()

  return (
    <CreateForm 
      // TypeScript Error!!!!
      router={trpc.cars} 
      utils={utils.cars} 
      entityFields={<CarFields />} 
    />
  )
}

In CreateCarPage we see something like: Error: Type '["cars"]' is not assignable to type '["cities"]'

In this case, this happens for two reasons:

tRPC's React-Query types encode the full path of the procedure into the type as a string literal. It needs this for internal reasons, but the outcome is that even if the underlying input/output types and procedure names are identical, you can't pass one router to a type demanding another
In this case a CityEntity does in-fact have some fields which a CarEntity does not, so you can't pass a CarEntity to typeof trpc["cities"] anyway

Wouldn't it be nice if we could create an abstract router type which future implementations can be passed to polymorphically? The answer lies in @trpc/react-query/shared.

First thing we need to do is create our abstract types. This is done next to your Router Factory:

// '@trpc/react-query/shared' exports several types which can be used to generate abstract client types
import { 
  RouterLike, 
  UtilsLike 
} from '@trpc/react-query/shared';

function createCrudRouter<TEntity extends EntityBase>(
  config: CrudRouterConfig<T>
) {
  // You've seen all this!
}

// Take the `typeof` the most basic CrudRouter
type CrudRouteType = ReturnType<
  typeof createCrudRouter<EntityBase>
>;

// Generate your abstract types
// These must be exported from your API 
//   exactly like your AppRouter type
export type CrudRouter = RouterLike<CrudRouteType>
export type CrudRouterUtils = UtilsLike<CrudRouteType>

Once this is done we can update our frontend to import these types and use them:

import { trpc, CrudRouter, CrudRouterUtils } from "./trpc-client";
import { useForm } from "./your-form-lib";

interface Props {
  router: CrudRouter
  utils: CrudRouterUtils
  entityFields: ReactNode
}

function CreateForm({ router, utils, entityFields }: Props) {
  const form = useForm()

  const entityCreator = router.create.useMutation({
    onSuccess() {
      utils.invalidate()
    }
  });

  return (
    <form 
      onSubmit={
        form.handleSubmit(data => {
          entityCreator.mutate(data)
        })
      }
    >
      {entityFields}

      <button type="submit">Create</button>;
    </form>
  )
}

// Great!
function CreateCityPage() {
  const utils = trpc.useContext()

  return (
    <CreateForm 
      router={trpc.cities} 
      utils={utils.cities} 
      entityFields={<CityFields />} 
    />
  )
}

// it works!
function CreateCarPage() {
  const utils = trpc.useContext()

  return (
    <CreateForm 
      router={trpc.cars} 
      utils={utils.cars} 
      entityFields={<CarFields />} 
    />
  )
}

By using the abstract types from our router factory, we can now pass around tRPC routers and utils, and use standard React patterns to invert control of the bespoke aspects of presentation, for instance entityFields just expects some presentation which renders the relevant inputs onto the form. This vastly increases the re-usability of frontend components for our CRUD routes, and reduces the new surface area that needs testing for each entity.

Typescript Runtime Validators and DX, a type-checking performance analysis of zod/superstruct/yup/typebox

Nick Lucas — Sun, 12 Feb 2023 11:18:41 +0000

Preface

In 2023, Typescript is rarely questioned as an important tool for modern JavaScript developers, but one of its biggest limitations is the lack of added runtime type safety, particularly when dealing with IO at the boundaries of your application.

To solve this problem a number of popular runtime validation and type-safety tools have popped up, usually competing on runtime parsing speed, API expressiveness, and a TypeScript vs JSON-schema based core.

Compile-time and Editor Performance Pain

What seems to be degrading is performance of the developer experience. TypeScript drives the code intelligence of many popular editors, and as you add type complexity and size to an application or monorepo, code-completion and type-checking starts to draaaaag. This is because driving features like autocomplete means compiling your code on the fly repeatedly, and the more types & files needed to populate autocomplete for a line of code, the longer it takes to get a response.

This is compounded by the growing movement of fully and deeply typed libraries like tRPC, and Tanstack's Query & upcoming Router, which utilise new layers of generics and mapped types over your DTOs, deepening your types, and surfacing compile-time performance issues which weren't as noticeable before.

What's prompted me to look at this is my tRPC+Zod editor experience at work has become unusable. 2-3 seconds to just get autocompletion options on a tRPC path, and then another 2-3 for the next path, repeat. When investigated using TypeScript's tracing tools the data entirely points back to my team's Zod DTOs. What I learned is that Zod's performance is okay at the start, but when you start using methods like .extend/.pick/.omit (and so on) the performance regresses in the order of a magnitude. Rather than making this into a "Zod considered bad" post, I wanted to investigate how the alternatives which can be integrated with tRPC fare, and see whether I can do better.

Analysis Setup

For these benchmarks I'm using a Macbook Pro with M2 Pro and 32GB of RAM.

I've worked on 2 codebases in an Nx monorepo to set up this experiment.

Light Type - an experimental runtime type-checker with feature parity in a number of key areas (and an inspired API) to Zod
Benchmarks - a tRPC API and React SPA which uses a number of popular and tRPC-compatible type-checkers

The type-checkers under test are:

Zod 3.20.6
Yup 1.0.0
Superstruct 1.0.3
TypeBox 0.25.21
Light-Type (Experimental Version)

There may be more libraries which can be integrated with tRPC, but the first 3 are recommended by the project, and TypeBox has had some effort made to integrate it. Light-Type is designed to fit the necessary interfaces of Zod so it's also compatible natively.

The Benchmarks are each made up of a tRPC router with types co-located, and a React Component using tRPC's @tanstack/query abstraction which consumes the router.

TypeScript is pretty good at optimising its compiles. It caches where it can and where there's a single path back to a type it doesn't spread itself wider. So when performance tracing this setup I've seen that compile-times for each router/component don't bleed onto the others, making benchmarking the compiler much easier.

The Actual Types

Each library has two benchmarks, a 'simple' one, and a 'complex' one which adds usage of the equivalent to extend/omit in each library. Except Yup which doesn't have the necessary features to join the complex round.

Simple Router Example

// simpleZodRouter.ts

import { z } from 'zod'
import { publicProcedure, router } from '../trpc'

const PersonDto = z.object({
  id: z.number(),
  firstName: z.string(),
  lastName: z.string(),
  tel: z.string().optional(),
})

const CarDto = z.object({
  id: z.number(),
  name: z.string(),
  age: z.number(),
  brand: z
    .union([
      z.literal('Volvo'),
      z.literal('Mercedes'),
      z.literal('BMW'),
      z.literal('Ferrari'),
      z.literal('Bazmus'),
    ])
    .optional(),
  previousOwners: z.array(PersonDto).default([]),
})

type DbCar = typeof CarDto['_input']
type Brand = typeof CarDto['_output']['brand']

export const simpleZodRouter = router({
  list: publicProcedure
    .input(z.object({ count: z.number().default(100) }))
    .output(z.array(CarDto))
    .query((opts) => {
      return new Array(opts.input.count).fill(null).map<DbCar>((_, idx) => {
        return {
          id: idx,
          name: 'Foobarmo',
          age: 3,
          brand: 'Bazmus' as Brand,
          previousOwners: [
            {
              id: idx * 100,
              firstName: 'Bob',
              lastName: 'Owneverythingman',
              tel: undefined,
            },
          ],
        }
      })
    }),
  get: publicProcedure
    .input(z.object({ id: z.number() }))
    .output(CarDto)
    .query((opts) => {
      return {
        id: opts.input.id,
        name: 'Foobarmo',
        age: 3,
        brand: 'Bazmus' as Brand,
        previousOwners: [
          {
            id: opts.input.id * 100,
            firstName: 'Bob',
            lastName: 'Owneverythingman',
            tel: undefined,
          },
        ],
      }
    }),
  create: publicProcedure
    .input(CarDto)
    .output(CarDto)
    .mutation((opts) => {
      return opts.input as DbCar
    }),
})

Complex Router Example

// complexZodRouter.ts

import { z } from 'zod'
import { publicProcedure, router } from '../trpc'

const EntityDto = z.object({
  id: z.number(),
})

const PersonDto = EntityDto.extend({
  firstName: z.string(),
  lastName: z.string(),
  tel: z.string().optional(),
})

const CarDto = EntityDto.extend({
  id: z.number(),
  name: z.string(),
  age: z.number(),
  brand: z
    .union([
      z.literal('Volvo'),
      z.literal('Mercedes'),
      z.literal('BMW'),
      z.literal('Ferrari'),
      z.literal('Bazmus'),
    ])
    .optional(),
  previousOwners: z.array(PersonDto).default([]),
})

type DbCar = typeof CarDto['_input']
type Brand = typeof CarDto['_output']['brand']

export const complexZodRouter = router({
  list: publicProcedure
    .input(z.object({ count: z.number().default(100) }))
    .output(z.array(CarDto))
    .query((opts) => {
      return new Array(opts.input.count).fill(null).map<DbCar>((_, idx) => {
        return {
          id: idx,
          name: 'Foobarmo',
          age: 3,
          brand: 'Bazmus' as Brand,
          previousOwners: [
            {
              id: idx * 100,
              firstName: 'Bob',
              lastName: 'Owneverythingman',
              tel: undefined,
            },
          ],
        }
      })
    }),
  get: publicProcedure
    .input(z.object({ id: z.number() }))
    .output(CarDto)
    .query((opts) => {
      return {
        id: opts.input.id,
        name: 'Foobarmo',
        age: 3,
        brand: 'Bazmus' as Brand,
        previousOwners: [
          {
            id: opts.input.id * 100,
            firstName: 'Bob',
            lastName: 'Owneverythingman',
            tel: undefined,
          },
        ],
      }
    }),
  create: publicProcedure
    .input(CarDto.omit({ id: true }))
    .output(CarDto)
    .mutation((opts) => {
      return opts.input as DbCar
    }),
})

Running the benchmark

# Set up the monorepo
yarn install

# The app/api can served as a test of the integration
yarn light-type-benchmark:start

# Run a traced compile of the repo
yarn tsc:trace
> tsc -p tsconfig.base.json --noEmit --incremental false --jsx react-jsx --allowSyntheticDefaultImports --generateTrace ./.trace

tsc:trace produces ./.trace/trace.json which can be analysed in your browser's dev tools or Perfetto.

This does produce a lot of tracing output which we don't need, but since the benchmarks are isolated in their own projects TypeScript produces reliable results for them. I've edited down the trace file just to the files we're concerned with.

Results

And here are the results:

You can download and explore the trace yourself here: https://gist.github.com/Nick-Lucas/ee964bf004f40ea6601bd2543869efee

While compile-times do vary slightly from run to run, the scale of difference between each library remains the same.

Library	Test	Router ms	Component ms	Note
Zod	Simple	24ms	10ms
Zod	Complex	281ms	17ms	281ms is not a typo
Yup	Simple	22ms	9ms
Yup	Complex			Could not join the fun
Superstruct	Simple	10ms	11ms
Superstruct	Complex	42ms	18ms
Typebox	Simple	28ms	11ms
Typebox	Complex	38ms	9ms
Light Type	Simple	8ms	25ms
Light Type	Complex	22ms	11ms

Conclusions

The big outlier here is Zod, which seems to have some structural issues with its types, ballooning in cost by 10+ times after just 2 .extend and 1 .omit call. All the other libraries perform pretty well in the same situation, though Superstruct seems to balloon by about 4x, and TypeBox is slower to begin with but then degrades less poorly. Yup doesn't compete on complex features but performs very well at what it's designed for.

It's clear that all the type processing for the libraries is happening in the processing of router files where those types are located, and those are quite varied in cost. These results are reproducible over and over again too, it's not a die-roll for the big differences. The react components themselves seem to be isolated and very fast to compile.
Light Type is consistently the fastest across multiple benchmarks. This is going to be down to a couple factors:
1. It has fewer features than the other libraries. I would predict it will slow slightly over time as I continue developing it.
2. Light Type has a lot of focus put into producing the simplest possible types at every stage. Which for example Zod doesn't seem to prioritise.

Notes specifically on Zod: That big block in the flamegraph is Zod's Complex test. Basically don't try to use any features outside of primitives, object, array, or you may immediately gain 10x 🤒 to that type.

The sum total of all type compilations is your editor experience, and 281ms is already about as high as you probably want your entire library of types to take for autocomplete.
Those brown tags in the top of the flamegraph are labelled recursiveTypeRelatedTo_DepthLimit, I'm really not an expert on TypeScript's insides, but that sounds like it's trying to tell us something important, and Zod has been guilty of causing "Type instantiation is excessively deep and possibly infinite" errors.
Zod right now, looks to be a big footgun and in need of rethinking its types, at least when combined with other modern tools like tRPC. If you're starting a new project it might one to avoid for now, though there's really not much alternative if you love Zod's API

Relational Data Tables in Unreal Engine 4

Nick Lucas — Sat, 26 Feb 2022 15:50:41 +0000

I had an interesting problem with Unreal Engine today. I have a Mech character which I want to attach stuff to, maybe it's segments of armour, or internal components, or weapons and other "stuff". Managing all these attach points and attachments is a pain, so I wanted to codify it in a data model, using Data Assets and Data Tables. This way I can have 100 mechs in the future with different skeletons and varieties of attachments, and codify them entirely as data with a single "MechCharacter".

Given my background as a full stack software engineer, my instinct it to reach for a relational database of sorts, which can:

Drive common behaviour with data
Tie together multiple data sources
Self validate and restrict inputs

This isn't necessarily a complete or cohesive list. I'm writing this quite quickly. But basically I want to build a data model and code which gives me an amazing developer experience while building future assets. No weird bugs because I made a minor typo in one row somewhere!

Lastly, maybe there's even more to this I haven't found yet? Maybe there's a superior way to do it? Maybe a best practices issue I'm going to hit down the line? Let me know in the comments!

So let's say I have these 3 structs and an overarching Data Asset. Each struct will be used to create a Data Table in the editor,

USTRUCT(BlueprintType)
struct FAttachPoint : public FTableRowBase
{
    GENERATED_BODY()

public:
    UPROPERTY(EditAnywhere, BlueprintReadWrite)
        FName Bone;
};

USTRUCT(BlueprintType)
struct FArmorPiece : public FTableRowBase
{
    GENERATED_BODY()

public:
    UPROPERTY(EditAnywhere, BlueprintReadWrite)
        FName AttachPointRowKey;

    UPROPERTY(EditAnywhere, BlueprintReadWrite)
        UStaticMesh* Mesh;
};

USTRUCT(BlueprintType)
struct FHardpoint : public FTableRowBase
{
    GENERATED_BODY()

public:
    UPROPERTY(EditAnywhere, BlueprintReadWrite)
        FName AttachPointRowKey;

    UPROPERTY(EditAnywhere, BlueprintReadWrite)
        UStaticMesh* Mesh;
};

UCLASS()
class MECHGAME_API UDA_MechDefinition : public UDataAsset
{
    GENERATED_BODY()

public:
    UPROPERTY(EditAnywhere, BlueprintReadOnly)
        USkeletalMesh* Mesh;

    UPROPERTY(EditAnywhere, BlueprintReadOnly)
        UDataTable* ArmorTable;

    UPROPERTY(EditAnywhere, BlueprintReadOnly)
        UDataTable* HardpointsTable;
};

This is not ideal, because ArmorPiece and Hardpoint Data Tables just expect a row key of an AttachPoint, but I could make a typo, or rename a row and forget to update other data. What if UE4's editor could restrict those FNames to only keys found in a particular AttachPoint Data Table?

Well UE4 actually has some buried ways to achieve this.

// For a reference to a table which uses a particular struct:

UPROPERTY(EditAnywhere, BlueprintReadOnly, meta = (RequiredAssetDataTags = "RowStructure=ArmorPiece"))
UDataTable* ArmorTable;

// For a handle to a single row within a table which uses a particular struct:

UPROPERTY(EditAnywhere, BlueprintReadWrite, meta = (RowType = "AttachPoint"))
FDataTableRowHandle AttachPoint;

The new meta information informs the editor how to provide lists of data, restricted by struct type which makes it harder to make mistakes when configuring data models later.