When ‘Magic’ Works: Type-Level Tricks in TypeScript

Sufficiently Advanced Technology

The received wisdom among developers is to avoid magic: magic numbers, magic strings, and especially framework-flavored "auto-magic". It's brittle, it's unpredictable, and it's not explicit.

On the other hand, Arthur C. Clarke stated that any sufficiently advanced technology is indistinguishable from magic.

I think the operative difference between harmful magic and sufficiently advanced technology is how well it works. And I think that TypeScript type tricks are an excellent platform for building the latter.

Bad Magic: An Anecdote

I once worked with a team which was building an auto-instrumentation framework for ES2020. The goal was to provide a low-friction mechanism (i.e.; an attribute) to instrument a wide variety of language features - method calls, property access, yields from a generator, and so on.

After two months, we were still struggling with edge cases and memory leaks. We ultimately pulled the plug.

Why wasn't this project successful? Without itemizing the individual technical shortcomings that made up the failure, I think the root cause was that the scope was too large.

The ES2020 runtime is very complicated. For example, check out these obscure corners. But a foolproof auto-instrumentation tool must handle all of these language features correctly.

Here's my takeaway: for "magic" to work reliably and without enormous engineering effort, a restricted scope seems necessary.

Type Tricks Have Limited Scope

Type systems are declarative and finite; runtimes are dynamic and stateful. This makes type-based auto-magic much simpler to pull off. This is where TypeScript really shines.

Zod, Prisma, and Hono all use extensive auto-magic typing tricks and are resounding successes. Zod, for example, is widely adopted.

Indeed, TypeScript's advanced type system seems designed to encourage magic.

Good Magic: Quick Case Study

Suppose we have some unmanaged resource that we retrieve by scoped ID:

• employee/543293 represents Employee 543293.
• department/55 represents Department 55.
• building/3 represents Building 3.

The resource is ingested into the runtime as some kind of untyped "thing" (bytes, JSON, database record, etc.).

So we have to cast with as.

For example, we might write:

const employee = (await retrieve(`employee/543293`)) as Employee;

Or, if we don't want to do that, we could write a special-built method per entity:

function retrieveEmployee(identifier : string) : Promise<Employee> {
   return retrieve(identifier) as Promise<Employee>;
}

const employee : Employee = await retrieveEmployee(`employee/543293`);

Both patterns do not scale well with the number of entities - we either have a proliferation of per-entity methods, or a proliferation of usages of as.

Is there a better way? Thankfully yes. It is possible to write the code very cleanly:

const employee : Employee = await retrieve(`employee/${key}`);

const department : Department = await retrieve(`department/${employee.departmentId}`);

const building : Building = await retrieve(`building/${department.buildingId}`);

Notice how that the call-site signature for retrieve is the same for every line (no generic types supplied in angle brackets, for example).

TypeScript is automatically inferring the return type based on the contents of a template literal!!!

Here's the source, followed by an explanation:

type Employee = {
    name : string
    departmentId : string;
}

type Department = {
    id : string
    name : string
    budget : number
    buildingId: string
}

type Building = {
    id : string
    name : string
}

type KeywordToObjectType = {
    employee : Employee,
    department : Department,
    building : Building
}

// Keyword ::= employee, department, building
type Keyword = keyof KeywordToObjectType;

// Format of the string literal
type ObjectIdentifier = `${Keyword}/${string}`;

// Utility: Extracts the keyword out of the string literal
type ExtractKeyword<T> = T extends `${infer K extends Keyword}/${string}` ? 
    K : 
    never;

// Utility: Maps ObjectIdentifier to appropriate type
type TypeForObjectIdentifier<T> = KeywordToObjectType[ExtractKeyword<T>];

// And this is the retrieve function, which handles the cast
export async function retrieve<T extends ObjectIdentifier>(identifier: T) : 
    Promise<TypeForObjectIdentifier<T>> {
    const retrievedValue = {}; // stub -> fetch / parse / etc.
    return retrievedValue as TypeForObjectIdentifier<T>;
}

Generic Types As "Functions On Types"

Note the return type of retrieve: TypeForObjectIdentifier<T>.

It's helpful to think of TypeForObjectIdentifier<T> as a "function on type T", rather than a generic type in the traditional sense.

Therefore the declaration of TypeForObjectIdentifier<T> amounts to a simple "type program" which accepts an object identifier T and returns the appropriate object type. For example, by mapping building/3 to object type Building.

Template Literal Parsing

Type-level parsing of string template literals is the heart of the entire trick. It is what allows TypeScript to infer the return type based on the first portion of the object identifier.

TypeScript has the ability to perform type inference on template literals - even if some portions of which may have values that are only known at runtime!

In our declaration of ExtractKeyword<T>, we ask TypeScript to match T to the template literal ${infer K extends Keyword}/${string}, otherwise return never (which can trigger a compiler failure).

type ExtractKeyword<T> = T extends `${infer K extends Keyword}/${string}` ? 
    K : 
    never;

When TypeScript matches building/${department.buildingId} to ${infer K extends Keyword}/${string}, department.buildingId is statically known to be a string.

Likewise, building is bound to K by way of the keyword infer: we can think of K as if it is a named capture group in a regex. Its value is a side effect of matching the object identifier to the template literal type.

Indexed Access Types

In TypeScript's "type language", the syntax Type['property'] produces the type corresponding to the property property on Type. This is called a indexed access type. Therefore, KeywordToObjectTypeMap['building'] produces the type Building.

Putting It All Together

type TypeForObjectIdentifier<T> = KeywordToObjectType[ExtractKeyword<T>];

Now it should be clearer how TypeForObjectIdentifier<T> works: Extract the Keyword portion of the string template via ExtractKeyword<T>, and then map it to the object type.

Adding Runtime Safety

This trick does not guarantee runtime safety. If the shape of the retrieved object does not conform to the expected type given its identifier, we're past the limits of what compile-time safety can guarantee. We can remedy this with some clever use of Zod schemas.

Here is the code in entirety:

import z from "zod"

const Employee = z.object({
    id: z.string(),
    name: z.string(),
});

const Department = z.object({
    id : z.string(),
    name : z.string(),
    budget : z.number(),
    buildingId: z.string()
});

const Building = z.object({
    id : z.string(),
    name : z.string()
});

const KeywordToZodSchema = {
    employee: Employee,
    department: Department,
    building: Building
};

type Keyword = keyof typeof KeywordToZodSchema;

type KeywordToObjectType = {
    [key in Keyword]: z.infer<(typeof KeywordToZodSchema)[key]>
}

// Format of the string literal
type ObjectIdentifier = `${Keyword}/${string}`;

// Utility: Extracts the keyword out of the string literal
type ExtractKeyword<T> = T extends `${infer K extends Keyword}/${string}` ? 
    K : 
    never;

// Utility: Maps ObjectIdentifier to appropriate type
type TypeForObjectIdentifier<T> = KeywordToObjectType[ExtractKeyword<T>];

function extractKeyword(identifier: string) : Keyword {
    const keyword = identifier.split('/')[0];
    if (!(keyword in KeywordToZodSchema)) {
        throw new Error("Invalid identifier");
    }
    return keyword as Keyword;
}

export async function retrieve<T extends ObjectIdentifier>(identifier: T) : 
    Promise<TypeForObjectIdentifier<T>> {
    const retrievedValue = {}; // fetch / parse / etc.
    const keyword = extractKeyword(identifier);
    const parsed = KeywordToZodSchema[keyword].parse(retrievedValue);
    return parsed as TypeForObjectIdentifier<T>;
}

The Runtime Validation Refactor - Step by Step

First, we redefined our types as Zod schemas.

import z from "zod"

const Employee = z.object({
    id: z.string(),
    name: z.string()
});

const Department = z.object({
    id : z.string(),
    name : z.string(),
    budget : z.number(),
    buildingId: z.string()
});

const Building = z.object({
    id : z.string(),
    name : z.string()
});

Then, we defined a mapping from keyword to schema as a const map.

const KeywordToZodSchema = {
    employee: Employee,
    department: Department,
    building: Building
};

Then, we recovered the KeywordToObjectType from our original implementation using a mapped type. This keeps the type definition automatically in sync with the const mapping from keyword to schema, which is available at runtime.

type Keyword = keyof typeof KeywordToZodSchema;

type KeywordToObjectType = {
    [key in Keyword]: z.infer<(typeof KeywordToZodSchema)[key]>
}

At runtime, we need to be able to map object identifier values to keyword values, so we defined a runtime counterpart to ExtractKeyword<T> called extractKeyword:

function extractKeyword(identifier: string) : Keyword {
    const keyword = identifier.split('/')[0];
    if (!(keyword in KeywordToZodSchema)) {
        throw new Error("Invalid identifier");
    }
    return keyword as Keyword;
}

And finally, we re-implemented retrieve to perform runtime validation with Zod. First, we extract the keyword from the identifier. Then, we index into KeywordToZodSchema to select the appropriate Zod schema object. Finally, we parse and return the result.

export async function retrieve<T extends ObjectIdentifier>(identifier: T) : 
    Promise<TypeForObjectIdentifier<T>> {
    const retrievedValue = {}; // fetch / parse / etc.
    const keyword = extractKeyword(identifier);
    const schema = KeywordToZodSchema[keyword]
    const parsed = schema.parse(retrievedValue);
    return parsed as TypeForObjectIdentifier<T>;
}

There's an unavoidable as because we have to narrow parsed from a type union covering all possible object types into the specific type as represented by TypeForObjectIdentifier<T>.

Conclusion

It is possible, helpful, and practical to implement "magic type tricks" for cleaner TypeScript code. In fact, many popular frameworks do it, and you can do remarkable things with template literals.

Rather than create trouble and confusion, these tricks increase type safety and make for a better developer experience.