DEV Community: Jo Franchetti

npm install && pray

Jo Franchetti — Thu, 12 Mar 2026 15:12:18 +0000

There’s a ritual every JavaScript developer knows.

You’re building something. You need to parse a date, validate an email, or format a number. So you do what any reasonable person does, you open a terminal, type npm install, pick a package with four million weekly downloads and a friendly README, and move on.

For a long time, that felt safe. The openness of npm and the kindess of opensource developers helped turn the JS ecosystem into something genuinely remarkable. We had a global library of shared effort, where someone could publish a utility at 2am and developers on the other side of the world could be using it in production by morning.

But the openness and expansiveness of the JavaScript ecosystem was built on trust. And, sadly, that trust is constantly being eroded.

The worm that changed the mood

In September 2025, ReversingLabs documented the first known self-replicating worm in the npm ecosystem. They traced the initial compromise to rxnt-authentication@0.0.3, then watched the malware spread through compromised maintainer accounts into hundreds of packages.

A couple of months later, Datadog described a second wave: Shai-Hulud 2.0. That campaign backdoored 796 npm packages totaling more than 20 million weekly downloads.

The mechanics were elegant in the worst possible way. Infect one developer, steal their npm credentials, then publish tainted versions of other packages they maintain, and let every new victim become the next distribution channel. The malware hunted for environment variables, cloud tokens, GitHub credentials, and npm auth tokens, then exfiltrated what it found while the application appeared to run normally. The infected developers none the wiser.

Then came the phishing attack against maintainer Josh Junon (qix). Attackers used a fake npmjs.help domain and a convincing support email to steal publishing credentials. From there, they pushed malicious versions of widely used packages including chalk, debug, ansi-styles, strip-ansi, and other foundational libraries. At the time of disclosure, the affected packages accounted for roughly 2.6 billion weekly downloads.

Not a particularly sophisticated exploit. Just a phishing email and the right publishing permissions.

Why this keeps working

Supply-chain attacks succeed because the trust model still assumes the ecosystem is friendlier and safer than it is.

Version ranges put trust on autopilot

When your package.json says ^2.1.0, your CI can quietly pull in 2.1.1 the moment it ships. That’s convenient! It means a maintainer’s latest publish can slide straight into your build without a human ever reviewing it, but you are implicitly trusting that every update is safe.

The soft target is the maintainer, not the package

The attack on Qix didn’t require finding a flaw in chalk or debug. It required compromising the person who could publish them. Once the account falls, all downstream trust falls with it. Opensource maintainers are only human, and often tired and overworked humans. That makes them a soft target, and the attack surface is huge: npm has millions of maintainers, many of whom have access to publish widely used packages.

Install-time execution is an attacker’s dream

npm lifecycle scripts like preinstall, install, and postinstall run during installation. That means malicious code can execute before you import anything, before your tests run, and before you’ve even finished reading the package name. Shai-Hulud took advantage of exactly that kind of install-time execution.

And the malicious behavior doesn’t need to look dramatic. A lifecycle script, a conditional payload, a quietly spawned subprocess, a snippet that only activates when certain environment variables exist — all of that can slip past casual review.

AI has widened the blast radius

AI coding assistants are genuinely useful. I use them. You probably do too. They’re great at boilerplate, they explain unfamiliar code faster than docs, and they often catch the kind of obvious mistakes that cost real time.

But they also create a new category of trust problem: code you didn’t write, don’t fully understand, and may run too quickly.

In February 2025, Truffle Security reported that a scan of the December 2024 Common Crawl archive found roughly 12,000 live API keys and passwords embedded in public web data — the same kind of data that can end up in model training pipelines.

In May 2025, KrebsOnSecurity reported that an xAI employee exposed a private API key on GitHub that could access private or fine-tuned models tied to internal data from companies including SpaceX and Tesla. In November 2025, Wiz reported that 65% of companies it analyzed from the Forbes AI 50 had verified secret leaks on GitHub.

Even the companies and systems building AI tools suffer from the same secret management failures as everyone else!

The second problem is more mundane and, in practice, more common. AI-generated code fails in ordinary ways. A cleanup script deletes the wrong directory. A migration points at the wrong database. A one-line “helper” turns into a subtle security bug. These aren’t exciting science-fiction failures. They’re the normal failures of code produced someone that has no real understanding of your environment (which is what an AI assistant is).

And once that code runs with your local permissions, a mistake isn’t just a bug. It can be data loss, credential exposure, or an accidental production incident.

We should also talk about prompt injection.

In August 2025, researcher Johann Rehberger showed that Claude Code could be tricked by an indirect prompt injection into reading sensitive local files and leaking their contents through DNS requests. Around the same time, GitHub detailed prompt-injection risks in VS Code and Copilot agent mode, including the possibility of leaking tokens, exposing confidential files, or even executing code without the user’s explicit approval.

The attack surface isn’t just the packages you install anymore. It’s also the code, issues, pull requests, docs, and comments your assistant is asked to look at.

Model poisoning is also a real, though still somewhat experimental, threat. Researchers have shown that injecting as few as 250 malicious documents into pre-training data can successfully backdoor an LLM with 90% attack success rates. You wouldn't know the model was compromised. It would behave normally except when specific conditions were met.

Defence-in-depth

This is the part where it’s worth being concrete, because there are real defences. They just require a different default mindset.

Least-privilege execution matters.

Deno’s permission model is a meaningful improvement here. By default, a Deno program cannot read environment variables, touch the filesystem, or make outbound network requests unless you allow it.

So when a malicious payload tries to read HOME or scrape your environment, the runtime can stop it cold:

┏ ⚠️  Deno requests env access.
┠─ To see a stack trace for this prompt, set the DENO_TRACE_PERMISSIONS environmental variable.
┠─ Learn more at: https://docs.deno.com/go/--allow-env
┠─ Run again with --allow-env to bypass this prompt.
┗ Allow? [y/n/A] (y = yes, allow; n = no, deny; A = allow all env permissions) >

But this is old news, we all already know about Deno's permissions model, and it isn’t a complete solution. If code legitimately needs network or environment access, you still have to grant it. But it does shrink the default blast radius in a way Node.js traditionally has not.

Treat AI-generated code as untrusted code.

That’s the mindset shift that matters most. Running model-generated code with the same implicit trust you give your own code feels normal, but it isn’t. You’re executing output from a system that may have learned from insecure examples, can misunderstand your environment, and can be steered by untrusted content it was asked to analyse. It doesn't have your best interests at heart, not because it is malicious by nature, but because it just doesn't know.

For execution you don’t control, use real isolation.

The gold standard is running untrusted code somewhere with no ambient access to your host filesystem, host secrets, or unrestricted network.

Docker can do this, but there’s real workflow friction there: building images, managing lifecycle, mounting the right files, and keeping the whole setup ergonomic enough that you’ll actually use it.

A solution - isolated dev environments

One emerging option is Deno Deploy Sandbox, which provides programmatic access to Firecracker-based Linux microVMs. These aren’t just containers with nicer branding. Each sandbox gets its own filesystem, network stack, and process tree, and the platform advertises startup times under 200 milliseconds — fast enough to use inside an agent loop.

Here’s what that looks like in practice: ask a model to write some code, run it in a sandbox, and make sure your secrets can be used only where you intend.

Below is an example script that asks Claude to write a Deno script that fetches the current Bitcoin price from the CoinGecko API, then runs it in a sandbox with no permissions except outbound network access to api.coingecko.com. Even if the generated code is malicious, it can only talk to that one host, and it can’t access any secrets or your local filesystem:

// Step 1: Import the Anthropic SDK and the Deno Sandbox SDK.
// The Anthropic SDK lets us talk to Claude, and @deno/sandbox gives us
// a secure microVM to run untrusted code in.
import Anthropic from "npm:@anthropic-ai/sdk";
import { Sandbox } from "jsr:@deno/sandbox";

// Step 2: Create an Anthropic client.
// It automatically picks up ANTHROPIC_API_KEY from the environment.
const client = new Anthropic();

// Step 3: Ask Claude to write some code for us.
// We're asking for a complete, runnable Deno script — Claude will return
// it wrapped in a markdown code block.
const response = await client.messages.create({
  model: "claude-opus-4-6",
  max_tokens: 1024,
  messages: [
    {
      role: "user",
      content: "Write a Deno script that fetches the current Bitcoin price from the CoinGecko API and prints it."
    }
  ]
});

// Step 4: Pull the generated code out of Claude's response.
// The response is an array of content blocks — we check it's a text block,
// then strip the markdown fences to get the raw source.
const firstBlock = response.content[0];
if (firstBlock.type !== "text") {
  throw new Error(`Unexpected content type: ${firstBlock.type}`);
}
const generatedCode = extractCode(firstBlock.text);
//          console.log("Generated code:\n", generatedCode);

// Step 5: Create a sandbox — a fully isolated Linux microVM.
// "await using" means it's automatically destroyed when this scope ends,
// so we never leak resources even if something throws.
await using sandbox = await Sandbox.create();

// Step 6: Write the AI-generated code into the sandbox filesystem.
// The sandbox has its own isolated filesystem — nothing here touches the host.
await sandbox.fs.writeTextFile("/tmp/generated.ts", generatedCode);

// Step 7: Run the code inside the sandbox with Deno.
// We only grant network access to the one host the code actually needs.
// stdout and stderr are piped so we can capture and display them.
const child = await sandbox.spawn("deno", {
  args: [
    "run",
    "--allow-net=api.coingecko.com", // Only the specific host we expect
    "/tmp/generated.ts"
  ],
  stdout: "piped",
  stderr: "piped"
});

// Step 8: Set a hard timeout.
// AI-generated code could accidentally (or maliciously) loop forever —
// this ensures we kill the process after 10 seconds no matter what.
const timeout = setTimeout(() => child.kill(), 10_000);

// Step 9: Wait for the process to finish and print the results.
// output.stdoutText / stderrText are pre-decoded UTF-8 strings.
// output.status.success is true only if the exit code was 0.
try {
  const output = await child.output();
  console.log("Output:\n", output.stdoutText ?? "No output");
  if (!output.status.success) {
    console.error("Error:\n", output.stderrText ?? "No error output");
  }
} finally {
  clearTimeout(timeout);
}

// Helper: extract the first code block from a markdown string.
// Falls back to returning the raw text if no fences are found.
function extractCode(text: string): string {
  const match = text.match(/```
{% endraw %}
(?:typescript|ts|javascript|js)?\n([\s\S]*?)
{% raw %}
```/);
  return match ? match[1] : text;
}

Notice what's happening here: if you run this script, you will get some AI gen code, which you have not read and not checked, but it can only reach api.coingecko.com. It is running on an entirely separate machine from your own. The generated code can't read your .env file. It can't write to your own filesystem. It can't call home to anywhere unexpected. And if it runs longer than 10 seconds, it gets killed.

The point is the layering.

The microVM keeps generated code away from your host OS. The --allow-net flag limits outbound traffic to exactly the host the script needs. You're giving yourself layers of safety.

And Deno sandboxes offer one more layer that’s worth mentioning, that we don't use in this example: they have a built-in secret management system. You can create secrets in the sandbox configuration with a secrets field. The value of the secret is replaced by a placeholder string inside the sandbox environment. The real values are only injected at the network layer when the code tries to send them to the specific hosts you allow. So even if the generated code tries to exfiltrate secrets, it can only send the placeholder values, which are useless to an attacker!

Imagine the following:

await using sandbox = await Sandbox.create({
  secrets: {
    ANTHROPIC_API_KEY: {
      value: Deno.env.get("ANTHROPIC_API_KEY") ?? "",
      hosts: ["api.anthropic.com"] // Only allow this secret to be sent to the Anthropic API
    }
  }
});

await sandbox.sh`echo $ANTHROPIC_API_KEY`;

The output key will look like: DENO_SECRET_PLACEHOLDER_55eb59627bc5700ae19f371791e0f54ad3204b745afcdc60acd66c23. Your key is nice and safe.

And yes, these are somewhat contrived examples, but I wanted something to show you quickly how easy it is to spin up and run code in a Deno Sandbox.

Since these sandboxes are full linux vms, you could of course install Claude directly on the sandbox and use that to generate code, rather than generating it on your local machine. Sandboxes allow for snapshotting of a vm image, so you could, absolutely, install Claude and any other tools or software you might want, snapshot the VM and then spin multiple versions of that up in 200ms allowing really fast execution times for your Claude prompts and code.

What actually needs to change

The tools are getting better. Our habits need to catch up.

We built a lot of modern JavaScript on the assumption that convenience deserved the benefit of the doubt. In the early days of npm, that mostly worked. The ecosystem was smaller, the incentives were different, and the threat model was simpler.

That isn’t the world we operate in now.

Supply-chain attacks are organized, repeatable, and economically attractive. AI assistants are generating code that runs with your permissions but without your oversight. Prompt injection has moved from theory to working demonstrations. And when something malicious does land, it usually goes after the credentials that unlock everything else.

The answer is not to stop using npm or to stop using AI assistants. Both are too useful, and neither is going away.

The answer is to stop giving code implicit trust just because it arrived through a convenient path.

Run code with the minimum permissions it needs. Treat generated code as untrusted until it earns more trust. For anything you don’t fully control, always execute it in isolation.

npm install && pray used to be a joke about dependency sprawl. It now reads more like a description of the default security model.

That model needs to change. Stay safe out there!

Using AI to prototype games in the browser

Jo Franchetti — Mon, 01 Dec 2025 14:19:46 +0000

There's a lot of discourse around the use of AI in games (and elsewhere!) at the moment. It's understandable, games are more expensive to make than ever, and teams are stretched thin trying to deliver bigger worlds, richer stories, and more reactive systems. Generative AI as a solution has landed with a mix of hype, suspicion, and legitimate curiosity. Many developers are still trying to make sense of where it actually fits into their existing workflows.

I've been experimenting with how large language models can help prototype narrative games in the browser, to see what happens when you mix modern LLMs with the kind of structure that is already used for procedural content. Expanding the toolbox, as it were. I'm not trying to replace writers or designers, instead I hope to augment their work. As a developer I honestly need the help writing, and as a player I want to explore more worlds!

What's in this post?

This post walks through building a simple prototype. We'll look at past uses of AI in games, cover using local models, and how to build a templated system for creating worlds and stories.

AI in games

AI in games is not new. Game developers have been using some form of AI almost from day one. The term “AI” itself is a bit of a catch-all, covering everything from simple decision trees to statistical models with billions of parameters. Games like Pong and Space Invaders had basic rule-based systems to control 'enemy' movement, Rogue and Elite introduced procedural generation to create varied experiences within limited memory.

As games grew more complex, so did the AI techniques used. Classic game dev techniques that use procedural generation include:

Path-finding - the moving of a character or NPC from point A to point B while avoiding obstacles. This often involves some variant of the A* search algorithm, or the use of navigation meshes in 3D spaces to guide NPCs around levels.
Procedural content generation - creating levels, items, or even entire worlds algorithmically. From Rogue's procedural dungeons to Minecraft's infinite voxel worlds and the vast cosmos of No Man's Sky.
Gameplay AI - gives NPCs a semblance of intelligence, using finite state machines, behaviour trees, or utility systems.
More bespoke approaches like Left 4 Dead's Director system, which adjusts pacing and difficulty dynamically based on the player's performance.

Generative AI then introduces a new layer to this landscape.LLMs can be used to generate NPC dialogue, quests, or even entire story arcs on the fly. This is a step beyond traditional procedural generation and relies heavily on predefined rules and templates.

Broadly speaking, AI in games falls into two camps:

AI that runs during gameplay, handling moment-to-moment decision making.
AI that assists development, generating assets, levels, or tools behind the scenes.

The second category is less visible to players, but it's been quietly growing for years. Developers use AI-assisted tools for everything from mesh generation and texture synthesis to complex animation pipelines - supported by inverse kinematics and motion-capture clean-up.

Machine-assisted authoring isn't new, but the explosion of modern generative AI has amplified it. The biggest question now is how to use these tools responsibly and effectively without breaking the systems and workflows that developers rely on.

Wildermyth and “structured randomness”

One of my favourite reference points for procedural storytelling is Wildermyth by Worldwalker Games. It's a tactical RPG that leans heavily on procedural generation to create bespoke stories and characters that evolve across a campaign.

Wildermyth mixes handcrafted narrative arcs with procedural character traits, relationships, and emergent events. The team published a helpful explanation of their templating system, which outlines how stories are assembled around randomised components: Wildermyth generic campaign.

The effect is pleasingly coherent. Characters age, form relationships, suffer injuries, retire and die. And the game doesn't just track those changes, it weaves them back into the story. A character who retired in one game might pop up as a wild old sage in the next. Your choices influence how events unfold, but the underlying system is still largely randomised. It's a system that feels hand authored, but isn't.

While Wildermyth is impressive, but it also exposes the limits of procedural storytelling.
Because no matter how clever your templates are, there's a ceiling to its playability.
Procedural generation is deterministic and rule-bound. It can remix, but it can't invent. Eventually, the system starts to repeat itself. As a player, you'll notice the seams and the repetition.

This is where modern generative AI becomes interesting: instead of relying solely on predefined templates, models can fill in gaps, reinterpret context, or expand narrative branches dynamically, giving you a more open-ended storytelling system without losing coherence entirely.

Modern generative AI

When people talk about “AI” today, they're usually referring to two types of models:

Large Language Models (LLMs), such as GPT-style transformers
Diffusion models for image generation

These models are trained on large datasets and learn statistical patterns in text or imagery. Once trained, they can generate new content in the style of what they've seen.

While people often focus on the big models, smaller and more specialised models are increasingly important. Codex for code generation, Stable Diffusion for images, and small LLaMA-class models for focused tasks are all good examples.

Using these models in real systems introduces a different style of programming. Instead of writing deterministic functions, we wrap models with validation and guardrails to coerce their unpredictable outputs into structured formats. For game development, that often means combining generative systems with the kind of tooling we already have for procedural content.

Building a Wildermyth-like system with AI

Inspired by Wildermyth, I started sketching out a system that blended procedural generation with AI-authored content. The goal was not to let the model run wild, but to use it to fill narrative space within a controlled structure.

The idea was to:

Model a world using data structures for narrative elements, character traits, histories and events.
Describe RPG-style dialogue, relationships, and interactions with TypeScript types.
Use concepts from EventSourcing to store world histories, character actions, and place-based records.
Let the LLM generate content within these rails, keeping everything internally consistent.
Pre-generate thousands of possible universes, settings, cities, and characters using automated scripts.
Mix pre-generated content with on-the-fly generation to build a sort of “infinite role-playing game”.

Using generative AI in this way feels more like an evolution of procedural generation rather than an entirely new system.

Taming the randomness

Working with LLMs introduces three major challenges:

1. Context window limitations

Models need enough world information to stay grounded, but you can't exceed their context window - the amount of text that they can process at once. Even large windows are easy to fill when you're generating worlds, cities, and characters.

The solution is a multi-pass hierarchy:

Generate high-level universe types.
Use universe types to generate world summaries and histories.
Generate cities using only world-level summaries.
Generate characters using city context, world summaries, and a short universe description.

Each level is fed only the context it needs. Moving up and down the hierarchy keeps prompts short but coherent.

The structure of a universe

Using types to define the structure of generated content helps keep things consistent. Here's an example of a Universe type that guides high-level world generation. You can be as specific or vague as you like with the parameters, allowing for a wide variety of generated settings and themes.

export interface Universe {
  preferredGenre?: string; // eg "fantasy", "sci-fi", "post-apocalyptic"
  tone?: string; // eg "hopeful", "grimdark", "mythic"
  scaleHint?: string; // eg "single planet", "system", "galaxy"
  techOrMagicBias?: string; // eg "high fantasy", "steampunk gadgets"
  factionCount?: number; // Desired number of factions (cap at 6 in output)
  historyLengthHint?: string; // eg "hundreds of years", "millennia"
  themesAndAestheticsBias?: string[]; // Vibe/style hints, eg ["solarpunk", "gothic"]
  randomSeed?: number; // Deterministic seed for proc-gen
  namingConventions?: {
    // Naming preferences for generated proper nouns
    planetStyle?: string;
    countryStyle?: string;
    cityStyle?: string;
    factionStyle?: string;
  };
}

Then you can prompt the model to fill in the details:

**You are a worldbuilder AI for a proc-gen + gen-AI assisted RPG.**

Accept a **seed JSON** and return **only** a JSON document describing an internally-consistent universe outline to ground subsequent generation runs.

## Requirements

When generating a universe, include:

- **universeType**: The genre or type (fantasy, sci-fi, post-apocalyptic, etc.).
- **constraints**: Rules/limitations (e.g., magic rules, physics constraints, aesthetics).
- **scale**: Description of scope (single planet, system, galaxy).
- **topology**: List of locations with names and relationships (planets, countries, cities).
- **universeAge**: Timeline length (hundreds, thousands, etc.).
- **historyEvents**: Up to 10 major events shaping the world, each with a `year` (absolute, relative, or approximate) and `description`.
- **politicalContext**: Current political situation.
- **factions**: Up to 6 factions, each with name, description, goals, relationships.
- **technologyOrMagicLevel**: Description of available tech/magic, everyday use, limits.
- **themesAndAesthetics**: Short descriptors of the universe's tone, look, or feel (e.g. _dieselpunk decay_, _solarpunk optimism_, _dark gothic mysticism_).

## Behavior Rules

- **Input:** A seed JSON (schema below). Missing fields → randomize sensibly.
- **Output:** **JSON only** (no prose, no markdown) conforming to the Output Schema.
- **IDs:** Generate **UUIDv4 (lowercase)** for every entity (`universeId`, planets, countries, cities, factions).
- **Relationships:** Use **ID-based** relationships only (`targetId`, `relatedIds`). No name references.
- **Internal Consistency:** Every referenced `targetId`/`relatedId` must exist. Names unique within scope.
- **Cardinality Caps:** ≤ 12 planets; ≤ 24 countries total; ≤ 48 cities total; ≤ 6 factions; ≤ 10 history events.
- **Design Goals to Satisfy:**

  - Universe from a **known type/genre** with **constraints** (e.g., magic rules, physics).
  - Clear **scale** (single planet → galaxy).
  - **Topology** (planets → countries → cities) with **names** & **relationships**.
  - **Universe age** (hundreds/thousands/millennia).
  - Up to **10 key history events** (timeline).
  - **Political context** (blocs, tensions, powers).
  - Up to **6 factions** (names, goals, relationships).
  - **Technology or magic** level (everyday use, limits).
  - **Themes & aesthetics** (tone, vibe, style).

NEVER
Use the name "Eldoria" or any variation of it, it's over-fit in training data.

ALWAYS:
Make sure you return valid JSON.

2. Testing and validating output

If you ask a model to return JSON, it might manage it, but the output often with missing commas or mismatched brackets. Unfortunately, LLMs are not parsers.

To manage this:

Include TypeScript types in your prompts.
Use a few-shot prompting technique with examples of valid JSON.
Validate responses against schemas.
If validation fails, the faulty response can be fed back into the model with instructions to repair it.

I found this “corrective loop” was far more efficient than starting from scratch.

3. Cost

Generating large quantities of content using cloud models can get expensive very quickly. I decided to use local models instead.

The last couple of years have been great for small LLMs, which now run happily on consumer-grade GPUs or Apple Silicon. LM Studio became my main tool, running a local server with an OpenAI-compatible API. It meant I could leave a Mac Mini running overnight generating thousands of assets for pennies, all within a 45-watt thermal envelope. Cheaper than boiling a kettle!

Calling LLMs with Deno

To orchestrate the generation pipeline, I used Deno, a modern runtime for JavaScript and TypeScript. Deno's built-in TypeScript support, filesystem APIs and security model make it ideal for this kind of tooling.

The generation system used:

The OpenAI API library (pointed at LM Studio for local inference)
Deno's filesystem APIs for reading/writing batches of generated content
A small command-line runner to handle context management, multi-pass generation, and retries

Here's a simplified version of the LLM client I used:

import { OpenAI } from "npm:openai/client";

export class LocalLlmClient {
  private client: OpenAI;
  private model: string;

  constructor() {
    this.client = new OpenAI({ apiKey: "sk-no-key" });
    this.client.baseURL = "http://localhost:1234/v1"; // LM Studio local endpoint
    this.model = "openai/gpt-oss-20b";
  }

  async generate(prompt: string): Promise<string> {
    const res = await this.client.chat.completions.create({
      model: this.model,
      temperature: 2,
      messages: [
        {
          role: "system",
          content: "You're a content generator for narrative stories. You're trying to make
                   original, creative, and interesting content, worlds, and characters. You
                   respond in perfectly formed JSON whenever requested.",
        },
        { role: "user", content: prompt },
      ],
    });

    return res.choices?.[0]?.message?.content || "";
  }
}

This script orchestrated universe → world → city → character → dialogue generation using the structured multi-pass approach described earlier.

Testing the conversation system

One of the easiest ways to test a narrative system is to prototype the conversation layer. I wanted something that felt like a classic “Bioware NPC conversation”, with a branching dialogue structure:

Conversations are represented as dialogue trees. Each node on the tree contains text, conditions, outcomes, and links to other nodes.
Nodes can modify sentiment values, or unlock knowledge tags, and could be expanded later to trigger quests or events or drop items.

I built a small React app and loaded a random pre-generated conversation from disk through a simple API.

The API was built with Hono and ran with Deno Serve, making it lightweight and easy to deploy locally:

import { Hono } from "hono";
import { cors } from "hono/cors";

const universeFiles = [...Deno.readDirSync("../author/output/")].filter((f) => f.name.endsWith("_universe.json"));

const app = new Hono();
app.use("/random", cors());
app.get("/random", (c) => {
  const random = universeFiles[Math.floor(Math.random() * universeFiles.length)];
  const uuid = random.name.split("_")[0];
  const universe = JSON.parse(Deno.readTextFileSync(`../author/output/${uuid}_universe.json`));
  const dialogue = JSON.parse(Deno.readTextFileSync(`../author/output/${uuid}_dialogue.json`));
  const flavour = JSON.parse(Deno.readTextFileSync(`../author/output/${uuid}_flavour.json`));

  return c.json({ id: uuid, universe, flavour, dialogue });
});

if (import.meta.main) Deno.serve(app.fetch);

A basic UI allowed me to step through the dialogue tree, making choices and seeing how the conversation evolved.

// Minimal React demo wired to the Hono /random endpoint
import { useEffect, useState } from "react";

type Sentiment = number;

type Condition = {
  minSentiment?: Sentiment;
  maxSentiment?: Sentiment;
  flagsAll?: string[];
  flagsAny?: string[];
};

type Effects = {
  sentimentDelta?: Sentiment;
  setFlags?: string[];
  clearFlags?: string[];
};

type Outcome = { type: "reveal" | "end" | "conflict" | "leave" | "reward"; data?: unknown };

type DialogueOption = {
  id: string;
  text: string;
  npcResponse: string;
  next: string | null;
  effects?: Effects;
  condition?: Condition;
  outcome?: Outcome;
};

type DialogueNode = {
  id: string;
  text: string;
  options: DialogueOption[];
  terminal?: boolean;
  outcome?: Outcome;
};

type DialogueTree = {
  id: string;
  title: string;
  npcName: string;
  startNodeId: string;
  initialSentiment: Sentiment;
  nodes: Record<string, DialogueNode>;
};

type StoryData = {
  id: string;
  universe: Record<string, unknown>;
  flavour: { flavourText: string } | null;
  dialogue: DialogueTree;
};

export function ConversationDemo() {
  const [data, setData] = useState<StoryData | null>(null);
  const [currentId, setCurrentId] = useState<string | null>(null);
  const [sentiment, setSentiment] = useState<Sentiment>(0);
  const [flags, setFlags] = useState<Set<string>>(new Set());
  const [ended, setEnded] = useState(false);
  const [log, setLog] = useState<string[]>([]);

  const load = async () => {
    const res = await fetch("http://localhost:8000/random");
    const json: StoryData = await res.json();
    setData(json);
    setCurrentId(json.dialogue.startNodeId);
    setSentiment(json.dialogue.initialSentiment);
    setFlags(new Set());
    setEnded(false);
    setLog(json.flavour?.flavourText ? [json.flavour.flavourText] : []);
  };

  useEffect(() => {
    load();
  }, []);

  const passes = (cond?: Condition): boolean => {
    if (!cond) return true;
    if (cond.minSentiment !== undefined && sentiment < cond.minSentiment) return false;
    if (cond.maxSentiment !== undefined && sentiment > cond.maxSentiment) return false;
    if (cond.flagsAll && !cond.flagsAll.every((f) => flags.has(f))) return false;
    if (cond.flagsAny && !cond.flagsAny.some((f) => flags.has(f))) return false;
    return true;
  };

  const choose = (opt: DialogueOption) => {
    if (!data) return;

    // apply effects
    if (opt.effects?.sentimentDelta) setSentiment((s) => s + (opt.effects!.sentimentDelta || 0));
    if (opt.effects?.setFlags?.length) setFlags((prev) => new Set([...prev, ...opt.effects!.setFlags!]));
    if (opt.effects?.clearFlags?.length)
      setFlags((prev) => {
        const next = new Set(prev);
        opt.effects!.clearFlags!.forEach((f) => next.delete(f));
        return next;
      });

    // echo player's choice and NPC response
    setLog((l) => [...l, `You: ${opt.text}`, `${data.dialogue.npcName}: ${opt.npcResponse}`]);

    if (opt.next) {
      setCurrentId(opt.next);
      const nextNode = data.dialogue.nodes[opt.next];
      if (nextNode?.terminal || nextNode?.options.length === 0) setEnded(true);
    } else {
      setEnded(true);
    }
  };

  if (!data || !currentId) return <div>Loading…</div>;

  const node = data.dialogue.nodes[currentId];
  const available = node.options.filter((o) => passes(o.condition));

  return (
    <div style={{ maxWidth: 720, margin: "0 auto", fontFamily: "system-ui, sans-serif" }}>
      <h3>
        {data.dialogue.title} — {data.dialogue.npcName}
      </h3>

      <div style={{ padding: "12px", background: "#111", color: "#eee", borderRadius: 8, marginBottom: 12 }}>
        <div style={{ opacity: 0.8, marginBottom: 8 }}>
          {log.map((l, i) => (
            <div key={i}>{l}</div>
          ))}
        </div>
        <div>
          <strong>{data.dialogue.npcName}:</strong> {node.text}
        </div>
      </div>

      {!ended ? (
        <ul style={{ listStyle: "none", padding: 0, display: "grid", gap: 8 }}>
          {available.map((opt) => (
            <li key={opt.id}>
              <button onClick={() => choose(opt)} style={{ width: "100%", textAlign: "left", padding: "10px 12px", borderRadius: 6 }}>
                {opt.text}
              </button>
            </li>
          ))}
        </ul>
      ) : (
        <div style={{ margin: "12px 0" }}>
          Conversation ended.
          <div style={{ marginTop: 8 }}>
            <button onClick={load}>Start another</button>
          </div>
        </div>
      )}

      <div style={{ marginTop: 16, fontSize: 12, opacity: 0.7 }}>Sentiment: {sentiment}</div>
    </div>
  );
}

This setup made it easy to iterate on the feel of the conversations before committing to building an entire game.

Some fun extras

To make the prototype feel more game-like, I added a couple of small procedural touches.

Character avatars

Each NPC was rendered using seeded procedural SVGs. Different shapes, colours, and layers combined to give characters a distinct look determined entirely by their name and role. Universes themed around sci-fi, fantasy, or modern settings got different palettes or accessories. If an NPC disliked you, their eyebrows and expressions shifted accordingly.

World maps

The worlds themselves used a tiny cellular automata system. A miniature run of Conway's Game of Life generated landmasses. Cities were placed according to simple rules. The map was then blurred and tinted to make it look like a globe or parchment map depending on the universe type.

The key point: everything was seeded, so the same input always produced the same output. Consistency is crucial when you want a coherent game world.

Applications outside games

The techniques discussed here aren't limited to games development. Wrapping non-deterministic model outputs in structured systems has wider use:

Systems where outputs must conform to strict data types, such as APIs or data pipelines.
Workflows where known data sources must be preserved while adding human-like variation, like content personalization.
Processes that benefit from some imaginative expansion while staying rooted in constraints, such as automated report generation or creative writing aids.

Games are just particularly good testing ground. They've always balanced authored content, procedural randomness, and systemic behaviour. Generative AI is simply another tool that can sit alongside our other tools, provided it's wrapped in the right guardrails.

If anything, the future will probably look like a blend of human authorship, procedural logic, and model-assisted generation. Not to replace writers or designers, but to allow small teams to produce richer worlds, in the same way that previous generations of tooling levelled the playing field in the 90s and 2000s.

The challenge is keeping human creativity at the centre. The technology should extend what's possible, not erase the artistry and human element that makes games enjoyable in the first place.

LLMs obviously still have their flaws

You may have noticed in the earlier prompt there was a rule to NEVER Use the name "Eldoria" or any variation of it, it's over-fit in training data. This is because if you ask any of the most popular LLMs to create and describe a word, it has an over-eagerness to tell you about "Eldoria". When I was generating worlds I found Eldoria repeated often in the data. Try it out yourself in an online LLM, you may find you also get this name or something similar, in fact there is an entire reddit conversation where others have found the same thing. Perhaps one day we will discover Eldoria, the land where AI dreams to be.

I fell foul of plenty of issues with JSON repair loops failing, generated dialog that went nowhere or confused the speakers, and occasionally contradicting characters and world outlines. LLMs are of course no match, nor replacement for human writers and creatives, but there is still plenty of room for improvement of the prompts and for a human editor to come in and make something from the thousands of generated concepts.

Let me know what you think

All in all, this was an interesting experiment to see how far a single developer could push narrative generation with a hybrid deterministic and generative pipeline and I'm excited to work on it more.

Thanks for reading this far! I hope this peek into my experiments helps spark your own ideas about blending procedural generation with modern LLMs. If you’ve been working with similar systems or have your own lessons learned, I’d love to hear about them! Drop me a comment down below or drop me a message, I'm @thisisjofrank on all the socials.

How to convert CommonJS to ESM

Jo Franchetti — Tue, 22 Oct 2024 17:46:36 +0000

ECMAScript modules (”ESM”) are the official, modern way of writing and sharing JavaScript — it’s supported in many environments (e.g. browsers, the edge, and modern runtimes like Deno, and offers a better development experience (e.g. async loading and being able to export without globals). While CommonJS was the standard for many years, supporting CommonJS today is hurting the JavaScript community.

All new JavaScript should be written in ESM for future proofing. However, there are many cases where a legacy code base needs to be modernized for compatibility reasons with newer packages. In this blog post, we’ll show you how to migrate the syntax of a legacy CommonJS project to one that supports ESM and tools to help smooth out that process.

In this document

Module imports and exports
Update package.json
Other changes
Tools for migrating
What's next

Module imports and exports

Here’s how you can update import and export syntax from CommonJS to ESM.

On the export side:

- function addNumbers(num1, num2) {
+ export function addNumbers(num1, num2) {
  return num1 + num2;
};

- module.exports = {
-   addNumbers,
- }

On the import side:

- const { addNumbers } = require("./add_numbers");
+ import { addNumbers } from "./add_numbers.js");

console.log(addNumbers(2, 2));

Note that in ESM, the file extension must be included in the module path. Fully specified imports reduce ambiguity by ensuring the correct file is always imported by the module resolution process. Plus, it aligns with how browsers handle module imports, making it easier to write isomorphic code that’s predictable and maintainable.

What about conditional imports? If you are using Node.js v14.8 or later (or Deno), then you’ll have access to top-level await, which you can use to make import synchronous:

- const module = boolean ? require("module1") : require("module2");
+ const module = await (boolean ? import("module1") : import("module2"));

Update `package.json`

If you’re using package.json, you’ll need to make a few adjustments to support ESM:

{
  "name": "my-project",
+ "type": "module",
- "main": "index.js",
+ "exports": "./index.js",
  // ...
}

Note the leading "./" in ESM is necessary as every reference has to use the full pathname, including directory and file extension.

Also, both "main" and "exports" define entry points for a project. However, "exports" is a modern alternative to "main" in that it gives authors the ability to clearly define the public interface for their package by allowing multiple entry points, supporting conditional entry resolution between environments, and preventing other entry points outside of those defined in "exports".

{
  "name": "my-project",
  "type": "module",
  "exports": {
    ".": "./index.js",
    "./other": "./other.js"
  }
}

Finally, another way to tell Node to run the file in ESM is to use the .mjs file extension. This is great if you want to update a single file to ESM. But if your goal is to convert your entire code base, it’s easier to update the type in your package.json.

Other changes

Since JavaScript inside an ESM will automatically run in strict mode, you can remove all instances of "use strict"; from your code base:

- "use strict";

CommonJS also supported a handful of built-in globals that do not exist in ESM, such as __dirname and __filename. One simple way to get around that is to use a quick shim to populate those values:

// Node 20.11.0+, Deno 1.40.0+
const __dirname = import.meta.dirname;
const __filename = import.meta.filename;

// Previously
const __dirname = new URL(".", import.meta.url).pathname;

import { fileURLToPath } from "node:url";
const __filename = fileURLToPath(import.meta.url);

Tools for migrating

While the above touches upon the changes necessary to convert a CommonJS code base to an ESM one, there are a few tools to help with that transition.

With VSCode, you can quickly convert all import and export statements from CommonJS to ESM. Simply hover over the require keyword, hit “quick fix”, and all of those statements in that file will be updated to ESM:

You’ll notice that VSCode can swap out the proper keywords for importing and exporting, but the specifiers are missing filename extensions. If you have Deno installed, you can quickly add them by running deno lint --fix. Deno’s linter comes with a no-sloppy-imports rule that will show a linting error when an import path doesn’t contain the file extension.

For a more end-to-end approach to converting CommonJS to ESM, there are a few transpilation options. There are npm packages cjs2esm and cjstoesm, as well as the Babel plugin babel-plugin-transform-commonjs. However, these tools may not be actively maintained and are not feature complete, so keep that in mind when evaluating them.

What’s next

ESM is the standard JavaScript way to share code and all new JavaScript should support it. Choosing to support CommonJS today can be extremely painful for module authors and developers who don’t want to troubleshoot legacy compatibility issues. JSR - the open source, modern JavaScript registry, explicitly forbids modules using CommonJS. Everyone can do their part in levelling up the JavaScript ecosystem!

A Gentle Intro to TypeScript

Jo Franchetti — Thu, 11 Jul 2024 10:55:33 +0000

For many who came to programming via JavaScript it is easy to fall in love with its low barrier to entry and versatile nature. JavaScript runs in a browser, can be written in notepad, is interpreted line by line and requires no complicated compilation or tooling. JavaScript has democratized software development by allowing developers from all backgrounds to pick it up and start coding. But with Javascript’s forgiving nature, comes increased chances to make mistakes and create bugs.

Consider this JavaScript program:

function add(a, b) {
  return a + b;
}

console.log(add(1, 2)); // 3

This is a simple addition program that is designed to take two numbers and add them together, returning the result. Unexpected, often unpredictable things can start happening when we call this function with values that aren’t numbers.

We only really want to call this function with numbers - it doesn't make sense otherwise - and in fact, if you pass values to it that are not numbers, you'll end up with often bizarre and unpredictable results:

console.log(add(1, "2")); // 12
console.log(add(1, true)); // 2
console.log(add(1, "hello")); // 1hello

JavaScript is a dynamically typed language. This means that the types of data that we store in our variables can change at runtime. This can make it complicated to capture our intention in our JavaScript code - that our add function should only be called with numbers.

What we are seeing happen here is called type coercion - JavaScript automatically converts values from one data type to another. This can happen explicitly, through the use of functions and operators, or implicitly, when JavaScript expects a certain type of value in a particular context. Implicit type coercion can sometimes lead to unexpected results, especially in complex expressions. Here are a few cases:

console.log(1 + "2"); // "12" (number 1 is converted to string)
console.log("2" + 1); // "21" (number 1 is converted to string)

console.log("5" - 1); // 4 (string "5" is converted to number)
console.log("5" * "2"); // 10 (both strings are converted to numbers)

console.log(0 == false); // true (number 0 is converted to false)
console.log(0 === false); // false (no type coercion, different types)

Confusing right?!

In a JavaScript codebase, the best we can do is add some guard checks to our program that will throw errors if invalid values are provided. The problem with having only these runtime checks to protect us from errors is that we'll often find out when a bug has been introduced into our code at the same time that our users do. So how do we protect ourselves from this often confusing JavaScript behavior?

TypeScript to the Rescue

TypeScript can help us to describe the intention of our code by being explicit about what types we expect where. TypeScript is a superset of JavaScript that adds additional type information to the language. When you compile TypeScript, JavaScript is produced that can run anywhere, so it's really easy to incrementally use it to make your development experience better without having to rebuild all of your software.

Types allow you to catch errors in your code before it runs. If you accidentally assign a value of the wrong type to a variable, you’ll get a compile error. Types also make your code easier to read because you can explicitly state what kind of value you expect. We can also use tools to make types even more powerful. Code editors and IDEs have inbuilt tools to help autocomplete your code as you write when you’re using types correctly.

How do we add type annotations?

You add a type annotation in TypeScript by appending a colon (:) followed by the desired type after the function parameter's name. If we were to extend our previous add example to convert it to TypeScript it would look like this:

function add(x: number, y: number) {
  return x + y;
}

console.log(add(1, 2)); // 3

This is the simplest change we can make to our code to make it more robust. Now the compiler knows that any code that attempts to call add() with anything but two numbers will fail at runtime, so it will throw an error upon compilation to tell you your program isn't valid.

If we try and call add with a number and a string, for example, we’ll get a compiler error:

TypeScript is smart enough to work out some of the types in your program for you based on the information you've given it, we call this "type inference". If we take the above example and expand it out, adding all the type annotations that we can, it'd look like this:

function add(x: number, y: number): number {
  return x + y;
}

const result: number = add(1, 1);
console.log(result);

Here we've explicitly added annotations for the function parameters, a return type of the add function, and the variable result. As a general rule, developers add "just enough" type annotations to tell the TypeScript compiler what's going on, and let it infer the rest. In our original example, by adding type annotations to the x and y parameters, the TypeScript compiler could inspect the code and realize we only ever add two numbers, so would infer both the function return type, and the type of the variable result to be of type number.

Even if you only ever use TypeScript to annotate function parameters you'll immediately remove an entire category of errors from your code.

Turning your TypeScript back into JavaScript

If your project is built using Node, you will need to add the typescript package and run the tsc compiler tool. We’ve written an introduction to configuring your TypeScript compiler.

Deno comes with the TypeScript compiler built right in, so if you're using Deno, you do not need any other configuration or tools. Deno supports TypeScript out of the box, automatically turning TypeScript into JavaScript as we execute your source code.

On the client side, you can use Vite, a tool after our own heart, which does a similar transparent compilation for you, so if you're a front end developer, you can still get TypeScript joy in your code.

Next up

In our next Deno Bite we'll talk about common types that you’ll need in your TS code and how to use them to build more complex types to make your code clear and bug-free!

Are there any topics on TypeScript you would like me to cover? Let me know in the comments or on Twitter or Discord!

How to document your JavaScript package

Jo Franchetti — Fri, 17 May 2024 16:22:09 +0000

Creating and publishing open source packages is a great way to contribute to the ecosystem and community. You made something cool and want people to use it. But simply publishing your module to a registry and crossing your fingers won't get users. Helping your users become successful with your package means not only writing concise, descriptive documentation, but also ensuring your users can access the documentation within their workflows (e.g. in VSCode) to save them time.

Thanks to JSDoc it's easy to write documentation that is coupled with your code and can be consumed by users in a variety of formats. When combined with a modern publishing flow like JSR, you can easily create comprehensive documentation for your package that not only fits within your workflow, but also integrates directly in the tools your users consume your package with. This blog post aims to cover best practices when writing JSDoc-style comments to get your users up and running as quickly as possible:

Why JSDoc?
A brief intro to JSDoc
Provide good type information
Tags, tags, tags
Add examples
But what should I document?
Use markdown
Link internally
Keep JSDoc up-to-date
Audit your JSDoc
What's next

Why JSDoc?

While a good README answers "why should I use your package?", good documentation should answer "how can I use your package?". Users browsing your documentation have a problem they need to solve, and your documentation should provide them with the answer in the fewest clicks and keyboard taps.

JSDoc is a great way to write reference documentation that is coupled with the code itself and can be consumed by users in a variety of formats, such as HTML, markdown, JSON, or in their IDE or text editor. Here is a quick diagram of an example JSDoc-style comment and how it appears as documentation in various mediums:

When you write JSDoc-style comments in your code and publish to JSR, it will appear formatted on your package's documentation page on JSR, VSCode tooltips and auto-complete, and in `deno doc` output.

Writing good JSDoc can improve the success of your package. Before we dive into some best practices, here's a brief high-level introduction to JSDoc.

A brief intro to JSDoc

JSDoc turns your comments in your code into a documentation object that can be rendered and displayed in a variety of formats.

JSDoc comments are any block comments that begin with /** and end with */ that precede a block of code. Here's an example:

/** Adds two values and returns the sum. */
function sum(value1, value2) {
  return value1 + value2;
}

This JSDoc will then appear as a tooltip in your IDE:

JSDoc comments can span multiple lines. Each line should start with * and should be indented by one space.

/**
 * Adds two values and returns the sum.
 *
 * NOTE: JavaScript math uses IEEE 754 floating point arithmetic, so there may
 * be some rounding errors when adding two numbers.
 */
function sum(value1, value2) {
  return value1 + value2;
}

The first paragraph of a JSDoc comment is the most important. It is a summary of the symbol and is shown in tooltips, auto-completions in your editor, and is indexed by search. The first paragraph should be a concise description of the symbol, and should be written in a way that helps users quickly understand what this function does.
For example, don't write:

/**
 * This function takes a string in the first and returns a string. It looks for
 * all the spaces in the input string using a regexp, and then replaces them one
 * by one with an underscore. The function then returns the modified string.
 */
function replaceSpacesWithUnderscores(value) {
  return value.replace(/ /g, "_");
}

Instead, concisely describe what the function does:

/**
 * Replaces all spaces in a string with underscores.
 */
function replaceSpacesWithUnderscores(value) {
  return value.replace(/ /g, "_");
}

Additional information like the implementation details, caveats, or examples should be added in subsequent paragraphs. Because JSDoc supports markdown, you can even use headings to separate different sections.

Simple and concise summaries help users quickly filter through a list of symbols during auto-complete and find the one they need. Once they've found the symbol, they can read through the rest to learn about the details.

Provide good type information

After the succinct descriptive summary, it's important to provide good type information for the symbols you are exposing in your package. This serves two main purposes:

It allows auto-completion on parameters and return values in your editor, because the editor knows the types of the parameters and return values. It helps users quickly filter through the list of functions to find the one they need. For instance, if they are looking for a function that combines two strings, they can filter out functions that don't take two strings as parameters.

Here, we'll use TypeScript to add type information. TypeScript, one of the fastest growing programming languages, is a strongly typed language built on JavaScript that enhances code quality and maintainability, while improving developer productivity.

/**
 * Adds two values and returns the sum.
 */
export function sum(value1: number, value2: number): number {
  return value1 + value2;
}

In your editor, you'll see the type information for the parameters and return value when you hover over the function:

When a user types sum( in their editor, they'll see the type information for the parameters:

On the return value, you can immediately get completion for methods on the returned number:

Tags, tags, tags

JSDoc supports a variety of tags that can be used to provide additional information about your symbols, such as @param for parameters, @returns for the return value, or @typeParam for type parameters. Here's an example of a function with type information and tags:

/**
 * Find a substring in a string and return the index of the first occurrence.
 *
 * @param value The string that will be searched for the needle.
 * @param needle The substring to search for in the string.
 * @returns The index of the first occurrence of the needle in the value, or -1 if the needle is not found.
 */
declare function find(value: string, needle: string): number;

In your editor, you'll see the type information for the parameters and return value, as well as the additional information provided by the tags:

On JSR, the tags are rendered in HTML. Here's an example of the @param and @return tags in the JSDoc of the
move function in deno_std/fs:

Add examples to JSDoc

Examples are another great way to help users quickly understand how to use your library. This is especially useful for functions that have complex behavior or many parameters. Examples can be added to your JSDoc comments using the @example tag:

/**
 * Find a substring in a string and return the index of the first occurrence.
 *
 * @example Find a substring in a string
 * ```

ts
 * const value = "hello world";
 * const needle = "world";
 * const index = find(value, needle); // 6
 *

@example Find a substring in a string that doesn't exist
``` ts
const value = "hello world";
const needle = "foo";
const index = find(value, needle); // -1 * */ declare function find(value: string, needle: string): number;

The best examples are concise and demonstrate the most common use cases of your function. They should be easy to understand and can be copied and pasted into a
project.

You can even provide multiple examples if there are multiple use cases worth mentioning. Here's an example of how multiple examples appear on JSR from the move function of deno_std/fs:

/**
 * (truncated for brevity)
 * @example Basic usage
 * ```

ts
 * import { move } from "@std/fs/move";
 *
 * await move("./foo", "./bar");
 *

This will move the file or directory at ./foo to ./bar without
overwriting. *
@example Overwriting
``` ts
import { move } from "@std/fs/move"; *
await move("./foo", "./bar", { overwrite: true }); * * * This will move the file or directory at `./foo` to `./bar`, overwriting * `./bar` if it already exists. */

Note the text immediately following @example serves as the title, and the text beneath the example becomes its description on JSR:

But what should I document?

You should document every symbol your package exports, including functions, classes, interfaces, and type aliases.

This extends beyond just one JSDoc comment per symbol. For classes and interfaces for example, you should document the symbol itself, each method or property on it, including constructors. Here's an example of an Oak interface with JSDoc comments on its properties:

/** Base interface for application listening options. */
export interface ListenOptionsBase {
  /** The port to listen on. If not specified, defaults to `0`, which allows the
   * operating system to determine the value. */
  port?: number;
  /** A literal IP address or host name that can be resolved to an IP address.
   * If not specified, defaults to `0.0.0.0`.
   *
   * __Note about `0.0.0.0`__ While listening `0.0.0.0` works on all platforms,
   * the browsers on Windows don't work with the address `0.0.0.0`.
   * You should show the message like `server running on localhost:8080` instead of
   * `server running on 0.0.0.0:8080` if your program supports Windows. */
  hostname?: string;
  secure?: false;
  /** An optional abort signal which can be used to close the listener. */
  signal?: AbortSignal;
}

This is
how the JSDoc comments on properties appear on JSR:

If your package consists of multiple modules, adding a JSDoc comment to the top of each module file with the @module tag will be helpful. This module comment should include a description and examples of how to use its exported symbols.

Here's an example of @module in Oak's
application.ts file:

/**
 * Contains the core concept of oak, the middleware application. Typical usage
 * is the creation of an application instance, registration of middleware, and
 * then starting to listen for requests.
 *
 * # Example
 *
 * ```

ts
 * import { Application } from "jsr:@oak/oak@14/application";
 *
 * const app = new Application();
 * app.use((ctx) => {
 *   ctx.response.body = "hello world!";
 * });
 *
 * app.listen({ port: 8080 });
 *

@module */ ```

In JSR, the first paragraph becomes the description beneath the modules on the main docs page of your package:

Note the main doc page only includes the first paragraph. The subsequent JSDoc comment appears when you click through to the module page:

Use markdown for a better documentation experience

Using markdown in JSDoc lets you organize your documentation in a more readable and engaging way. This can help you create documentation that is easier to understand, and allows you to link to external resources or other parts of your documentation using links.

Some useful markdown features you can use in your JSDoc comments include:

# my heading for section headings
- hello world for bullet points
**important** for emphasis
_noteworthy_ for italics
> quote for block quotes
[foo](https://example.com) for links
`console.log("foo")` for inline code snippets

On JSR, you can also use [!IMPORTANT] to highlight important information in your documentation that you want to draw attention to.

// Copyright 2018-2024 the oak authors. All rights reserved. MIT license.

/** Middleware that converts the oak specific context to a Fetch API standard
 * {@linkcode Request} and {@linkcode Response} along with a modified context
 * providing some of the oak functionality. This is intended to make it easier
 * to adapt code to work with oak.
 *
 * There are two functions which will "wrap" a handler that operates off a
 * Fetch API request and response and return an oak middleware. The
 * {@linkcode serve} is designed for using with the {@linkcode Application}
 * `.use()` method, while {@linkcode route} is designed for using with the
 * {@linkcode Router}.
 *
 * > [!IMPORTANT]
 * > This is not intended for advanced use cases that are supported by oak,
 * > like integrated cookie management, web sockets and server sent events.
 * >
 * > Also, these are designed to be very deterministic request/response handlers
 * > versus a more nuanced middleware stack which allows advanced control.
 * > Therefore there is no `next()`.
 * >
 * > For these advanced use cases, create middleware without the wrapper.
 *
 * @module
 */

This module-level JSDoc comment will appear at the top level in JSR as such:

Link internally to other parts of your documentation

Sometimes, your documentation refers to another symbol within your package. To make it easy for your users to navigate throughout your docs, you can link
within your documentation using the @link , @linkcode , and @linkplain tags. These tags accept a name paths or URL, from which it generates an HTML
anchor element. Here's an example:

/** Options to use when styling text with the {@linkcode print} function. */
export interface StyleOptions {
  /** The color to print the message in. */
  color: "black" | "red" | "green";
  /** Whether to print the message in bold. */
  bold: boolean;
  /** Whether to print the message in italic. */
  italic: boolean;
}

/**
 * A function that prints a message to the terminal with the given options.
 *
 * Note that on some versions of Windows, {@linkcode StyleOptions.color} may not
 * be supported in combination with {@linkcode StyleOptions.bold}.
 */
declare function print(message: string, options: StyleOptions): void;

In VSCode, the hover tooltip now includes clickable links that will take you directly to the code where that symbol is defined:

Here's an example of how @linkcode appears in JSR. In the JSDoc for Oak's serve function, it references Application, which is becomes a clickable link on JSR:

You can also reference built-in JavaScript objects, like ArrayBuffer, and JSR will automatically link to the relevant MDN documentation.

Keep JSDoc up-to-date with code changes

One benefit about using JSDoc is that writing documentation in the comments happens at the same time as writing the code. That means anytime we need to make a change to a function, interface, or module, we can make the necessary change to the JSDoc with minimal context switching costs.

But how does one make sure that the docs in the comments are updated? Starting with docs, á la docs-driven-development, can help you spec out and reason about the requirements before writing a single line of code. Sometimes, this means catching potential problems earlier and saving time from having to re-write code. (For a more macro-level approach to this, check out "Readme-driven development".)

If you've included code examples in your documentation, you can type check them from the command line with deno test --doc. This is a helpful tool to ensure that your examples within your documentation are up-to-date and working.

For example, building on our sum function from earlier, let's add an example with a code snippet:

/**
 * Adds two values and returns the sum.
 *
 * @example
 * ```

ts
 * import { sum } from "jsr:@deno/sum";
 * const finalValue = sum(1, "this is a string"); // 3
 *

*/
export function sum(value1: number, value2: number): number {
return value1 + value2;
}




Then, we can check the code in this example block by running `deno test --doc` :



```jsx
deno test --doc
Check file:///Users/sum.ts$8-13.ts
error: TS2345 [ERROR]: Argument of type 'string' is not assignable to parameter of type 'number'.
const finalValue = sum(1, "this is a string");
                          ~~~~~~~
    at file:///Users/main.ts$8-13.ts:2:27

Whoops! After we fix the type error in the code in our documentation:

deno test --doc
Check file:///Users/sum.ts$8-13.ts

ok | 0 passed | 0 failed (0ms)

This offers a quick way to type check your code examples in your documentation before publishing.

Audit your JSDoc

If you're publishing to JSR, it will handle all of the formatting and generation of documentation based off your JSDoc-style comments. However, if you're interested in using your tooling to audit or test what the JSDoc comment output looks like, here are some suggestions.

deno doc <file>: This Deno command will print the JSDoc documentation for each of thefile's exported members. This command also accepts an--htmlflag that generates a static site with documentation, and a --json flag to generate JSON output that you can display yourself.
deno doc --lint`: This command will check for problems, such as missing return type or missing JSDoc comment on a public type. These lints help you write better documentation and catch potential issues before you publish.
deno test --doc`: We mentioned this command earlier in this post, but this allows you to easily type check your documentation examples.
jsdoc <directory>: JSDoc's own CLI can generate a static documentation site with its default template, and offers a variety of configuration option flags. If the default template is a bit boring, there are others such as docdash, which provides hierarchical navigation and syntax highlighting.

What's next?

Writing good JSDocs for your JavaScript package is critical to its success.
Let's recap the best practices:

Write a concise summary: The first paragraph of your JSDoc comment should be a concise description of the symbol that helps users quickly understand what it does.
Provide good type information: Type information helps users quickly filter through the list of functions and find the one they need.
Use tags: Tags like @param, @returns, and @typeParam provide more information about specific parts of your function or class.
Add examples: Examples help users quickly understand how to use your library.
Document everything: Document every symbol you are exposing in your package, including whole modules if you expose multiple modules.
Link internally: Use @link, @linkcode, and @linkplain to link to other parts of your documentation to help users navigate your docs.
Test your documentation: Use deno test --doc to type check your documentation examples before publishing, and deno doc --lint to check for issues in your JSDoc comments.

By following these best practices, you can create comprehensive documentation for your package that helps users get up and running with your package as quickly as possible.

🚨️ Read more about JSR 🚨️

JSR.io

JSR announcement

Intro to JSR

How we built JSR

JSR is not another package manager

An intro to TSConfig for JavaScript Developers

Jo Franchetti — Tue, 23 Apr 2024 17:16:39 +0000

JavaScript is constantly evolving, from its roots as a simple scripting language into a robust, modern tool for building complex applications. To manage larger, complicated code bases, JavaScript developers are constantly looking for ways to improve their workflows, their code quality and productivity. TypeScript is a major innovation towards improving code quality and maintenance by adding types, so it’s no surprise that
it’s one of the fastest growing languages.

TypeScript can feel scary if you've never used a compiled language or a compiler before. Or maybe you have, and you encountered a complex tsconfig.json file that you didn’t completely understand. This blog post is an introduction to TypeScript (TS) and how to configure your project to use TypeScript with ease:

From JS to TS
TSConfig settings
- compilerOptions
- Other TSConfig settings
Additional features and capabilities of TSConfig
What’s next

🚨️ Want to write and run TypeScript without any config? 🚨️

Deno natively supports TypeScript.
Simply create a .ts file and run deno run yourfile.ts.

From JS to TS

TypeScript is built on top of JavaScript. It's a superset — any valid JavaScript is valid TypeScript. If you're new to TypeScript, it's easy to think of it as a "super powered linter", adding new features to the language to help you write JavaScript safely. It is designed to be strictly additive — TypeScript with the types stripped out is just JavaScript, but with types present, you get a much improved tooling, debugging and general developer experience.

Because the JavaScript ecosystem has grown organically over time, TypeScript aims to fit in with your existing tooling. Modern editors, build tools, package managers, testing frameworks and CI/CD tools all integrate with TypeScript. In order to adopt TypeScript, and tailor it to your specific project requirements and tools, you will need to configure the TypeScript compiler. This can be done using a file called tsconfig.json.

If you're doing TypeScript for the first time in a new codebase, you will probably leave most of the options in tsconfig.json as default. For projects with tooling that needs interop, or has particular quirks, tsconfig.json offers all the leavers you may need to pull to interact with your ecosystem.

TSConfig settings

The tsconfig.json file allows you to configure how the TypeScript compiler processes your TypeScript code. The tsconfig.json file is just a JSON object with properties that define your compiler options and project settings. We’ll go through some of the properties you’re likely to need when setting up your own tsconfig.json file.

The first property to take a look at is compilerOptions, where you specify compiler settings.

Compiler settings in `compilerOptions`

The compilerOptions property is where you define TypeScript compiler settings.
These options include -

target - Specifies the ECMAScript target version for the emitted JavaScript. Defaults to ES3. To ensure maximum compatibility, set this to the lowest version that your code requires to run. ESNext setting allows you to target the latest supported proposed features.

module - Defines the module system to use (CommonJS, AMD, ES6, etc.). Usage depends on your project’s requirements and the environment that your code will run in. Most modern projects will use ES6 or ESNext.

outDir - Specifies the output directory for compiled JavaScript files. Usually set to dist to create a dist directory for your compiled files.

strict - Enables strict type-checking options to help catch errors in your code. Set to true for strict type checking.

alwaysStrict - Automatically set to true if strict is enabled, this parses the code in JavaScript strict mode and emits use strict for each source file.

esModuleInterop - In JavaScript, there are two main module systems: ECMAScript modules (ESM) and CommonJS modules (CJS). They have different syntax and semantics for imports and exports. When working in a TypeScript project that uses both ESM and CJS modules, setting esModuleInterop to true ensures that TypeScript handles imports and exports in a way that is compatible with both module systems. This is recommended if you are working with third party libraries that use both CJS and ESM.

lib - Specifies the libraries to include when type-checking. TypeScript includes type declarations (type definitions) for JavaScript built-in objects, such as Array, String, Promise, etc. These declarations define the shape and behavior of these objects, allowing TypeScript to provide accurate type checking and IntelliSense support. By default, TypeScript includes a standard set of library declarations (dom, es5, es6, etc.) based on the ECMAScript version targeted by your project. However, you can customize the included libraries using the "lib" option to match your project's environment more precisely. For example, if you're targeting only Node.js, you might exclude browser-specific declarations like dom.

sourceMap - Generates source map files (.map) to aid debugging. Source maps are files that map the generated JavaScript code back to its original TypeScript source code. When using debugging tools, source maps allow you to set breakpoints, inspect variables, and step through your TypeScript code as if you were debugging the original TypeScript source. Set to true to enable the use of source maps.

Other settings that may be useful:

jsx - If you are using JSX (for example with React), this setting determines how your JSX files should be treated(preserve, react, react-native, etc.).

removeComments - Strips comments from your compiled code. Helpful if minifying your compiled code.

sourceRoot - Specifies the location where your debugger should locate TypeScript files instead of source locations. Use this flag if the sources located at run-time are in a different location than that at design-time. The location specified will be embedded in the source map to direct your debugger.

Other TSConfig settings

include - specifies an array of file paths or glob patterns that TypeScript should include in the compilation process. Only files matching the specified patterns will be considered for compilation. You can use glob patterns (e.g. "src/*/.ts") to include files from specific directories or with specific file extensions. If include is not specified, TypeScript includes all .ts, .tsx, and .d.ts files in the project directory by default.

exclude - This setting specifies an array of file paths or glob patterns that TypeScript should exclude from the compilation process (even if they match the patterns specified in the include setting). You can use exclude to ignore files or directories that you don't want to be compiled, such as test files, build artifacts, or third-party libraries. Usually you will want to exclude your node_modules folder.

Additional features and capabilities of TSConfig

Declaration Maps - TypeScript can generate declaration map files (.d.ts.map) alongside declaration files (.d.ts) if declarationMap is set to true in your tsconfig.json. Declaration maps serve a similar purpose to source maps but are specific to TypeScript declaration files. These declaration maps provide mappings between the generated declaration files and their corresponding source map files, aiding in debugging and providing better tooling support.

Watch Mode - TypeScript's watch mode tsc --watch monitors changes to your TypeScript files and automatically recompiles them whenever they're modified. This is useful during development, as it speeds up the feedback loop and eliminates the need to manually trigger compilation after each change.

Incremental Builds - TypeScript's incremental builds feature keeps track of changes to your project's files and dependencies, allowing it to only rebuild the parts of your project that have changed since the last compilation. This can improve compilation times for large projects.

Override Options - You can override specific compiler options for individual files or sets of files using comment directives in your TypeScript source files. For example, you can disable certain strict checks with // @ts-ignore or specify custom compiler options with// @ts-nocheck for specific sections of code.

Use your tsconfig.json tile as a gateway to unlock the full potential of TypeScript in your projects. By understanding its purpose and leveraging its capabilities, you can embrace TypeScript with confidence, and gain a more reliable, efficient, and enjoyable development experience.

To get you started there is a community owned repository of TSConfig base files for your chosen runtime, then all you need to focus on is the unique choices for your project.

What’s next?

More and more developers are using TypeScript to build higher quality code bases and be more productive. Hopefully, this post demystifies setting up a new TypeScript project using tsconfig.json.

However, if you’re interested in diving into TypeScript without having to do any configuration,
Deno supports TypeScript natively. Simply create a .ts file, write some types, and run it immediately with
deno run your_file.ts.

My experience of Modern Frontends Conference

Jo Franchetti — Fri, 18 Nov 2022 13:59:46 +0000

I was reached out to by Modern Frontends conference back in June this year. November is the most busy month of my DevRel calendar, I always expect to be travelling and speaking a lot around this time. As far as I was concerned, this was another interested conference who wanted me to speak. I submitted via Sessionize, after being reached out to by the conference on twitter, with a TBC talk title and description and was accepted to speak. This by itself isn’t completely unusual, but it was a little odd as I didn’t at that point know who was behind the conference. They had reached out to me on twitter via a ‘modern frontends’ twitter handle and didn’t sign off their messages from anyone, doing some sleuthing on the website didn’t bring any clarification. I assumed that it must be someone who knows me and had accepted my blank CFP on faith.

I later discovered that the organiser of Modern Frontends is Gen Ashley, she mentions that she has a team working with her, but has not named them, nor given them credit for their work.

Modern frontends then reached out to the company I work at for sponsorship, we eventually declined to sponsor due to clashes.

Deciding to speak at Modern Frontends was easy for me, I live in the area of the conference and I already have prepared talks. My job is in developer advocacy, so my time writing and giving the talk is covered by my salary.

However this is not the case for everyone, I have heard that speakers who had to travel not only didn’t have their travel covered, but no hotel either.

Making speakers, especially new speakers, pay to attend and speak at your conference is unacceptable. Paying for travel and paying for a hotel counts as paying to attend. The speakers are the content, without them there is no conference. Compensate speakers for their time. It feels absolutely bizarre to me that this even needs to be said. ‘Networking opportunities at the event’ or ‘exposure’ do not count as compensation, as they say, people die of exposure. The least a conference should do is cover travel and accommodation, offering a speaker stipend is what conferences should do.

Because I work in DevRel, I am often contractually obliged not to take a speaker fee, usually, I will ask that it be donated to diversity tickets and scholarships, but I do appreciate when a conference compensates me for my time and effort in other ways. I have spoken at many conferences and usually they will do something nice for their speakers, like a gift in the hotel room, or a tour of the local area, or a speaker dinner or combination of the above. NDC gives a donation to a charity in the speakers’ name, for example.

I have seen multiple people online making statements to the above effect and being virtually silenced, shut down, shamed and bullied by Gen Ashley into retracting and deleting their posts. She has been rude to speakers, sponsors and anyone who has called her out. Speakers who have stood down from speaking at the event still have their faces plastered all over the conference website and associated advertising.

When I arrived at the event itself it was pretty grim, the venue was hard to find, unbranded, starkly lit. It was giving corporate-strip-light-carpet-tiled-hell vibes. The sponsors had tables around the hallway and were clearly unimpressed with the setup and the event itself. Sponsors were sold on 3000 attendees, there were certainly not 3000 people at the event, perhaps a few hundred.

I have spoken openly about my discomfort around the excess and entitlement that is shown at developer events, with free food, clothing and other swag being given out to the already fortunate and well situated. However this event took things to the other extreme. For the price of a £600 ticket, attendees were treated to a single coffee service, with urns of pre-made coffee and tea and a lunch of ‘packet sandwiches’. It was swept away within an hour too, so late attendees or those working over their lunch-break did not get a lunch. I don’t want to sound entitled, but for the price of the sponsorship and the quantity of sponsors, along with the price of the tickets, and the fact that they weren’t paying their speakers (I assume they might have paid some of the bigger names) it feels like the funds have been spent elsewhere…

Where the funds weren’t spent was on recording and livestreaming the talks. There was no recording equipment in any of the rooms. When I asked Gen about what had happened to the livestream she just told me that she was going to refund the people who had paid for livestream tickets and offer them a free ticket to next year’s event (ha!). At the time I asked, tickets for the livestream were still available for purchase on the event website at £43. This feels very close to fraud to me. I know people have been asking online for refunds, I hope they get them soon. I hope that if the event runs next year it is better managed.

I gave my talk and it went well, there was no one in the room to set me up, just an AV setup leaning against the podium. I set myself, up, gave the talk and left the room. As I walked through the hallway Gen came up to me and told me that she thought it was ‘funny’ that people were complaining about the event on twitter, and that she didn’t care. I really didn’t know what to respond to this, so I said that it wasn’t really funny and seemed quite serious and left it at that.

My final point to make about this conference is that it was booked to run on the same day that HalfStack London is on every year. Dylan has been running an absolutely excellent, inclusive and creative front end focused conference for almost 10 years on the same day every year. Modern Frontends booked to run on exactly the same day, which shows either rudeness or a lack of market research. Dylan ended up moving the conference to run on the Wednesday before Modern Frontends ran and it was a roaring success.

I know others have shared their experiences, and I do not want to center myself in this debacle, but it also does not sit well with me to watch this unfold and say nothing. I always advocate for fairness and diversity in tech and I will keep doing so. I apologise if my involvement in this conference and silence up until now has hurt anyone.

Visualize your commits in realtime with Ably and GitHub webhooks

Jo Franchetti — Fri, 12 Aug 2022 11:32:00 +0000

Have you ever wanted to see the work of your entire engineering organization in a visualization as it happens? In this article, I'll tell you how I used Github webhooks and Netlify serverless functions, along with a simple Svelte web app, to do just this in my new interactive visualizer tool.

Graphical visualizations are a fun way to see progress and to help your organizations to understand that, while tech work can sometimes seem invisible, the amount of effort involved in building software is a huge team endeavour!

In the early 2000s, a tool was built that could visualize your software version history, called Gource. It was built for CVS server, and later ported to work with Subversion, and later, finally git. Gource displays changes to a repository in an animated force tree diagram.

A video of Gource as a visualization tool

While Gource is an excellent and powerful tool, it only works locally, you have to install the software and run it. It parses historic commit log data, and generates a video file with the visualization using OpenGL. It doesn’t show data in realtime.

Taking inspiration from Gource, and with the advantage of the versatility of the web in 2022, I decided to build a modern visualization tool that works in a browser, in realtime. I wanted to depict code committed to a GitHub repository, or organization, as and when it is pushed or merged.

What you need to build a GitHub Visualizer:

To send a message via WebSockets every time a commit gets pushed.
A hosting platform to provide a payload URL
To convert commit data into a file/folder hierarchy.
A tool to display the commit data in a graphical form.
Some code to stop the app from crashing if it runs for a long time or receives millions of messages.

GitHub webhooks to the rescue

GitHub offers webhooks which allow integrations to subscribe to events on github.com. Triggering an event by, for example, pushing a commit, will send an HTTP POST payload to a URL configured in your GitHub account. Webhooks can be installed for an entire organization, or per repository, which means that I can get commit data in realtime over HTTP. The next task is setting up a hosting provider to accept the webhook data and provide a callback.

How to host an app on Netlify

My GitHub Visualizer is hosted on Netlify. Netlify, and other cloud products, provide native support for serverless functions which let us run server-side code without having to manage a server ourselves. These serverless functions can be bundled with a web application, and are version controlled, built and deployed along with the rest of the app. By binding a serverless function to the GitHub Visualizer web application I could set up that function as the target of a GitHub webhook. Once the data arrives from GitHub I could publish it to the UI using Ably (see the code).

How to build the UI with Svelte

Building a single page web application that is reactive to incoming changes in data usually requires a framework. I chose to build the GitHub Visualizer with Svelte because of its simple approach to state management, readable syntax, small size and speed. The app consists of two Svelte components; one that subscribes to an Ably channel and keeps a GitHub commit history stored as a variable in the application, and another that can render the GitHub commit history as a "force directed graph".

When a message arrives from Ably, it is pushed to the history variable. Svelte has built in support for reactive variable assignment, which means that when a new value is assigned to that variable, the change will be immediately reflected in the UI. The next piece of the puzzle is visualizing the data.

How to make a force directed graph with VisJS

In order to display the GitHub data in a graph, I used VisJS, a library specially designed to handle large amounts of dynamic data. The graph is rendered in the browser in an HTML canvas, making it possible to show complex animations efficiently. The VisJS network component was exactly what I needed to display a complex diagram of “nodes” and their connecting lines, called “edges”. Nodes and edges are represented in a JS objects, for example:

var nodes = new vis.DataSet([
    {id: 1, label: 'Node 1'},
    {id: 2, label: 'Node 2'},
    {id: 3, label: 'Node 3'},
    {id: 4, label: 'Node 4'}
]);

var edges = new vis.DataSet([
    {from: 1, to: 3},
    {from: 1, to: 2},
    {from: 2, to: 4},
]);

Which means that the data that the app receives from the GitHub webhook needs to be converted into this format to be rendered.

How to convert GitHub webhooks data into JS objects

When a message arrives, the app will iterate over the commits in the message. It will store the file names that have been created, modified and deleted. If a file has been created then the app will split up each part of the file path and create a "node", and an "edge" connecting each to its parent part, eventually meeting at the root in the centre, which is the repository. Every node created has the full path to this point as an id, and each edge gets its id set to "{from}=>{to}" so that they can be found by file path in the future.

An example node might look like this:

{
    id: "docs/content/tutorials/github-vis.js",
    label: "github-vis.js"
    group: "docs",
    isFile: true,
    isRepository: false,
    isDirectory: false
}

And an example edge:

{
    from: "docs/content/tutorials"
    to: "docs/content/tutorials/github-vis.js"
    id: "docs/content/tutorials/=>docs/content/tutorials/github-vis.js"
}

For every created file new nodes are made and for any removed file nodes are deleted. Edges are generated for all of the created and modified nodes. The app also keeps a queue (a first in first out array) of every node that is created, and when it reaches a maximum number of nodes to display, it removes the oldest items from the top of the array. This will stop the app from consuming unbounded amounts of memory if an organization is very busy and millions of messages arrive. Nodes that are modified are removed from the queue and put back on the end of it, so that they aren't prematurely removed.

How to display the data

The data is now in a structure which represents the files present in the last N commits (however many you decide to store). The next job is to write some Svelte code that runs each time the data structure changes, which will do the following:

Get the currently displayed nodes
Find any nodes that are in the data structure that haven't yet been rendered
Add those nodes to the force directed graph
Add new edges to the graph to join the added nodes
Remove any nodes that aren't in the data structure any more

VisJS has a reactive DataSet that can be used for this, the app just needs to to check the node IDs and add/remove nodes for it to trigger a re-render. Diffing the data set and data structure like this means that we don't continually recreate all the nodes on each commit. It also means that the physics applied to the graph by VisJS (the force which makes the nodes repel one another) is consistent and they won’t bounce around.

How to verify GitHub webhook data

There is one final gotcha in this project, it is necessary to ensure that folks can't just push data at our API and break the app. Luckily, GitHub supports signing webhooks with a HMAC signature. You make up a password and put it in the secret field in GitHub when you set up your webhooks. In the serverless function, we use the crypto module built into NodeJS to verify the HMAC and only forward it over Ably channels if the signatures match the shared secret.

Using this in your organization

Clone the repository: https://github.com/ably-labs/github-commit-visualizer.git
Deploy it to Netlify (or other hosting/serverless functions provider), remembering to set your Ably API key and GitHub webhooks secret as environment variables.
Add the function URL as a GitHub hook from your org or repository to your deployed app.

Watch as your beautiful visualizations pop into existence!

Potential extensions

There are so many ways that this project could be expanded. There is a lot of data about the commit returned in the payload from GitHub, so the visual could be extended to show usernames next to file changes.

Other ideas

Colour code the nodes by latest changes, make the older updates start to fade as newer ones come in
Make nodes link to their respective file in GitHub on click
Add a rewind option, so that you can look back in time
Add some UI to pick a maximum number of nodes shown, for those on smaller screens.

The code for this project is open source and available on GitHub, in our Ably-Labs organisation. Please take it and make it your own. I am very open to PRs and do let me know if you find any issues. I hope you enjoy visualizing your own projects!

Build your own live chat web component with Ably and AWS

Jo Franchetti — Fri, 16 Jul 2021 13:45:03 +0000

Web Components are a great way to build reusable functionality you can use in different web pages and web apps. Imagine sharing components between different frameworks like React, Vue.js, or Next.js! In this post, we delve into Web Components and show you how to build a chat web component with Ably to use it in an application built with AWS Amplify and AWS Lambda.

What is a Web Component?

Web Components are a collection of web technologies, standardized in the browser, to ease writing reusable pieces of markup and functionality. They’re a combination of Custom Elements, the Shadow DOM and HTML templates. When they’re used all together, they’re collectively called Web Components.
Web Components are an alternative to framework-specific approaches to component reuse such as React Components, and Vue.js templates. Web Components are interesting because they’re not bound to any runtime framework, and they’re supported by all modern browsers. Competing approaches to components for the web were established prior to Web Components being widely supported, which caused a lot of the frameworks to develop their own approaches to reuse and encapsulation. Web components offer a framework-independent way of building reusable functionality for the browser.

The basics of a Web Component

When talking about Web Components, most people refer to Custom HTML Elements, as these are the closest thing to the component models found in React or Vue. Custom Elements can:

Render HTML into themselves by setting their own innerHTML properties.
Encapsulate data by storing it as attributes on instances of themselves.
Track changes to data stored in their attributes, and trigger a re-render.
Allow you to write code that is triggered when they are added and removed from the DOM.

Writing a basic web component

The first thing we'll do in this post is create an example of a Custom Element that displays the number of times a button has been clicked. To create this Custom Element, we need to create a new JavaScript file that we can reference from our web page. Start by creating a new file called index.js. We'll define a class for the Custom Element and call it CountComponent. This will extend HTMLElement (the browser's base type for all its elements):

Inside this class, we need to define the attributes to be observed for change tracking. For the sake of this demo, we will return an array of a single string – the attribute name count:

Next, we need to define some custom attributes and their behaviour. We create a getterfor the custom “count” attribute, and in the function, the attribute value count is loaded with a call to getAttribute, defaulting to the string "0". Once the data is loaded, we’re parsing it from JSON – we do this because any data that we store must be a string.

We also need to create a setter for our attribute, and here we’re serializing the value to JSON and setting it on our element:

Even though Custom Elements are defined as classes, you can’t put any logic inside their constructors other than calling super to trigger the base logic for the HTML Element class. Add any other logic, or miss out calling super and you’ll see an error in your console and the element won’t work.

Because we can’t do any meaningful work in the constructor, Custom Elements provide lifecycle callbacks – functions that you can implement at different parts of the component lifecycle. We will use the connectedCallback function. Implementing this callback will cause the code to run when the component is added to the DOM – it is similar to React's componentDidMount function. In connectedCallback, we’re setting a default for the counter, and calling a function we’ll look at shortly called setContent:

The next lifecycle callback function available to us is disconnectedCallback (shown here only for illustration purposes as we’re not running any code). The code in this function executes when your element is removed from the DOM and destroyed.

Custom Elements also provide us with an attributeChangedCallback. We will use this function to execute code whenever an observedAttribute changes:

setContent is a function that the component calls when the component is added to the DOM, and when an update happens. The click handler in the above code can access the attributes of the element using the this keyword, so on each click, we increment the value, in turn triggering a re-render.
Note: This demo is not particularly performance focused, in that it sets the entire innerHTML and wires up a click handler to the button each time it’s called. This is not how one would build an application for production! In a real application, you would not reset your entire DOM. Instead you would only change the parts of the UI that required updating. Furthermore, the Shadow DOM APIs could be used in more performance-sensitive scenarios.

Finally, at the end of the JavaScript file, we call the browser's customElements.define API – providing a tag name for the element, and a reference to the class we just defined.

Using the basic web component

It is now possible to reference the Custom Element JavaScript file as a module and add the element we just made to the page using the tag name that we defined:

And the component renders thus

Building a Chat web component with Ably

We're going to build a text chat component – Ably Chat component – by building two Custom Elements:

AblyBaseComponent.js – a base component that encapsulates Ably interactions
AblyChatComponent.js – an element that inherits from AblyBaseComponent and implements chat logic on top of Ably.

The base component is designed strictly to be built upon, and to encapsulate the basic requirements for Ably API key management and subscribing to channels. Using this base component, we could build any kind of Custom Element that relies on realtime messaging.

In order to connect to Ably, you need an Ably account and an API key.

Building the Base Component

The base component starts with the standard Custom Element boilerplate – AblyBaseComponent extends HTMLElement, and calls its constructor.

In the connectedCallback function, we call this.connectToAblyChannel (which we’ll examine shortly):

The disconnectedCallback function loops through this.subscribedChannels (which we will create in a connect function) and unsubscribes from any Ably channels that have been used:

The connectToAblyChannel function is where a lot of the real work happens. We start off by loading some configuration – we will set up a convention that we expect users to supply either their Ably API key, or a callback URL to an API that responds with an Ably token authentication request. To supply these values, and because we’re creating HTML elements, we expect them to be set as attributes on the element when it’s created in the markup.

In order to read the data attribute when the element is connected, we expect the element to look as follows:

Next we need to design a way to configure the Ably client. In regular JavaScript you might use the Document API to interact with elements in the DOM – but because we’re literally inside the custom element, we can use DOM API calls like getAttribute from the this object.

This code tries to load data-api-key and data-get-token-url. If an API key is found, it takes precedence, otherwise, we create an Ably JavaScript SDK configuration object that contains an “authUrl” property. Now that we have a configuration object (either an API key or a URL), we can create an instance of the Ably Realtime JavaScript SDK. It is important to point out that this Custom Element relies on the containing page having already included the Ably JavaScript SDK using a script element before it is executed. The following code comes after the configuration, inside the connectToAblyChannel function:

Once we have the SDK instance, stored in the variable this.ably, we will also create an empty array called subscribedChannels.

We’re going to conclude with two additional functions in the AblyBaseComponent – a publish function and a subscribe function.

The regular Ably SDK exposes publish and subscribe functions once you have called ably.channels.get(channelName)to get a channel. For this chat example, we want to let the AblyChatComponent decide which channels it will publish and subscribe on, effectively extending the API surface area of the Ably SDK (the set of things the API can do).

The augmented publish and subscribe functions take an extra parameter at the start – the channel name – then pass on the rest of the arguments to the Ably SDK to do all the hard work for us. We do this by destructuring the arguments array, and ignoring the first element. This allows us to capture all of the parameters, except the first one, in our newly defined variable args:

We can then use .apply on the publish or subscribe calls from the Ably SDK to pass the rest of the variables to the SDK to publish or subscribe to messages.

We will use the first parameter, the channelName, to get the correct channel and keep track of it so we can unsubscribe when we unmount from the DOM. The following code will go inside the publish function, below the arguments assignment. The subscribe function works similarly

As you can see, both publish and subscribe are virtually identical functions – with the exception that we’re passing to them the call to channel.publish or channel.subscribe, respectively.

This all might seem a little framework-like, but what it means is that developers building Ably components on top of this base class can call publish or subscribe with an extra channelName parameter at the start of the call, and not worry about connecting or disconnecting from Ably channels.

This base class contains all our Ably code, and is now ready for us to build exciting components on top of it, so let’s create our second component, the AblyChatComponent.

Building the Chat Component

These components are designed to be imported as an ES6 Module – where the script tag you use in your HTML has type="module"as an attribute. When using ES6 modules, we can use import in browser components, so to start here, we’re importing our AblyBaseComponent in order to extend it.

The next thing to do is set up some boilerplate code for the element. Define an attribute called “messages”, and set it up to be observable. Next, set up constructor, which in turn calls the constructor of the AblyBaseComponent using a call to super();.

We can use connectedCallback to configure the chat application. We call super.connectedCallback to trigger all of the Ably configuration logic from the base component. The following code goes below constructor, inside the closing bracket of the AblyChat class:

We call the super.subscribe* *function that was defined on the base to subscribe to messages on a channel called chat. The rest of the parameters, as we pointed out earlier, are standard Ably JavaScript SDK parameters – we’re subscribing only to messages on the topic chat-message. When a message is received, we call a function called this.onAblyMessageReceived (which we’ll implement in a moment), passing the received message as an argument.

In order to ensure that any CSS styles applied to the page don't affect the component, and vice versa, inside the body of connectedCallback, we will generate a random string and assign it to a property called id:

Next we call a function named renderTemplateAndRegisterClickHandlers, which we’ll look at shortly.

Finally, we place the browser focus on an element called this.inputBox that is generated when the template is rendered, so that people using the chat UI will be able to instantly start typing.

Next we use the attributeChangedCallback to update the innerHTML of the chat bubbles in the chat window when messages are received. When an attribute is changed, the innerHTML of this.chatText is set and scrolled into view. We’re using a function called formatMessages, which takes the message history, and converts it to HTML elements fit for display:

Next we set up the renderTemplateAndRegisterClickHandlers function, which is named for what it does! The start of this function calls another function called defaultMarkup, that takes a single parameter – the *id *of the element, and returns the innerHTML we want to display on the screen – an empty chat box element.

Once the element has been rendered into the DOM, we can use querySelectorAll to find the chatText, inputBox, sendButton, and messageEnd elements so that we can use them in our code:

Inside we also wire up the eventListeners for clicks on the sendButton and on each keyPress in the input box so that we can process user input.

We mentioned the onAblyMessageReceived function earlier when we subscribed to Ably messages – this function takes the current message history from this.messages, makes sure it is at most 199 messages long, and adds the latest message to the end of the array:

This new array is then assigned to this.messages, which triggers the UI to re-render because this.messages is an observed property.

The sendChatMessage function is called either when Enter is pressed, or when the button to send a message is clicked. Because we’re extending the AblyBaseComponent, it calls the super.publish function, passing the channel name and the Ably message payload:

You can see that it’s also responsible for clearing the text box and focusing on it so the user can continue chatting.

The handleKeyPress function is triggered on each keypress. If the keypress is the Enter key and there is a message in the chat box, it sends the chat message:

The formatMessages function is responsible for mapping the Ably message history into span elements. There’s a little bit of logic here to detect whether or not the message was sent by the current user of the app by checking the message.connectionId property against the ably.connection.id property, and adding a me or other CSS class that can apply styles to. The data property from the message is used to carry the received text message.

The above concludes the custom element class. After the custom element class ends, we have two functions. The first is the uuidv4() function – which generates a unique id for the component:

The second is defaultMarkup which takes a single parameter – the ID of the component – and uses it to set the id property on the generated HTML.

Once we’ve set this id property, we can embed CSS that is scoped specifically to this element ID directly into the output code. This means that if multiple instances of the custom element appear on the same page, they won’t have conflicting ids or styles.

At the bottom of this next snippet you can see the markup for the component – a div for holding message history, and a form for capturing user input, complete with the class names used in our query selector calls earlier.

And of course, much like our example element at the start, we’re calling customElements.define to register the HTML tag:

The custom element is now functionally complete, and so long as both the AblyBaseComponent.js and AblyChatComponent.js files are included in a web application, they can be used by referencing our AblyChatComponent.js as a module.

Using the Chat Component

To use the now-registered custom element, we use it like any old HTML tag and provide it with the correct attributes:

The element renders like this in the page, and when configured with an API key or get-token-url, it just works!

API Key Management

As hinted above, we need to talk about API key management. While this custom element supports reading Ably API keys straight from your markup – which is great for local development and debugging – you absolutely should not store your Ably API keys in your markup, otherwise they could get stolen and misused.

The recommended way to use API keys in the front-end is to use Ably Token Authentication. Token authentication is an exchange mechanism where you use your real API key to generate limited-use tokens that can be passed back to your clients to use in the front end.

In order to use token authentication in this way, we need to create an API somewhere, to call from the front-end, that has your real Ably API key stored in it. We can then use a function in the Ably SDK to exchange your real API key for a token that gets returned to the Ably JavaScript SDK. The Ably JavaScript SDK manages this token exchange process for you. When you provide a URL that points to an API that will return a token, the SDK will manage and refresh the token as required, so you don’t need to worry about it. This demo will walk through using AWS Lambda functions and the AWS API Gateway to achieve this.

Serverless Token exchange with AWS Lambda

The following example AWS Lambda function provides the necessary token exchange functionality. All we need to do is require the Ably JavaScript SDK, and create an instance of the Ably.Realtime client passing your Ably API key from process.env.ABLY_API_KEY.

We use client.auth.createTokenRequest to generate a temporary token, and return it back to the client.

The onus is on the owner of the API key to make sure that the users requesting a temporary token have access to chat – you can authenticate the requests any way you’d like, and it’s no different from authentication in any other lambda function. In the next section, we go through hosting this on AWS Lambda

Using AWS Lambda for Authentication

To deploy to AWS Lambda, we need to create a new directory called /api/createTokenRequest with two files in it – package.json and index.js Here’s the package.json file

And this is the index.js file

These two files, along with their node_modules are required by the AWS Lambda runtime. We're going to use npm to restore the node modules and then compress the contents of the /createTokenRequest directory into a zip file. In the terminal execute the following:

After that, zip up the contents of the createTokenRequest directory (this process is OS dependent). We will use the AWS UI to create a Lambda function and upload this zip file as the source code.

We'll walk through this process now. You need to log in to your AWS account first:

Search for lambda in the Services search bar, and click the Lambda services box that shows up in the results:
Click the “Create function” button to create a new Lambda function
Select "Author from scratch" and give your function a name, then click "Create function":
Once the function has been created, click “Add trigger” to make the Lambda function accessible over HTTP:
Select “API Gateway” from the “Trigger configuration” dropdown
On the resulting page, select your function from the dropdown and click "Add".
Set the Deployment stage to default and the Security to Open, then click "Add".
Once you have added your trigger, the UI shows a URL in the Triggers tab under Configuration. (This is what you will add as the data-get-token-url parameter when you use your component in the HTML, but we still have some more setup to do!)
Now you need to upload the zip file that we created earlier. Click on the "Code" tab, then "Upload from" and select ".zip file":
Once the zip file is uploaded, you will need to set up your environment variables with your Ably API key. Under "Configuration", select "Environment variables", then click the "Edit" button: Add your Ably API key to the “Environment variables” settings And that's it, your Lambda is set up

And that's it, your Lambda is set up

In the index.js file we’re reading from process.env.ABLY_API_KEY. You will need to generate a new ably API key and then define this environment variable, with it’s key value in the AWS UI (or using an automation tool of your preference).

Once our Lambda function is created, we’ll need to add an AWS API Gateway Trigger to give our lambda an externally accessible URL. This is the URL that we can safely configure in our HTML markup in lieu of our actual API key. The Ably SDK will take care of the rest.

Hosting your Web Component on Amplify

Now we can walk through hosting your component on Amplify, the AWS static web hosting service.

Search for Amplify in the Services search bar and click on the resulting AWS Amplify link.
Click "GET STARTED"
Scroll down the resulting page to "Host your web app, and click on "Get started":
Authenticate AWS Amplify with your GitHub account
Select the repository of your web component
Edit your build settings to include npm run ci as a prebuild command and set the baseDirectory to "/build".
Click the "Save and deploy" button to host your component
If all is well, the component will provision, build, and deploy successfully. The UI will provide you with a URL to view your hosted component.

The hosted web component will look something like this

Using the NPM package

Because components are built as plain old JavaScript, we can distribute and consume the component using NPM and any of the variety of browser-friendly ways to add NPM packages to your front end.

Here at Ably, we’ve published the component to NPM as ably-chat-component, and you can reference it directly using the Skypack CDN. This ensures packages are browser-compatible.

You need to reference the client-side Ably SDK for the component to work. However, once you have done that, you can reference the Skypack URL for our component, and add the ably-chat tag into your page, set your API key, and everything will just work.

This is the simplest supported way to use this component in development mode. As mentioned above, however, you will need to switch out your API key for a token request URL of your own.

Component Architecture

Diagrammatically, the component architecture can be depicted as such:

With the corresponding sequence diagram

Conclusion

In this piece we’ve broken down how web components work, explored Ably and Web Components, and walked through how we can use AWS Amplify and AWS Lambda to host applications that support realtime chat.

If you’ve already got a web application and know how to host it, we’ve also touched on how you can use Skypack to include this component directly from NPM.

Chat is just one way that you can use realtime messaging and Web Components, and we’d love to see what you can do with this codebase.

Build a tamagotchi game with Realtime TFL Data — Tamago-Train!

Jo Franchetti — Thu, 30 Jul 2020 09:25:52 +0000

The Station Manager game with Realtime Tube data

I’ve been learning a lot recently about using realtime data streams and how and why one might use them in an app. In order to better understand the differences between realtime streaming data and REST APIs (which I’ve had more experience with), I decided to build a game, the mechanic of which uses realtime train arrival data. As trains arrive into the station in real life, effects are triggered in the game which the user has to manage.

Make it your own!

All of the code for the game is on Glitch. This means that you can see the code, ‘remix’ it and make it your own. There is a thorough readme file in the repo and I’ll also go over some of the methods used in this blog post.

Getting Arrivals Data

Ably has a Hub of realtime data streams for developers to try out and build apps with. I used the London Tube Schedule stream, which provides a stream of various data from TFL; including arrivals at a given station. For the tamagotchi game, I wanted to find out the arrivals at a busy station, so that I would have lots of trains arriving in close succession. I chose King’s Cross station for this reason. The data stream uses the station NAPTAN code rather than it’s name to get the correct data, so I had to look up the correct code for King’s Cross (you can look up station codes here), which is 940GZZLUKSX.
The stream I’ll therefore be using is:

[product:ably-tfl/tube]tube:940GZZLUKSX:arrivals

Every time the data from TFL is updated, this channel will publish a message with the new arrival times of trains into Kings Cross. The joy of the data being a realtime stream means that I don’t have to poll for the data, as I would with a REST API, instead, I establish a connection once and data is published to the channel as and when updates happen.

Connecting to a Data Stream

In order to connect to the data stream I used the Ably Javascript SDK. To do so, I need an Ably API key which comes with a free account. To keep my API key safe, I also used Token Authentication which makes a Token Request on the server side which is handed to the Ably realtime client to authenticate. There is a brilliant walkthrough of how to implement Token Authentication here:

TFL Data

The data published by the stream looks a little like this ↓

It is a large array of train objects each with a lot of information. For the sake of this game, the only information I’m really interested in is the TimeToStation value. I can use these numbers to calculate when to cause a train to arrive into the station in the game.

I could have created all kinds of interesting extensions to the game, with multiple platforms and colour coded trains for their lines, even maybe an arrivals board with train destinations, but let’s not get too carried away…

Game mechanics

Congratulations! You’re the newest TamagoTrain Station Controller! Now you’ve got to keep your station in fine fettle.
Don’t let your station get too hot, don’t let your platform fill up with passengers, litter or mice!

Trains raise the temperature of your station, as do passengers
If it gets too hot, passengers will start to faint!
Unconscious passengers can’t leave the platform
Passengers sometimes drop litter
Too much litter attracts mice!
Trash and mice all take up space on the platform making it difficult for your passengers to exit
If your platform gets too full, too hot or too cold your station will have to shut and your game will end

How to play

Clean the platform to clear away litter
Vent cold air through the station to keep everyone cool (but don’t get carried away!)
Passengers departing through the exit will cool the platforms down a little
Departing trains also cool the platform slightly
You can charm mice with songs! They’ll find their way off the platform if musically enticed
Music will also wake up fainted passengers

Game Code

The game is an expressJS app. It is split into two parts — the simulation/game loop, which runs in ‘ticks’ and the ui/render loop which runs at 30 frames per second. This separation prevents us from tying the game logic to the frame rate, which will reduce the chance of the frame rate dropping if the game logic gets complicated. (If you’re interested, this is a great intro to game loops.)

Game.js

The Game.js file is the main control loop for the game - in it, we define a JavaScript class called Game. When games are created, we create a new instance of this class to hold the game state. This is also where we set up the tick() function, which is called once per second. This ‘tick’ steps the simulation forward by iterating the game loop. It ‘ticks’ the game entities (the platform, passengers and trains), applies any problems (adding litter and mice) and applies any buffs (cleaning, venting or music).

The only input the user can supply is applying a Buff — either clean, vent or music, which are triggered by the buttons in the UI. When pressed, these buttons add a Buff object to an array in the Game instance, that we use as a queue of actions. Buffs can only be added to the queue a maximum of 3 times, after that clicking the buttons in the UI will just return until the Buff has been taken off the queue.

The Game instance is responsible for three core things

Handling train arrival/departure messages, and routing them to the platform
Creating instances of Buffs
Checking for game over

All the rest of the game logic happens in the tick() functions found on the Entities, Problems and Buffs.

GameUI.js

The GameUi.js file is where the rendering of the game happens. It uses an observer pattern to keep track of the game state.

30 times a second the GameUI.draw() function is called and passed a snapshot of the game state. GameUI instance keeps track of the last state it was called with so that it can avoid re-drawing things that have not changed.

The GameUi class has a collection called _renderingFunctions — a list of functions it calls in order, each being passed the current game state. If any rendering function returns a value of -1, we use this as a signal to stop drawing to the screen and to display the** Game Over** screen. The rendering code places absolutely positioned divs on the page which are styled in the css. The divs contain animated gifs of the entities, buffs and problems. The appearance of the divs is changed by adding css classes and data-attributes dependant on the problems or buffs that have been applied in the game state.

Entities, Buffs and Problems

By default, when an instance of Game is created, a Platform entity is created. This platform has some basic state (an age measured in ticks, a width, a height) along with the three core stats the game is ranked on - hygiene, temperature and capacity. The game is won or lost based on the state of these variables, which the game evaluates each tick. As the game ticks, the Game instance will process any objects in its queue first in first out, creating an instance of the requested Buff and applying it to the Platform.

When the Platform ticks, the following things happen -

Any unprocessed messages are read, FIFO.
If a message for a train arrival or departure is found a train is created on the platform or removed from it.
All tickables are ticked.
Any completed contents or buffs are removed — an item is deemed complete if a property completed is present, and set to true on the object.

The tickables that the platform stores are:

Any present train
All of the contents of the platform
All of the buffs applied to the platform

On each tick, the item that is being ticked gets handed the current instance of the platform, and based on the logic in that item’s class, it can mutate the properties of the platform. For instance - every tick, a Mouse could reduce the hygiene property of the platform.

The rest of the entities, Buffs and Problems are all JavaScript classes that can mutate the state of the Platform instance in their tick method.

Both Entities and Problems have x and y coordinates that are used to move them around the user interface.
Problems all inherit from a Base Class called Problem which creates these properties by default for them.

A problem looks like this:

Entities and problems hold state which will cause effects during the lifetime of a game. For example:

Travellers walk towards the exit by moving 10 pixels closer to the exit each tick
Travellers have a chance of dropping litter
Litter has a chance of adding mice to the platform
Trains add an extra Traveller to the platform every tick

All of this logic exists in the tick function of each kind of entity or problem.

Starting the Game

The game uses webpack to bundle the client side JavaScript. The script.js file is the webpack entry point, webpack bundles together all of the JavaScript files before they are sent to the browser. This is great because it means we only need to reference script.js to start the game.

The script.js file is referenced in the index.html file and it takes care of starting new games. It contains a startGame() function which does all the work:

This function:

Creates a game instance
Creates an instance of the GameUI class, passing it a reference to the new game instance
Calls game.start() passing a configuration object of two actions - one to execute on start, one on end.
the onGameStart action listens for events on the dataSource
the onGameEnd action disconnects the dataSource to stop the game from using up messages that we don't need.
The ui.startRendering() function is called which will set up the render loop
Finally the game is returned so that the UI buttons will work in the browser.

Game Over

Game failure states are managed in the Game.js file, in the function isGameOver(). This contains a collection of objects with functions for different failure conditions. At the start of each tick, each of those functions are run and if any of them return true then the game is over.

Have fun!

I hope you’ve enjoyed playing the game and will enjoy making your own versions, or even adding some extensions to mine. If you’ve any questions about the game or about realtime data streaming you can drop me a message in the comments or get me @thisisjofrank on twitter. I’d also love to see any remixes you make!

Getting started with IoT Wearables and MQTT

Jo Franchetti — Wed, 27 May 2020 10:04:04 +0000

How to make your own internet connected t-shirt with LEDs that respond to realtime data

As a developer advocate I get to make all kind of fun projects. I particularly like to make wearable technology, specifically, things with lots of lights on that can connect to the internet. Because I like to be sparkly! I started making wearable tech projects a couple of years ago, my first being a light up, tweet controlled wedding dress.

When I started working with hardware, I had no real clue what I was doing and made a few very silly mistakes around choices of hardware and how to stick them all together, My hope for this blog is to give a little direction to those who are looking to get into hardware projects and to help you over that initial ‘hardware is scary’ hurdle.

The project we’ll be making is a wearable LED array which responds to data sent from a web app. The app allows people to draw in a 16x16 pixel array, kind of like a low res version of MS paint, but multiplayer! In order to realise this dream we’ll need a few things:

Firstly an array of lights which can change colour
A microprocessor to control those lights
An app where users can set colours
A method of sending messages to and from the app and the microprocessor.

Making a wearable LED array

Let’s start by building the array of lights. It is going to be a 16 by 16 grid of LEDs.

The LEDs that we’ll be using in this project are called Addressable RGB LEDs. Each LED has a tiny microcontroller which allows it to be lit up with a unique colour and brightness. Addressable means that we can target each LED individually, we send them an RGB colour value (similar to that which you might be familiar with from CSS).

For this project we’ll be using 5V LED strips. These are great as we don’t have to individually solder each LED, this also makes them nicely robust for a wearable, fewer solder points means fewer points of weakness where the connection could potentially snap - a lesson I sadly learnt the hard way with my dress project!

Addressable RGB LED strip

The strips can be bought by the meter or spool and you can cut them along the lines indicated to get them to the length you require. We’ll be cutting them into lengths of 16 LEDs. They also have clear labelling for which lines carry power, data and ground; which makes them easier to solder together correctly.

Things to bear in mind when purchasing LED strip:

The price of the strip increases as the pixel density increases
The voltage — we want 5V for this project so that we can run it with a USB battery.
The colour of the ribbon itself — try and match the clothing you’ll put the array on
The type of RGB LEDs (some are RGBW which will require you to send an extra “whiteness” value in your code)

I opted for a strip with a white ribbon (as the tshirt I’ll be mounting it in is white) and with LEDs that are 2cm apart. I cut them down to 16 strips with 16 lights per strip and laid them out in a square:

To keep the strips safe in a wearable I sewed two piece of fabric together to make long pockets to insert each strip:

Diagram of sewn pockets

(In actuality, my sewing was much more messy than this)

Insert each strip into each pocket and carefully solder together each of the connections at ether end, to create continuous connections throughout the strips. Pay attention to the direction of the data line indicators on the strip while doing this. Connect together the power data and ground lines:

You can buy a soldering iron and solder from around £30 and I used single core copper wire to connect them all up (because it doesn’t fray so I find it easier to solder). There are lots of tutorials and videos on soldering online so I won’t go into it here, but it isn’t too scary and once you’ve had a few goes at practising you’ll get the hang of it quickly.

Once soldered, you will have a flexible LED array which can be sewn into clothing.

Controlling the array

So, now that we have a display, we need a way to actually control what colours are sent to it. A microprocessor will be useful here as they can do enough calculations to control the lights, but are nice and small so can be easily hidden in a wearable. For this project I am using the Adafruit Feather Huzzah, a small, lightweight, affordable board with onboard WiFi. Adafruit have written some great step by step instructions on how to get started with this board and the Arduino IDE.

The Huzzah runs C/C++ out of the box and the Arduino IDE comes with a collection of example code to get you up and running. Here’s an example of how to set all of the lights in the array to show red:

You’ll need to do a little more soldering to connect the Huzzah up to the LED array. As seen above, we’re putting the data out on pin 4 and we’ll be using a USB battery, so you’ll want to connect your pins up as follows:

Board⠀⠀⠀** ⠀LEDs**
Pin 4⠀⠀ ↔ ⠀Data
GND⠀ ⠀↔ ⠀⠀−
USB ⠀ ⠀↔⠀ +5V

Connect up the board and run the above code to see your LEDs light up!

Making a web app to set the colours on the array

So, now we have a display, and a way of controlling it, we need a way to tell the controller what colours to set. Being a web developer, I opted to build a web app. It looks like this:

The app has some buttons at the top where the user can pick a colour, and an SVG of squares that represent the pixels in the array.

Each square in the SVG has an id with its number in the array — 0, 1, 2, 3, etc etc

Once a colour and a square has been selected by the user; the app updates the colour of that square to match the selected colour.

You can check out the app and its code here: https://ably-mqtt-iotshirt.glitch.me/

As well as updating the display of the app, we want a selected square to illuminate the correct LED on the array. We can do that using a **realtime data service and the pub/sub model. We’ll create a data channel on which we will which **publish messages that contain the colour updates. We can also subscribe to this channel to listen for changes made by other people also using the web app.

To achieve this, I used Ably’s messaging platform to set me up with a channel. Then I used their JavaScript SDK to manage publishing and subscribing to messages on this channel. In order to use the SDK, you’ll need an API key which you can get with a free account.

On the front end of the app I use the following code to set up my channel and subscribe to it:

The cool thing about using realtime pub/sub streaming is that all of the other users of the web app are also subscribed to these messages, so their clients will update when people paint cooperatively with them!

We’re using the Ably SDK’s createTokenRequest feature to authenticate to be allowed to connect to the data stream. On the back end, in a node server we will require the Ably promises SDK to do our authentication and to create Ably Token Request objects. Using tokens instead of sending the API key directly minimises the amount of work that our server needs to do and keeps our API keys nice and safe.

Now that we’ve established a connection to a channel, and subscribed to it, we need to publish a message to that channel when someone clicks on a square. We add a event listener for a click event to each of our squares:

When clicked, we’ll publish a message to the “tshirt” channel with the number of the pixel to change and the RGB value of the selected colour. An example of the data sent would look like this:

**name:** tshirt
**data:** 001#aa00ff

Where we’ve clicked the second square in the array and selected a purple colour.

Receiving data on the Huzzah

Now that we’re subscribed to and publishing to the channel in the web app, we need a way to get the data from that to the Huzzah board.

You are probably familiar with HTTP — the protocol that browsers use to request web pages. The word “protocol” just means “the way two things talk to each other”. HTTP is great, it powers websites, and APIs and is built to be descriptive and flexible, and it can be used for IoT connections, but it’s not lightweight and it’s not terribly fast. Another problem with HTTP is that it’s pull only, the device has to constantly connect and ask “Are there any updates?” “What about now?” “Anything now?” which is both data and time consuming.

On low power IoT devices, we don’t have much memory, power or bandwidth, so we need protocols that are designed to be small and fast. Message Queuing Telemetry Transport — MQTT is a connectivity protocol which was designed to be extremely lightweight. Connecting to a server only takes about 80 bytes and the device stays connected the entire time. Data is published when it is pushed from the device to the server and the device subscribes to data pushed from the server. Because the size of the data being sent over MQTT is small by design, messages can be sent quickly, making the hardware very responsive. Making it feasible to change the lights on the tshirt in realtime!

In order to use MQTT, we’ll need an MQTT broker. This is just a server that the devices connect to using the MQTT protocol in order to listen for messages. It keeps track of all the clients that are connected and the topics they are subscribed to, forwarding any messages to any subscribers. For this project I am using mqtt.ably.io as my broker. A third party that your IoThings can connect to to send and receive messages.

The cool thing about the Ably MQTT broker is that any messages we send in our browser to an Ably Channel using the JavaScript SDK, are also sent out over MQTT automatically, so we have no extra setup to do!

The code to get our Huzzah board up and running with MQTT is as follows:

Every time ensure_MQTT_connected is called, if the client is already connected, it returns immediately, otherwise, it’ll loop until it can establish a connection — subscribing when it does. This function is called every time the hardware runs its main loop to make sure it doesn’t accidentally disconnect from the MQTT broker due to our internet connection dropping. The process_messages function calls the client’s loop function. This function is part of the MQTT library, it calls the callback to get any messages that have arrived in the MQTT buffer since the last time it was called.

If we take the example message we used earlier we can look at how we will process it once received by the board:

The final thing to mention is the order of the lights in the array. You may have noticed that when I wired up the lights, I wanted to keep my wires nice and neat, so i soldered these nice little jumps at each end. But this means that the data line doesn’t run from left to right as would in normal array enumeration - it runs in what I have affectionately called ‘Snake mode’. :

Not only that, i connected up the cable at the wrong end of the first strip, all of this means that we have to essentially reverse every other line of the array in order for the numbering of the LEDs to match that of the array in the app. Oops! The code to do this is as follows:

The code to run an array of neopixels wired up in this configuration is now available as an Arduino library.

The IoTshirt in action

Here are some photos of the t-shirt

1)⠀The t-shirt with the array on the front (and me pointing at it)
2)⠀An example of drawing on the t-shirt using the app, I drew a heart.
3)⠀People using the app in realtime, they created a beautiful Paolozzi like piece!
4+5)⠀A message I wrote on the app and it showing on the tshirt.

What’s next?!

I’d love to take this idea further and perhaps make a wearable game, something like a tamagotchi or digimon, which would require a slightly larger pixel density, and therefore a new pixel array. I’d also love to get some animations running on the t-shirt. If you’ve got any ideas for how I can develop this further or some fun demos we could run on the t-shirt, do let me know! I’d love to work with others on this project.

All of the code for the t-shirt and the app is opensource and available on GitHub: https://github.com/thisisjofrank/interactive-lights
you can see and use the app on
Glitch: https://ably-mqtt-iotshirt.glitch.me/ and it’s code is
remixable: https://glitch.com/~ably-mqtt-iotshirt

I hope this long, rambling blog gives you some starting points for hardware projects and realtime data. Let me know if you make your own or if this inspires you to build other projects, I’d love to see them!