React Static Site Generation from scratch

So far we have a server-rendered react application but sometimes it's nice to just have some flat files that you can throw onto S3 or whatever. This technique is called "static site generation" or "SSG." This was popularized by JAM Stack architecture but it's also just relevant for things like blogs. If you don't actually need re-renders or dynamic data you can just push everything to the client and forget about the server altogether and it's very performant (assuming you are making the right trade-offs). Since this is supported by frameworks like Next and Astro, I want to try adding it too.

Luckily, the way we've set things up, the basics are fairly easy. Since each file corresponds to a route already we just need to walk through the directory of pages and for each generate a corresponding output file. And since we already have the responders that do conversions, we can just call them like we would for the served versions.

Let's just start by walking through with a glob and printing out the files. We'll create a new script static.js that will be the SSG equivalent of server.js.

//static.js
import { expandGlob } from "https://deno.land/std@0.207.0/fs/mod.ts";
import { join } from "https://deno.land/std@0.207.0/path/mod.ts";

const baseDir = "./routes";

const files = expandGlob(join(Deno.cwd(), baseDir, "**"));

for await (const file of files) {
    console.log(file);
}

Easy. Now let's pass those to our responders.

// static.js

import { expandGlob } from "https://deno.land/std@0.207.0/fs/mod.ts";
import { join, resolve, toFileUrl } from "https://deno.land/std@0.207.0/path/mod.ts";

const baseDir = "./routes";
const outDir = "./out";
const responders = await Promise.all(Array.from(Deno.readDirSync("./responders")).map(f => import(toFileUrl(resolve(join(Deno.cwd(), "./responders"), f.name)))));
const files = expandGlob(join(Deno.cwd(), baseDir, "**"));

for await (const file of files) {
    if(file.isDirectory) continue;
    for(const responder of responders){
        if(!responder.match(file.path)) continue;
        const response = responder.default(file.path);
        //write
        break;
    }
}

We gather the responders like last time. First we see if the file is a directory, if so then we skip it. Then we check each responder for a match, if so we'll call the responder's main method, write it out, and then we need to break so that we don't try to re-process it with another responder (order matters here, but I never added a mechanism for that so it's just whatever order the responders come in but we could add a priority value to them or ditch the auto-loading and manually add them in the order we want etc.). Anyway we'll run into a problem here. The server-responder doesn't work because it wants a request. This highlights a difference between things that run on the server versus statically. There's two way I see to do this:

1) We add a property that tells us to skip this responder in SSG mode
2) We get clever and add a function that gives us the defaults to use when static rendering and then output the response.

Most SSGs will pick #1 because it's very sensible, so for fun let's go with behavior 2. We can add a getStaticRequest method to each server.(j|t)sx? route, or not, if not then we skip.

//./routes/nested/hi3.server.jsx - at the bottom

export function getStaticRequest(path) {
    return new Request(path, {
        method: "get"
    });
}

We can call this to get a default request. We'll take in the path incase it does something meaningful but that's really all the data we have. So now in the server responder we can make the request optional.

// ./responders/server-responder.js
export default async function serverResponder(path, req){
    const moduleImportPath = toFileUrl(resolve(Deno.cwd(), path));
    const mod = await import(moduleImportPath);

    if(!req && mod.getStaticRequest){
        req = await mod.getStaticRequest(path);
    }

    if(!req?.method){
        return new Response("No Method or request", { status: 405 });
    }

    if (req.method === "GET") {
        return mod.get?.(req) ?? mod.default?.(req)
            ?? new Response("Method not allowed", { status: 405 });
    } else if (req.method === "DELETE") {
        return mod.del?.(req)
            ?? new Response("Method not allowed", { status: 405 });
    } else {
        return mod[req.method.toLowerCase()]?.(req)
            ?? new Response("Method not allowed", { status: 405 });
    }
}

If there's no request we make one with the method. If the method doesn't exist then we'll fall into the the guarding if and return a 405 status. Now let's wire this up to a file output. For now I'm excluding all other responders because they need a bit of work.

To get it to output we need to write some files but the subdirectories need to exist first. Deno can do this with ensureFile which will see if a file exists but also create the subdirectories. Once the path is good we can create the file and write to it.

// static.js - at the bottom
for await (const file of files) {
    if(file.isDirectory) continue;
    for(const responder of responders){
        if(!responder.match(file.path)) continue;
        const response = await responder.default(file.path);
        const outPath = join(Deno.cwd(), outDir, relative(baseDir, file.path));
        ensureFileSync(outPath);
        const outFile = Deno.createSync(outPath);
        response.body.pipeTo(outFile.writable);
        break;
    }
    console.log("Fell through", file.path)
}
console.log("Done!")

So we need to get the path relative to the routes directory and then output them into the "out" directory. We ensure the subdirectories are created and then create a file and pipe the output of the response. If we do this it'll work but we might get an unexpected result, server files that do not have a default request method will still get rendered but with error messages. Now this is probably why this isn't a feature in most frameworks, is that expected behavior or not? Is this even useful at all? In our framework we'll take the position that you should know what you are doing and because this is static it shouldn't return non-200 status codes unless you meant it to fail. If it fails then we don't write a file.

To start let's also purge the directory so that we don't accidentally retain files that shouldn't be there. Then try again taking only success statuses.

// static.js - at the bottom

import { join, relative } from "https://deno.land/std@0.207.0/path/mod.ts";

Deno.removeSync(join(Deno.cwd(), outDir), { recursive: true });
for await (const file of files) {
    if(file.isDirectory) continue;
    for(const responder of responders){
        if(!responder.match(file.path)) continue;
        const response = await responder.default(file.path);
        if(!response.ok){
            console.log(`${file.path} returned a non-successful status, skipping generation`);
            continue;
        }
        const outPath = join(Deno.cwd(), outDir, relative(baseDir, file.path));
        ensureFileSync(outPath);
        const outFile = Deno.createSync(outPath);
        response.body.pipeTo(outFile.writable);
        break;
    }
    console.log("Fell through", file.path)
}
console.log("Done!")

Now we only generate a file for routes/nested/hi3.server.jsx with the response. One more issue here is that the file extension is still .jsx but we returned text. Let's fix that by using the mime type. Deno has the reverse of going from mime-type to extension in it's standard library (if we can't match it assume it's text because that's as good a default as any).

// static.js - at the bottom
import { extension } from "https://deno.land/std@0.207.0/media_types/mod.ts";


try {
    Deno.removeSync(join(Deno.cwd(), outDir), { recursive: true });
} catch(e){}

for await (const file of files) {
    if(file.isDirectory) continue;
    for(const responder of responders){
        if(!responder.match(file.path)) continue;
        const response = await responder.default(file.path);
        if(!response.ok){
            console.log(`Skipped: ${file.path} returned a non-successful status`);
            continue;
        }
        const outPath = join(Deno.cwd(), outDir, relative(baseDir, file.path));
        const ext = extension(response.headers.get("Content-Type")) ?? "txt";
        const contentOutPath = outPath.split(".").filter(x => x)[0] + "." + ext;

        ensureFileSync(contentOutPath);
        const outFile = Deno.createSync(contentOutPath);
        response.body.pipeTo(outFile.writable);
        console.log(`Wrote: ${contentOutPath}`);
        break;
    }
    console.log("Fell through", file.path)
}
console.log("Done!")

I also added a try around the Deno.removeSync because it can error if the out directory doesn't exist at all. With this we can get output with the correct extension.

All the rest

We stubbed out the "fell through" case but let's add that in so we get out static assets.

//static.js - replacing console.log("Fell through", file.path)

const outPath = join(Deno.cwd(), outDir, relative(baseDir, file.path));
ensureFileSync(outPath);
Deno.copyFileSync(file.path, outPath);
console.log(`Wrote: ${outPath}`);

It's pretty easy. If there's no responder we assume it's static content and just copy it.

Handlers that produce files with extensions

One more ergonomic thing. Let's imagine from client js we try to fetch /nested/hi3, this won't work in static mode because the file is now called /nested/hi3.txt . This might not be a huge issue, we'll pretty much always need to make some changes to work with static file servers because they only tend to recognize index.html (this is again why most server frameworks probably forego weird cases like static endpoints) but could we get /nested/hi3.txt to work in server-mode as well? We can probably allow the sub-extensions to flow through so that file path /nested/hi3.txt.server.ts can serve content for url /nested/hi3.txt .

Since this is getting a bit complicated I'll make some tests.

export function match(path){
    return /\.server\.(jsx?|tsx?)$/.test(path);
}

export function defaultPaths(barePath) {
    return `${barePath}{,.*}.server.{js,jsx,ts,tsx}`;
}

//tests/responder/server-responder.test.js
import { assertEquals } from "https://deno.land/std@0.207.0/assert/mod.ts";
import * as serverResponder from "../responders/server-responder.js";
import { globToRegExp } from "https://deno.land/std@0.207.0/path/glob_to_regexp.ts";

Deno.test("Server Responder: Matchers", () => {
    [
        ["hi.js", false],
        ["hi.jsx", false],
        ["hi.ts", false],
        ["hi.tsx", false],
        ["hi.server.js", true],
        ["hi.server.jsx", true],
        ["hi.server.ts", true],
        ["hi.server.tsx", true],
        ["nested/hi.js", false],
        ["nested/hi.ts", false],
        ["nested/hi.jsx", false],
        ["nested/hi.tsx", false],
        ["nested/hi.server.js", true],
        ["nested/hi.server.ts", true],
        ["nested/hi.server.jsx", true],
        ["nested/hi.server.tsx", true],
        ["hi.txt.js", false],
        ["hi.txt.jsx", false],
        ["hi.txt.ts", false],
        ["hi.txt.tsx", false],
        ["hi.txt.server.js", true],
        ["hi.txt.server.jsx", true],
        ["hi.txt.server.ts", true],
        ["hi.txt.server.tsx", true],
        ["nested/hi.txt.js", false],
        ["nested/hi.txt.jsx", false],
        ["nested/hi.txt.ts", false],
        ["nested/hi.txt.tsx", false],
        ["nested/hi.txt.server.js", true],
        ["nested/hi.txt.server.jsx", true],
        ["nested/hi.txt.server.ts", true],
        ["nested/hi.txt.server.tsx", true],
        ["hi.txt", false],
        ["hi.server.txt", false]
    ].forEach(([input, expected]) => {
        assertEquals(serverResponder.match(input), expected);
    });
});

Deno.test("Server Responder: Default Paths", () => {
    [
        ["hi.js", false],
        ["hi.jsx", false],
        ["hi.ts", false],
        ["hi.tsx", false],
        ["hi.server.js", true],
        ["hi.server.jsx", true],
        ["hi.server.ts", true],
        ["hi.server.tsx", true],
        ["hi.txt.js", false],
        ["hi.txt.jsx", false],
        ["hi.txt.ts", false],
        ["hi.txt.tsx", false],
        ["hi.txt.server.js", true],
        ["hi.txt.server.jsx", true],
        ["hi.txt.server.ts", true],
        ["hi.txt.server.tsx", true],
        ["hi.txt", false],
        ["hi.server.txt", false]
    ].forEach(([input, expected]) => {
        const regex = globToRegExp(serverResponder.defaultPaths("hi"));
        assertEquals(regex.test(input), expected);
    });
});

This should make it more clear which things we want to match. We'll also need to update the probStat to use globs (this actually makes it simpler):

//utils/fs-util.js
import { expandGlob } from "https://deno.land/std@0.207.0/fs/expand_glob.ts";

export async function probeStat(filepaths){
    for(const filepath of filepaths){
        for await(const fileInfo of expandGlob(filepath)){
            return fileInfo;
        }
    }
}

Here's the new matching. Keep in mind match is a regex and defaultPaths is a glob. We can then update server.js with the new changes.

//server.js
import { typeByExtension } from "https://deno.land/std/media_types/mod.ts";
import { toFileUrl, resolve, join } from "https://deno.land/std@0.205.0/path/mod.ts";
import { probeStat } from "./utils/fs-utils.js";

const baseDir = "./routes";
const responders = await Promise.all(Array.from(Deno.readDirSync("./responders")).map(f => import(toFileUrl(resolve(join(Deno.cwd(), "./responders"), f.name)))));

Deno.serve(async req => {
    const url = new URL(req.url);
    let inputPath = url.pathname;
    const potentialFilePaths = [];

    //normalize path
    if (inputPath.endsWith("/")) {
        inputPath += "index";
    }
    if(!inputPath.includes(".")){potentialFilePaths.push(baseDir + inputPath + ".html");

    } else {
        //basic path
        const path = baseDir + inputPath;
        potentialFilePaths.push(path);
    }

    potentialFilePaths.push(...responders.flatMap(responder => responder.defaultPaths(baseDir + inputPath)));

    //find
    const fileMatch = await probeStat(potentialFilePaths);
    if(!fileMatch){
        return new Response("Not Found", { status: 404 });
    }

    const filePath = fileMatch.path;

    for(const responder of responders){
        if(await responder.match(filePath)){
            return responder.default(filePath, req);
        }
    }

    const file = await Deno.open(filePath);
    const ext = filePath.split(".").filter(x => x).slice(1).join(".");
    return new Response(file.readable, {
        headers: {
            "Content-Type": typeByExtension(ext)
        }
    });
});

I might actually want to make a base responder after all... but anyway this should work. However when we generate files we're still keeping the sub-extensions. So let's clean that up too.

//static.js - at the bottom
for await (const file of files) {
    if(file.isDirectory) continue;

    let handled = false;
    for(const responder of responders){
        if(!responder.match(file.path)) continue;
        const response = await responder.default(file.path);
        if(!response.ok){
            console.log(`Skipped: ${file.path} returned a non-successful status`);
            continue;
        }
        const outPath = join(Deno.cwd(), outDir, relative(baseDir, file.path.split(".")[0]));
        const ext = extension(response.headers.get("Content-Type")) ?? "txt";
        const contentOutPath = outPath + "." + ext;

        ensureFileSync(contentOutPath);
        const outFile = Deno.createSync(contentOutPath);
        response.body.pipeTo(outFile.writable);
        console.log(`Wrote: ${contentOutPath}`);
        handled = true;
    }

    if(!handled){
        const outPath = join(Deno.cwd(), outDir, relative(baseDir, file.path));
        ensureFileSync(outPath);
        Deno.copyFileSync(file.path, outPath);
        console.log(`Wrote: ${outPath}`);
    }
}
console.log("Done!")

Biggest thing to note is the outPath now hacks off the extensions and lets Content-Type to figure it out. This now means we can have something like this:

<!-- routes/fetch.html -->
<html>

<body>
    <h1>Fetch!</h1>
    <script type="module">
        const res = await fetch("/nested/hi.txt");
        const div = document.createElement("div");
        div.textContent = await res.text();
        document.body.append(div);
    </script>

</body>

</html>

And it will work in both server-mode and static-mode in-case you ever wanted to do that (I'm guessing not really but it was a neat experiment).

React

We have two of the main responders done, the base one for static assets and the one for server handlers. Let's do react next. This should almost work and if you try it you'll get a mostly working result with react compiled to html and the static assets working but the page scripts will not because the transpile-responder is broken (specifically it was hanging indefinitely for me). But there's actually a little API discussion we can have before addressing that.

We only used a simple example for getServerProps where we just made some default props. In reality this should be based on the request because we mean to vary it by request. So let's make that more clear in react-responder change the call to getServerProps to take request.

//./responders/react-responder.js - getting server props
//we need to take in the request parameter on the default function first, then
const props = await mod.getServerProps(request);

The update the example

// routes/app.react.jsx - getServerProps
export function getServerProps(req){
    const name = new URL(req.url).searchParams.get("name");
    return {
        name: name ?? "[Unknown]",
        userId: 123,
        dob: "5/6/1990"
    };
}

Now we can actually see this in action (and be made uncomfortable at the potential XSS attack). So now that we've clarified the usage this means that this doesn't work for static sites. The way NextJS tackles this is with another function getStaticProps. This works largely the same way as getServerProps but it doesn't take parameters, there's only one context. We could certainly copy that. You could imagine adding that method to any react page component and calling it before the render method. This is fine, but you'll also need to let the react responder know to call that instead of getServerProps which means we'll probably need to pass in more data to the responder to let it know that we're rendering in a static context. So far we've been able to avoid the responders needing to know about static vs server context. I'd like to keep it that way if possible so instead we'll be a bit extra and try something different. We can use the same strategy as the server responder, we can add functions that generate the static request instead.

Now from an API design standpoint this is kinda nice, from an actual usage perspective it's probably more annoying to have to know how to fake a request to get the data you want rather than just pass it directly. In any case let's update app.react.jsx.

// routes/app.react.jsx
export function getServerProps(req){
    const name = new URL(req.url).searchParams.get("name");
    return {
        name: name ?? "[Unknown]",
        userId: 123,
        dob: "5/6/1990"
    };
}

export function getTitle(){
    return "My React App";
}

export function getRootComponent(){
    return ["./js/components/app.jsx", "App"];
}

export function getStaticRequest(path) {
    return new Request(path + `?name=Bob Boberson`, {
        method: "get"
    });
}

Then wire this up in the react-responder.js

// responders/react-responder - at the top of the default function
export default async function reactResponder(path, request){    
    const moduleImportPath = toFileUrl(resolve(Deno.cwd(), path));
    const mod = await import(moduleImportPath);

    if (!request){
        if(mod.getStaticRequest) {
            request = await mod.getStaticRequest(path);
        } else {
            return new Response("No Request", { status: 405 });
        }
    }

    const props = await mod.getServerProps(request);
    const title = await mod.getTitle?.();
    const [componentPath, exportName] = await mod.getRootComponent();
// ... etc

We check if we have a request and if not then we can make a default one or error out. Now we have our static generation with props!

Transpilation

So to actually make the JSX work we need transpilation. This doesn't quite work. One problem is that these files are output as text. Debugging this, the reason was because the content-type returned was text/javascript but the actual mime type is application/javascript, so updating that fixed that issue. There's still two other problems.

1) Imports like ./counter.jsx are not converted to ./counter.js so the paths misalign
2) The process seems to hang once done

The second issue is because esbuild tries to be smart. It's actually starting up itself in a process and leaving that process around incase we use it again so Deno never finishes. This behavior is not well documented either. The static generation definitely doesn't need it running once it's done so we need a way to turn it off. Unfortunately, this means we need to build up our plugin API some more with a phase to clean up stuff at the end because we don't want references to esbuild in static.js.

//responders/transpile-responder.js
import { transform, stop } from "https://deno.land/x/esbuild/mod.js";

export const name = "transpile";

const extensions = [
    "jsx",
    "ts",
    "tsx"
];

export function match(path) {
    const ext = path.split(".").filter(x => x).slice(1).join(".");
    return extensions.includes(ext);
}

export function defaultPaths(barePath) {
    return extensions.map(ext => barePath + "." + ext);
}

export default async function transpileResponder(path) {
    const codeText = await Deno.readTextFile(path);
    const transpiled = await transform(codeText, { loader: "tsx" });

    return new Response(transpiled.code, {
        headers: {
            "Content-Type": "application/javascript"
        }
    });
}

export function dispose(){
    stop();
}

//./server.js - right before the end
// ...stuff

for(const responder of responders){
    responder.dispose?.();
}
console.log("Done!")

But how to deal with the path re-writing? This actually a pretty hard problem. We can try to build up an importmap. We can compare the input file with the output file, if the paths don't match then we can add it to the importmap lists.

// static.js - between getting a response from the responder and ensuring the file path
//if(!response.ok){
    //console.log(`Skipped: ${file.path} returned a non-successful status`);
    //continue;
//}
    const ext = extension(response.headers.get("Content-Type")) ?? "txt";
    const filePathToRoot = relative(baseDir, file.path.split(".")[0] + "." + ext);
    const outPath = join(Deno.cwd(), outDir, filePathToRoot);

    const originalFilePathToRoot = relative(baseDir, file.path);
    if(originalFilePathToRoot !== filePathToRoot){
        importMap[originalFilePathToRoot] = filePathToRoot;
    }

//ensureFileSync(outPath);

Once built, we can also write this file (note that naming conflicts are possible if the user also creates an importmap with the same name and we'll ignore this edge-case). But how can we use this? We basically need to shove it into all of our html files so that paths work. Sadly, we reach another impasse. This cannot be done without HTML parsing. So the next best thing is to import a DOM parser. For this we can use deno-dom.

//static.js before end
import { DOMParser } from "https://deno.land/x/deno_dom/deno-dom-wasm.ts"; //at top

//possible conflict if this exists...
Deno.writeTextFileSync(join(outDir, "importmap.json"), JSON.stringify({
    imports: importMap
}, null, 4));
console.log("Wrote importmap.json");


const htmlFiles = expandGlob(join(Deno.cwd(), outDir, "**/*html"));
for await(const htmlFile of htmlFiles){
    const content = Deno.readTextFileSync(htmlFile.path);
    const document = new DOMParser().parseFromString(content, "text/html");
    const script = document.createElement("script");
    script.setAttribute("type", "importmap");
    script.setAttribute("src", "/importmap.json");
    if(document.head.childNodes.length > 0){
        document.head.insertBefore(script, document.head.childNodes[0]);
    } else {
        document.head.appendChild(script);
    }
    Deno.writeTextFileSync(htmlFile.path, document.documentElement.outerHTML); 
}

console.log("Done!")

What we can do is read in the html files with a DOMParser and shove the importmap in there and then write it back out. This is inefficient but probably the easiest way to get around it. Now we have a couple more problems. Chrome doesn't support external importmaps, so we actually have to inline this, but this is kinda good because we have one less network request and don't have to deal with the file conflict.

- Deno.writeTextFileSync(join(outDir, "importmap.json"), JSON.stringify({
-   imports: importMap
- }, null, 4));
- console.log("Wrote importmap.json");

//in html loop
- script.setAttribute("src", "./importmap.json");
+ script.innerText = JSON.stringify({ imports: importMap }); //this is for demo and inefficent, we can serialize once and insert

Still not enough because Chrome can't support multiple importmaps. So we'll have to merge them.

//static.js - at the bottom
const htmlFiles = expandGlob(join(Deno.cwd(), outDir, "**/*html"));
for await(const htmlFile of htmlFiles){
    const content = Deno.readTextFileSync(htmlFile.path);
    const document = new DOMParser().parseFromString(content, "text/html");
    const script = document.createElement("script");
    script.setAttribute("type", "importmap");

    const importMapEl = document.querySelector("script[type=importmap]");
    if(importMapEl) {
        const oldImportMap = JSON.parse(importMapEl.innerText);
        Object.assign(oldImportMap.imports, importMap.imports);
        script.innerText = JSON.stringify(oldImportMap);
        importMapEl.remove();
    } else {
        script.innerText = JSON.stringify(importMap);
    }

    if(document.head.childNodes.length > 0){
        document.head.insertBefore(script, document.head.childNodes[0]);
    } else {
        document.head.appendChild(script);
    }
    Deno.writeTextFileSync(htmlFile.path, document.documentElement.outerHTML); 
}
console.log("Done!")

I move them all to the top just to avoid issues, the old one also has to be removed. But that's still not enough! Since I'm using Windows the paths in the import map need to be normalized from backslashes to forward slashes. We've run into this enough that I'm going to create a helper function (surprisingly Deno does not provide this directly...):

import { SEP } from "https://deno.land/std@0.208.0/path/windows/mod.ts";
//fs-utils.js
export function normalizeSlashes(path){
    return path.replaceAll(SEP, "/");
}

//static.js where we do the import mapping
if(originalFilePathToRoot !== filePathToRoot){
    importMap.imports[normalizeSlashes(originalFilePathToRoot)] = normalizeSlashes(filePathToRoot);
}

And it's still not enough. The paths also need to be either relative or absolute, but ours look like components/counter.js. I don't really want to figure out which path is relative to each document, so let's do it to the root.

//static.js where we do the import mapping
if(originalFilePathToRoot !== filePathToRoot){
    importMap.imports["/" + normalizeSlashes(originalFilePathToRoot)] = "/" + normalizeSlashes(filePathToRoot);
}

We know that these paths are relative to root and the domain base should be the root on the browser side (we have no concept of a context path). With this we have enough to finally statically generate the site with working js!

Conclusion

This turned out to be harder than I thought again. Just dealing with the file path conversions is a lot to get right. I am extremely disappointed we needed to add a heavy DOMParser package. A better way to deal with that would be to re-write the extensions and this is something that most transpilation tools do not support (they typically want to bundle the code and remove import statements entirely). So doing this might require something custom. Again we see all the problems that ts and jsx cause us and how we need sophisticated build tools to work around them. I'm just glad we could do it in a mostly web-centric way so we didn't need to learn new APIs.

The Code

• https://github.com/ndesmic/dev-server/pull/3