Note: This chapter was split into 2 parts because the second half deals with a different topic I thought would be easy be really wasn't.
Now that things are starting to get a bit complex, manual testing isn't really an option, it's simply too easy to break things so I'd like a way to confirm past features still work after refactors and such. Unfortunately, while I've billed this series as having no dependencies this is one where it really makes no sense to write from scratch (that could be it's own series though). So we'll use playwright for this task as I've found it to work well in the past and it's conforms pretty closely to puppeteer which I've also used a bit.
One small issue is that I'm not using node. Deno has support for most things in node but this one is a little tricky because it does some weird things. If you are using node you can skip a lot of the next section which helps deal with playwright's idiosyncrasies.
Install Playwright with Deno
Firstly we start with:
deno add npm:playwright
deno add npm:@playwright/test
This will install playwright and the associated test framework. playwright has other setup stuff it must do like download browsers and such so we also need to run
deno run playwright install
You might want to slap a -A on that because it's going to do a lot of scary file work.
We'll setup out new test in a folder called "visual-tests":
//basic.spec.js
import { test, expect } from "@playwright/test";
test("demo loads", async ({ page }) => {
await page.goto("localhost:4507");
await expect(page).toHaveTitle("Geo");
});
If we try to run it, it won't work for a few reasons. One we need to give it permissions because that's how Deno works. Sadly, because it's accessing lots of stuff outside of the project folder you can't really scope the permissions down, so I gave up and gave it all read permissions and write permissions. This is bad, but it's a node package and they do that. Here's the deno.json:
{
"permissions": {
"playwright": {
"env": true,
"sys": ["osRelease", "cpus"],
"read": true,
"write": true,
"run": true
}
},
"nodeModulesDir": "auto",
"tasks": {
"start": "deno run -A https://deno.land/std/http/file_server.ts",
"test:one": "deno test ./js/utilities/buffer-utils.test.js",
"test:visual": "deno run -P=playwright npm:playwright test visual-tests/"
},
"imports": {
"@playwright/test": "npm:@playwright/test@^1.55.0",
"@std/expect": "jsr:@std/expect@^1.0.16",
"@std/testing": "jsr:@std/testing@^1.0.14",
"playwright": "npm:playwright@^1.55.0"
}
}
You will need nodeModulesDir: "auto" because playwright wants to use the node_modules directory to do stuff. Now as long as the tests are scoped to the visual-tests directory so they don't pick up the unit-tests it should run with:
deno task test:visual
As long as the server is running this should work.
Use a playwright config
Let's use a playwright.config.js instead as this will give us access to more features.
//deno.json
{
"permissions": {
"playwright": {
"env": true,
"sys": ["osRelease", "cpus"],
"read": true,
"write": true,
"run": true,
"net": true
}
},
"nodeModulesDir": "auto",
"tasks": {
"start": "deno run -A https://deno.land/std/http/file_server.ts",
"test:one": "deno test ./js/utilities/buffer-utils.test.js",
"test:visual": "deno run -P=playwright npm:playwright test --config ./visual-tests/playwright.config.js"
},
"imports": {
"@playwright/test": "npm:@playwright/test@^1.55.0",
"@std/expect": "jsr:@std/expect@^1.0.16",
"@std/testing": "jsr:@std/testing@^1.0.14",
"playwright": "npm:playwright@^1.55.0"
}
}
We need some more permissions for net and then we will point it at the playwright config in the visual-test folder. We no longer need the path as the config will contain that.
//./visual-tests/playwright.config.js
import { defineConfig } from '@playwright/test';
export default defineConfig({
testMatch: './visual-tests/**/*.spec.js',
// Run your local dev server before starting the tests
webServer: {
command: "deno task start",
url: 'http://localhost:4507',
reuseExistingServer: "http://localhost:4507",
stdout: 'ignore',
stderr: 'pipe',
},
});
This will filter to just the .spec.js tests in the visual-tests folder. Keep in mind since the config is in the visual-tests folder, everything is relative to that. The web server command will run if none is found on localhost:4507 which is where the deno file server I've setup for dev and test is.
This will not actually work, at least on the current Deno on windows. There seems to be some problem with the way the stio streams get picked up by the runner. We need to remove the stdout and stderr settings in the config. Then we need to wrap the deno commands up:
#./visual-tests/start.ps1
#wrapper for deno webserver on windows because playwright doesn't like to run it directly
deno run -A https://deno.land/std/http/file_server.ts
You could also use cmd if powershell isn't your speed.
#!/usr/bin/env sh
#./visual-tests/start.sh
#wrapper for deno webserver on mac/linux because playwright doesn't like to run it directly
deno run -A https://deno.land/std/http/file_server.ts
//./visual-tests/playwright.config.js
import { defineConfig } from '@playwright/test';
export default defineConfig({
testMatch: '**/*.spec.js',
// Run your local dev server before starting the tests
webServer: {
command: process.platform === 'win32'
? `powershell -ExecutionPolicy ByPass -File "start.ps1"`
: `sh -c "./start.sh"`,
url: 'http://localhost:4507',
reuseExistingServer: !process.env.CI,
},
});
This will wrap the process and lets it work and work on multiple platforms.
Writing a test
I'm going to group each test with an html page which will render the test. So each folder under visual-tests will be one test with a .spec.js file and a .html file.
<!-- ./visual-tesats/should-render-teapot/should-render-teapot.html -->
<!doctype html>
<html lang="en">
<head>
<title>Geo - Should render teapot</title>
</head>
<body>
<h1>Geo - Should render teapot</h1>
<wc-geo>
<geo-camera key="main" position="0, 1, 2"></geo-camera>
<geo-texture key="red-fabric" src="../../img/red-fabric/red-fabric-base.jpg"></geo-texture>
<geo-texture key="red-fabric-roughness" src="../../img/red-fabric/red-fabric-roughness.jpg"></geo-texture>
<geo-texture key="gold" color="0, 0, 0, 1"></geo-texture>
<geo-material key="red-fabric" roughness-map="red-fabric-roughness" albedo-map="red-fabric"></geo-material>
<geo-material key="gold" roughness="0.2" metalness="1" base-reflectance="1.059, 0.773, 0.307" albedo-map="gold"></geo-material>
<geo-group key="teapot-rug-1" pipeline="main">
<geo-mesh
key="teapot"
normalize
bake-transforms
reverse-winding
src="../../objs/teapot.obj"
resize-uvs="2"
material="gold"
attributes="positions, normals, uvs">
</geo-mesh>
<geo-surface-grid
key="rug"
row-count="2"
col-count="2"
translate="0, -0.25, 0"
bake-transforms
material="red-fabric"
attributes="positions, normals, uvs">
</geo-surface-grid>
</geo-group>
<geo-light key="light1" type="directional" color="1, 1, 1, 1" direction="0, -1, 1" casts-shadow></geo-light>
</wc-geo>
<script src="../../js/components/wc-geo.js" type="module"></script>
</body>
</html>
//./visual-tesats/should-render-teapot/should-render-teapot.spec.js
import { test, expect } from "@playwright/test";
test("Should render teapot", async ({ page }) => {
await page.goto("http://localhost:4507/visual-tests/should-render-teapot/should-render-teapot.html");
await expect(page).toHaveTitle("Geo - Should render teapot");
await expect(page).toHaveScreenshot();
});
Now if we run deno task test:visual it should start the webserver if not already running and run the tests. At this point we have two problems. One is that there is no screenshot, but that will be easily solved when one is created the first time. The bigger issue is the page doesn't render correctly because the path to the pbt.wgsl was hardcoded such that it can't be accessed from the new folder.
The path we are looking for is in js/engines/gup-engine/pipelines.js. What we should probably do is fix the path lookup so it is relative to the module, rather than relative to the page (this is closer to how ESM behaves and means we can still hardcode it as long as the structure doesn't change).
//pipelines.js
- const shaderModule = await uploadShader(device, import.meta.resolve("./shaders/pbr.wgsl"));
+ const shaderModule = await uploadShader(device, import.meta.resolve("../../../shaders/pbr.wgsl"));
- const shaderModule = await uploadShader(device, "./shaders/shadow-map.wgsl");
+ const shaderModule = await uploadShader(device, import.meta.resolve("../../../shaders/shadow-map.wgsl"));
We also need to move the background pipeline in here since we never got around to that.
//pipelines.js
export async function getBackgroundPipeline(device) {
const vertexBufferDescriptor = [{
attributes: [
{
shaderLocation: 0,
offset: 0,
format: "float32x2"
}
],
arrayStride: 8,
stepMode: "vertex"
}];
const shaderModule = await uploadShader(device, import.meta.resolve("../../../shaders/space-background.wgsl"));
const pipelineDescriptor = {
label: "background-pipeline",
vertex: {
module: shaderModule,
entryPoint: "vertex_main",
buffers: vertexBufferDescriptor
},
fragment: {
module: shaderModule,
entryPoint: "fragment_main",
targets: [
{ format: "rgba8unorm" }
]
},
primitive: {
topology: "triangle-list"
},
depthStencil: {
depthWriteEnabled: true,
depthCompare: "less-equal",
format: "depth32float"
},
layout: "auto"
};
return device.createRenderPipeline(pipelineDescriptor);
}
I'm not going to explicitly write out how to update gpu-engine.js for this.
Anyway once the code is fixed we can run the test. It will create a base image from which to compare future runs. Unfortunately, we still have more problems. If you actually look at the image you'll see the screen is white, but if you actually open it in Chrome it renders normally. This is because playwright doesn't enable the GPU so it's not actually rendering anything!
Debugging
One thing I found helpful is to have a baseline test that makes sure the environment itself is setup correctly.
//basic.spec.js
import { test, expect } from "@playwright/test";
test("Environment should be setup correctly", async ({ page }) => {
page.goto("chrome://gpu");
await expect(page).toHaveTitle("GPU Internals");
await page.evaluate(async () => {
document.querySelector("info-view").shadowRoot.querySelector(".info-table:first-child tr:first-of-type").style.display = "none";
});
await expect(page.getByText('WebGPU: Hardware accelerated')).toBeVisible();
await expect(page).toHaveScreenshot({ maxDiffPixelRatio: 0.05 });
});
This basically ensures we can hit a page in the browser like our previous samples, but it also takes a screen shot of chrome://gpu which you can inspect to see what GPU settings are actually enabled and if they aren't displaying correctly. We look for "WebGPU: Hardware accelerated" so we know that it's turned on. The evaluate script hides a row that shows the current date which will make the test always fail but we also increase the maxDiffPixelRatio because version changes can make small text changes too. You can play around with the tolerances or hide other elements.
To enable the GPU we need to set a flag.
//./visual-tests/playwright.config.js
import { defineConfig } from '@playwright/test';
export default defineConfig({
use: {
channel: "chrome",
launchOptions: {
// force GPU hardware acceleration
// (even in headless mode)
args: [
'--enable-gpu'
]
}
},
testMatch: '**/*.spec.js',
// Run your local dev server before starting the tests
webServer: {
command: process.platform === 'win32'
? `powershell -ExecutionPolicy ByPass -File "start.ps1"`
: `sh -c "./start.sh"`,
url: 'http://localhost:4507',
reuseExistingServer: !process.env.CI,
},
});
Now running the tests should produce a screenshot of the teapot that we can compare future versions against. If you need to regenerate the screenshot run deno task test:visual --update-snapshots. Note that even after the fixes sometimes I found that playwright just kinda dies on waiting for a pipe or unexpected closure. I don't think much can be done about that on our end.
Clipping just the element
The screen shots are of the page but the rest of the page really doesn't matter. We can alter the test slightly to just get the element.
//./visual-tesats/should-render-teapot/should-render-teapot.spec.js
import { test, expect } from "@playwright/test";
test("Should render teapot", async ({ page }) => {
await page.goto("http://localhost:4507/visual-tests/should-render-teapot/should-render-teapot.html");
await expect(page).toHaveTitle("Geo - Should render teapot");
const canvas = await page.locator("canvas");
await expect(canvas).toHaveScreenshot();
});
Note that I found that playwright took screenshots that were 1px taller than the actual element. Clipping based on the bounding box didn't help so I'm not really sure what to do about that, just some bug in the underlying libraries I guess.
Waiting for the element
Sometime you might need to wait for the first render because the screenshot will happen too fast. One way is to add an element to the page once it's finished rendering.
import { test, expect } from "@playwright/test";
test("Should transform group mesh", async ({ page }) => {
await page.goto("http://localhost:4507/visual-tests/should-transform-group-mesh/should-transform-group-mesh.html");
await expect(page).toHaveTitle("Geo - Should transform group mesh");
const canvas = await page.locator("canvas");
const rendered = await page.locator(".rendered"); //llok for this
expect(rendered).toBeVisible();
await expect(canvas).toHaveScreenshot();
});
We can add this from the element:
<script>
const geo = document.querySelector("wc-geo");
geo.onRender = () => {
if (!document.querySelector(".rendered")) {
const div = document.createElement("div");
div.className = "rendered";
div.textContent = "rendered";
document.body.append(div);
}
}
</script>
We'll also need to add the onRender callback to wc-geo:
//wc-geo.js
async connectedCallback() {
//...
this.engine = new Engine({
canvas: this.dom.canvas,
onRender: this.onRender
});
//...
}
onRender() {} //stub in case it's not used
And finally to the engine itself:
//gpu-engine.js
#onRender;
constructor(options) {
//...
this.#context = options.canvas.getContext("webgpu");
//...
}
render() {
this.renderShadowMaps();
//this.renderDepthBuffer(this.#shadowMaps.get("light1").createView());
this.renderScene();
this.#onRender?.();
}
Conclusion
This setup kinda works but isn't great. I still get lots of handle invalid errors when running tests though. Might have better luck with playwright if we switch back to node. One of my bigger concerns is the amount of code and performance of tests. We'll probably want a lot and I don't know how well this will scale. An idea might be to use deno's WebGPU capability itself to render the scenes for the tests which would remove the entire browser and ultimately much more flaky setup of attaching to the browser debug socket which is how Playwright -> Puppeteer work under the hood. This however would require that we rebuild the screenshot testing. Perhaps a future chapter could look at that.
In the next half, we'll finish the scene graph implementation so everything works more consistently and the code is a bit cleaner.
Top comments (0)