Katsuo Imai

Posted on May 24

Building PROTO RECON — From Vague Idea to a Browser-Based Tactical HUD

#javascript #tensorflow #vite #vibecoding

Introduction

This post walks through how PROTO RECON — an experimental app that combines phone sensors with in-browser ML — evolved from pre-coding requirements to its current implementation. I've verified the UI behavior, but I'm still at the stage of reading the source code for the first time. Here I organize the original requirements and an AI-generated source map as a foundation for deeper dives in future posts.

Live demo (experimental): Proto Recon (Cloudflare Workers)
Repository: github.com/katsutoshi0katsutoshi/proto_recon
Sandbox for isolated experiments: CodePen / @wlzpaovm-the-bashful

Note: This is still in testing. Performance tuning for heavy workloads hasn't been done yet — use at your own risk.

Where It Started

It began with a vague question: Can we build a digital user experience on top of the real world? The seed of the idea was whether today's phone sensors could overlay something like the robot-view or cockpit-view perspectives from old sci-fi and games onto live scenery.

Before writing code, I summarized the requirements, turned functional specs into prose through dialogue with AI, and then started development.

Technology Choices (Before Implementation)

Area	Adopted / Considered	Notes
Object detection	TensorFlow.js (COCO-SSD / BlazeFace)	"Object recognition" as the core of the robot-view feel, fully in the browser
Map	Leaflet	Familiar "you are here" display from games
Noise / effects	Initially GLSL / three.js	Wanted to reuse past effect-building experience

The noise layer was added later. When I had Cursor implement it, three.js was skipped in favor of Canvas 2D and CSS for CRT-style effects. Writing directly without extra modules kept things lighter — a decision that has paid off on real devices.

Gyro, compass, altitude (on supported devices), GPS, and similar sensors were also in scope, wired to the HUD and minimap.

Requirements at the Time (Summary)

The planning document — built through repeated conversations with AI — was titled Tactical Recon & Guidance Terminal "PROJECT: LOCK-ON (working title)". At that point, sensor accuracy led us to assume Phase 2 would require a Unity (native) rewrite. After coding with Cursor, however, things moved better than expected even without three.js, and the outlook shifted to Phase 1 might be achievable on the web alone.

Core Concept

Visuals: Monochrome green military-terminal UI, scan lines, noise
AI: Real-time object detection for targeting and lock-on
Immersion: Gyro, altitude, BEEP sounds — instrument and audio feedback
Future: VR-goggle HUD, gaze control (concept stage)

Plan vs. Current State

Item	Plan (as of Apr 2026)	Current (PROTO RECON)
Stack	Next.js + Three.js + TF.js	Vite + vanilla JS + TF.js (no Three.js)
Altimeter	Barometric sensor (native assumed)	Geolocation `altitude` (supported devices only)
Map	(lightly specified in the plan)	Leaflet minimap implemented
Missile effect	Tap to fire a virtual missile	Not implemented
Phase 2 Unity	Barometric altitude, haptics, etc.	Not started (web prototype continues)
Distance display	(mostly undefined in planning)	WebXR hit-test measurements only (no bbox-estimated distance)

The full planning document is collapsed below due to length.

Full planning document (PROJECT: LOCK-ON draft)

Tactical Recon & Guidance Terminal App — "PROJECT: LOCK-ON (working title)" Planning Document

This document summarizes a concept for a smartphone camera-filter app that blends retro-futuristic UI with modern AI.

1. Project Overview

A "play" app that recreates the experience of a military recon terminal from 1980s–90s sci-fi films and games, using live smartphone camera feed and AI.

2. Core Concept

Visuals: Monochrome green evoking old terminals. Retro feel via scan lines and noise.
AI interaction: Automatic targeting and lock-on through real-time object detection.
Immersion: Gyro, barometric sensor (altimeter), BEEP feedback.
Extensibility: HUD and gaze control with VR goggles (e.g. Hacosco) — concept stage.

3. Main Features (excerpt)

3.1 Video filter

Green overlay (night-vision goggle texture)
Edge bar graphs and scan indicators

3.2 AI lock-on

Automatic moving-object detection with green bounding-box tracking
SE accelerates when the target stays in the reticle center
Lock complete: box turns red, warning sound
Tap to fire a virtual missile (effect)

3.3 Sensor-linked instruments

Gyro: pitch and roll numeric display
Altimeter: relative altitude from barometric sensor (iPhone, etc.)

4. Development Phases (roadmap at the time)

Phase 1: Web prototype (Next.js + Three.js + TF.js). Shareable via URL.
Phase 2: Unity port. Barometric altitude, vibration.
Phase 3: VR/XR. Stereoscopic display, gaze lock-on.

5. Distribution Strategy

Dev logs on Qiita / Zenn, short-form video on social, posts on Reddit, etc.

Created: April 29, 2026 (planning stage)

How Development Proceeded

Coding was delegated to Cursor, adding features one prompt at a time until the current state. For prompts, I load the requirements document into Gemini and ask what kind of prompt would work best. I've checked JS behavior through the UI, but reading the source comes next. The file list and dependency diagram below were AI-generated and cleaned up for this article.

Source Layout

HTML (root level)

File	Role
`index.html`	Main screen. BIOS/splash, boot progress, camera feed, reticle/detection overlay, compass/altitude tape, minimap, CRT effects, etc. Loads `/src/main.js` as a module
`privacy.html`	Privacy policy (static page: collected data, purpose of use, etc.)
`oss-notice.html`	Static OSS / third-party license notice

`src/` — runtime files

Entry, config, UI

File	Role
`main.js`	App hub. Boot sequence, camera acquisition, start/stop of modules, HUD updates, user actions (start/stop, lock, .etc.)
`appConfig.js`	Dev tuning defaults (camera, CRT, object detection, face mask, minimap, motion linkage, live health, etc.)
`applyDevConfig.js`	Applies `devConfig.crt` as CSS variables on `document.documentElement` (vignette, scan lines, etc.)
`uiLexicon.js`	Single source for UI copy (splash, status, errors, minimap labels, etc.) including functions to apply text to HTML
`style.css`	Global layout, HUD, CRT, minimap, reticle styles (green military UI)

Video & AI inference

File	Role
`tfBackend.js`	One-time TensorFlow.js backend init (WebGPU → WebGL → CPU)
`personDetection.js`	COCO-SSD object detection. Bounding-box drawing, lock zone, target events, inference health notifications
`facePrivacy.js`	BlazeFace face detection and masking (privacy overlay)
`lockTrackKeys.js`	Stable IDs for targets in the lock zone (position quantization + match with previous frame)

Distance & AR

File	Role
`objectRange.js`	Distance to target on HUD — measured distances only (WebXR, etc.); no bbox-estimated distance
`webxrHitTestRange.js`	WebXR immersive-ar + hit-test for real-world distance (m) fed into `objectRange`

Navigation, map, sensors

File	Role
`compass.js`	Device heading (compass) wrapper. Heading parse, cardinal labels, iOS permission request
`miniMap.js`	Geolocation watch and minimap facade (position, heading, speed, altitude callbacks)
`miniMapLeaflet.js`	Leaflet map init, current-location marker (heading arrow), tile display
`motionHud.js`	Tilt β/γ from `DeviceOrientation` for the right vertical tape HUD
`motionStabilityLink.js`	Links tilt/shake to reticle/minimap parallax and notifications (including vibration)

Feedback & monitoring

File	Role
`liveHealth.js`	Lightweight live health monitoring (inference latency, consecutive failures, video stall) and HUD notifications
`audioFx.js`	Web Audio API for boot, detection, lock-on, error sounds, etc.

`src/` — tests only (not used on screen)

File	Role
`lockTrackKeys.test.js`	Unit tests for `lockTrackKeys.js`
`liveHealth.test.js`	Unit tests for `liveHealth.js`
`tfBackend.test.js`	Unit tests for `tfBackend.js`

Dependencies (overview)

flowchart TB
  index[index.html] --> main[main.js]
  main --> config[appConfig / applyDevConfig / uiLexicon]
  main --> cam[camera + HUD]
  main --> det[personDetection]
  main --> face[facePrivacy]
  det --> tf[tfBackend]
  face --> tf
  main --> map[miniMap]
  map --> leaf[miniMapLeaflet]
  main --> nav[compass / motionHud / motionStabilityLink]
  main --> range[objectRange]
  range --> xr[webxrHitTestRange]
  main --> health[liveHealth]
  main --> sfx[audioFx]

main.js ties almost everything together; appConfig.js settings and uiLexicon.js copy drive behavior and display across modules.

Closing — What to Read Next

Future posts will open the source and go deep feature by feature. The study roadmap below is the planned reading order (01 is done with this article's file map).

Source reading roadmap (32 parts — click to expand)

Stage 1 — Config and skeleton (no permissions needed)

#	Title / theme	Main files	Core tech	Common pitfalls
01	~~Full map — roles in `src/`~~	~~`-`~~	~~Which file owns what~~	~~Memorizing filenames without reading code~~
02	`devConfig` — behavior remote control	`appConfig.js`	Camera, detection, CRT key meanings	One change affects the whole app
03	Single source for UI copy — `uiLexicon`	`uiLexicon.js`, `index.html`	Centralized strings and error messages	Finding leftover hardcoded text
04	CRT tokens to CSS — `applyDevConfig`	`applyDevConfig.js`, `style.css`	JS config → CSS variables (`:root`)	Typo in variable name breaks styles
05	HTML skeleton — splash & HUD DOM	`index.html`	Layer stack (video / canvas / HUD)	`z-index` vs `hidden` attribute

Stage 2 ⭐⭐ — Look and boot "theater" (testable without camera)

UI animation and audio control without device permissions.

#	Title / theme	Main files	Core tech	Common pitfalls
06	Splash, BIOS, scramble text	`main.js`, `style.css`	First-run UI flow, timer control	Duplicate timers, garbled text
07	Boot overlay — progress and steps	`main.js`	Async status during startup	Failed step never transitions to `error`
08	Web Audio — play after click	`audioFx.js`	`AudioContext` and user gesture	Autoplay policy, silent bugs
09	Reticle SVG and fixed layout	`index.html`, `style.css`	Center-fixed CSS layout	Why `motionStabilityLink` was disabled
10	Live HUD frame and CRT noise	`style.css`, `main.js`	Scan-line effects over video	Rendering cost of noise `canvas`

Stage 3 ⭐⭐⭐ — Media, sensors, tapes (real device required)

#	Title / theme	Main files	Core tech	Common pitfalls
11	Camera — `getUserMedia` and rear preference	`main.js`	`facingMode`, stream acquisition	iOS permission timing, connection hang
12	Video presentation — `object-fit: cover`	`style.css`, `main.js`	Hidden → visible preview transition	Unintended camera zoom
13	Full boot sequence — `startCamera`	`main.js`	After tap: camera → model load	Failure logs when changing parallel vs sequential
14	Heading tape — continuous angle and N jump	`compass.js`, `main.js`	0°–360° unwrap, SVG control	Clipping at north (N), insufficient padding
15	Tilt tape — β / γ and `motionHud`	`motionHud.js`, `main.js`	DeviceOrientation	`deviceorientationabsolute` behavior differences
16	Altitude tape — GPS altitude callback	`miniMap.js`, `main.js`	Geolocation API	Devices where altitude is `null`
17	Minimap — Leaflet and OSM/CARTO	`miniMap.js`	External tile map, marker	Handling location denial
18	Speed HUD — km/h from `coords.speed`	`miniMap.js`	Speed from coordinate changes	Speed shows `--` when stationary

Stage 4 ⭐⭐⭐⭐ — Detection, privacy, ML

#	Title / theme	Main files	Core tech	Common pitfalls
19	TF.js backend — WebGPU→WebGL→CPU	`tfBackend.js`	Graphics API fallback	Unsupported devices, first compile, heat
20	COCO-SSD intro — `ensureModel` and throttling	`personDetection.js`	Model load, inference throttling	CDN load failure, dynamic `import()`
21	Detection box drawing — `drawVideoCover` and canvas	`personDetection.js`	Drawing aligned to `video` aspect	Coordinate drift, DPR
22	Lock-on — zone, dwell, key stabilization	`personDetection.js`	Tracking ID, smoothing	ID swaps, flicker
23	Distance — WebXR hit-test only	`objectRange.js`, `webxrHitTestRange.js`	Measured distance on HUD	No distance on unsupported devices
24	Face mosaic — BlazeFace and canvas compositing	`facePrivacy.js`	rAF + throttled inference	Draw cycle vs inference collision
25	Mosaic interval and heat — `intervalMs`	`facePrivacy.js`	Inference FPS vs load tradeoff	Device freeze, thermal throttling

Stage 5 ⭐⭐⭐⭐⭐ — Integration, ops, errors

#	Title / theme	Main files	Core tech	Common pitfalls
26	Live health — latency, video stall	`liveHealth.js`	Main-thread stall monitoring	False positives under load spikes
27	Startup error handling — timeout and `catch`	`main.js`	Async error gaps	Model load timeout
28	Second visit and cache	`vite.config.js`	PWA, Vite build	`failed to fetch dynamically imported module`
29	Revisit and restart — model disposal	`main.js`	Memory leak prevention	Second run extremely heavy
30	`motionStabilityLink` — why parallax was cut	`motionStabilityLink.js`	Parallax vs readability	Janky rendering
31	Integration — `main.js` orchestration	`main.js` (full)	Lifecycle management	Tight coupling breaks on touch
32	Retrospective — bug list and what's next	App overall	Settling on load mitigation	Defining "fatal bugs are gone"

The next post will start with roadmap #02 (appConfig.js).

DEV Community

Building PROTO RECON — From Vague Idea to a Browser-Based Tactical HUD

Introduction

Where It Started

Technology Choices (Before Implementation)