DEV Community

Cover image for AetherScan: Building a Real-Time 3D Reconstruction App with Fast3R, WebSockets, and React Three Fiber
Prashant Piprotar
Prashant Piprotar

Posted on

AetherScan: Building a Real-Time 3D Reconstruction App with Fast3R, WebSockets, and React Three Fiber

#machinelearning #python #react #webdev

The Problem

You've got a bunch of photos of an object taken from different angles. You want a 3D model. Most tools make you upload images, wait, download a file, then open it in a separate viewer.

I wanted something better — drop your images in a browser and watch the 3D point cloud build in front of you, in real time.

That's AetherScan.

AetherScan UI

What It Does

  1. Drag & drop images into the web UI
  2. Images are streamed to the backend via WebSocket
  3. Fast3R (Facebook Research's neural 3D reconstruction model) runs GPU inference
  4. Reconstructed 3D points stream back to the browser in real-time
  5. The point cloud renders live at 60fps using React Three Fiber
  6. Export the result as a .PLY file

No page reloads. No file downloads mid-process. Just upload → watch → export.

Architecture 

┌──────────────────────────────┐          ┌───────────────────────────────┐
│       Next.js Frontend       │          │       FastAPI Backend         │
│                              │          │                              │
│  DropZone ──► FileReader     │          │  WebSocket Endpoint          │
│              │               │          │      │                       │
│         base64 chunks        │◄──WS───►│  Decode & Save to /tmp       │
│              │               │          │      │                       │
│  usePointStream Hook ◄───────│          │  Fast3R (PyTorch/CUDA)       │
│      │                       │          │      │                       │
│  React Three Fiber Canvas    │          │  Stream points back          │
│      │                       │          │                              │
│  Live 3D Point Cloud         │          │                              │
└──────────────────────────────┘          └───────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

The entire pipeline is bidirectional over a single WebSocket — no REST calls for the heavy lifting.

Tech Stack

Layer Tech
Frontend Next.js 15, React 19, React Three Fiber, Tailwind CSS
Backend FastAPI, PyTorch, Fast3R (ViT-Large), CUDA
Transport WebSocket (binary-friendly JSON streaming)
Infra Docker Compose with NVIDIA GPU passthrough

The Interesting Engineering Bits

1. Streaming Points Over WebSocket

The backend doesn't wait for the full reconstruction to finish. As soon as points are computed, they're streamed in chunks of 100:

# backend/main.py
chunk_size = 100
for i in range(0, len(points), chunk_size):
    chunk = points[i:i+chunk_size]
    await websocket.send_json({
        "type": "points",
        "data": chunk
    })
    await asyncio.sleep(0.01)  # backpressure
Enter fullscreen mode Exit fullscreen mode

This gives the user instant visual feedback — points start appearing within seconds, even for large reconstructions.

2. Zero-Re-render Point Cloud Updates

Normally, React re-renders the entire component tree when state changes. With 100K+ points arriving in rapid chunks, that would be a disaster.

The usePointStream hook bypasses React's state entirely:

// frontend/hooks/usePointStream.ts
const positionsRef = useRef<Float32Array>(new Float32Array(maxPoints * 3));
const colorsRef    = useRef<Float32Array>(new Float32Array(maxPoints * 3));

const addPoints = useCallback((points: PointData[]) => {
    const positions = positionsRef.current;
    for (const point of points) {
        const idx = currentCount * 3;
        positions[idx]     = point.x;
        positions[idx + 1] = point.y;
        positions[idx + 2] = point.z;
        currentCount++;
    }

    // Update GPU buffer directly — no setState
    positionAttributeRef.current.needsUpdate = true;
    geometryRef.current.setDrawRange(0, currentCount);
}, []);
Enter fullscreen mode Exit fullscreen mode

Key decisions:

  • Float32Array in refs, not state — no re-renders
  • DynamicDrawUsage tells Three.js the buffer will change often
  • setDrawRange incrementally reveals points without re-uploading the entire buffer
  • Direct needsUpdate flag pushes data to the GPU

The result: 1M+ points at 60fps with zero React re-render overhead.

3. Fast3R Inference with Dynamic Resolution

Consumer GPUs have limited VRAM. The backend dynamically adjusts the input resolution based on image count to prevent CUDA OOM:

# backend/services/reconstruction.py
if num_images <= 3:
    img_size = 512
elif num_images <= 6:
    img_size = 384
else:
    img_size = 256
Enter fullscreen mode Exit fullscreen mode

It also uses mixed precision (float16) on CUDA for 2x memory savings:

precision = "16-mixed" if self.device == "cuda" else "32"
Enter fullscreen mode Exit fullscreen mode

4. Eager Model Loading at Startup

The Fast3R ViT-Large model is ~1.8 GB. Instead of lazy-loading on the first request (and making the user wait), the model loads eagerly at server startup:

@app.on_event("startup")
async def startup_event():
    service = get_reconstruction_service()
    await loop.run_in_executor(None, service.initialize)
Enter fullscreen mode Exit fullscreen mode

A /status endpoint returns 503 until the model is ready, so health checks and load balancers know when the service can accept traffic.

5. Client-Side PLY Export

The export happens entirely in the browser — no round-trip to the server:

const header = `ply
format ascii 1.0
element vertex ${points.length}
property float x
property float y
property float z
property uchar red
property uchar green
property uchar blue
end_header
`;

const blob = new Blob([header + vertices], { type: 'text/plain' });
Enter fullscreen mode Exit fullscreen mode

The user gets a standard .PLY file compatible with MeshLab, Blender, CloudCompare, and other 3D tools.

Running It Yourself

Docker (Recommended) 

git clone https://github.com/prashplus/AetherScan.git
cd AetherScan
docker-compose up
Enter fullscreen mode Exit fullscreen mode

Requires NVIDIA GPU + NVIDIA Container Toolkit.

Local Development 

# Backend (Python 3.11 + conda)
conda create -n aether python=3.11
conda activate aether
cd backend && pip install -r requirements.txt
git clone --recursive https://github.com/facebookresearch/fast3r.git ../fast3r
pip install -e ../fast3r
uvicorn main:app --reload --host 0.0.0.0 --port 8000

# Frontend
cd frontend && npm install && npm run dev
Enter fullscreen mode Exit fullscreen mode

Apple Silicon Support

The backend auto-detects the best available device. On a Mac with Apple Silicon, it falls back to MPS:

def get_best_device() -> str:
    if torch.cuda.is_available():
        return "cuda"
    if torch.backends.mps.is_available():
        return "mps"
    return "cpu"
Enter fullscreen mode Exit fullscreen mode

Lessons Learned

  1. WebSocket > REST for streaming workloads. Chunked HTTP responses work, but WebSockets give you true bidirectional communication — the client can send images and receive points on the same connection.

  2. Bypass React for hot data paths. React's reconciliation is great for UI state. For high-throughput data like point clouds, write directly to GPU buffers via refs.

  3. Eager model loading pays off. A 30-second startup delay is much better than a 30-second hang on the first user request. Health checks and readiness probes make this safe in production.

  4. Dynamic resolution prevents OOM. Instead of failing with a CUDA out-of-memory error, scale down input resolution based on the workload. Users get results at slightly lower fidelity rather than a crash.

  5. Ship a working fallback. If the ML model fails to load (no GPU, no internet for weights), the system falls back to demo points. The UI never breaks.

What's Next

  • Binary WebSocket frames — switch from JSON to raw Float32Array over binary frames for 5-10x bandwidth reduction
  • Level-of-Detail (LOD) — progressive point cloud rendering for very large scenes
  • Multi-object support — segment and reconstruct individual objects from a scene
  • Mesh generation — go from point clouds to textured meshes using Poisson surface reconstruction

Links

If you found this useful, star the repo and let me know what you build with it!

Top comments (1)

Collapse
 
zyvop profile image
ZyVOP •

I was particularly impressed by the way you wove together complex technologies like Fast3R, WebSockets, and React Three Fiber to create a seamless real-time 3D reconstruction experience in AetherScan. You've clearly done some groundbreaking work here, and I think your unique approach would really resonate with our community at ZyVOP - would you be interested in cross-posting your article with us to reach an even wider audience?