DEV Community

Matthew Armstrong
Matthew Armstrong

Posted on

Kinect 3D Scanner

WORK IN PROGRESS: MANY FEATURES MAY BE BROKEN.
Version 0.2.0-wip. Free for non-commercial use (PolyForm Noncommercial 1.0.0).
Copyright Matthew Armstrong (@The-Dorkknight). No warranty.

A single-file desktop app that turns a Kinect v1 (Xbox 360 / Kinect for
Windows) or Kinect v2 (Kinect for Windows v2 / Kinect for Xbox One)
sensor into a tabletop 3D scanner: live preview → scan while you turn the
object → build a mesh → save .stl / .obj / .ply. The camera type is
detected automatically.

It is a standalone sibling of the realsense-scanner
project (same author, same house style) -- it shares no code and no runtime
dependency with it. kinect_camera.py's camera classes were deliberately
written to match realsense-scanner's own camera-object contract, so that a
real, single-app integration (one scanner, RealSense or Kinect, picked from a
dropdown) is possible later -- but that integration is not done in this
release; the two projects just happen to be able to share a scanning engine
if that's ever wanted.

NONE of this has been run on real Kinect hardware

Being completely straight about this, because it matters more here than
usual: kinect_camera.py, camera_bridge.py, and this app's camera code
have never been run against a real Kinect v1 or v2 sensor, and have never
even been run against a real compile of libfreenect or libfreenect2.

They were written from the published APIs, header comments and examples of
those two (old, community-maintained, and by now largely unmaintained)
driver projects, without access to either piece of hardware or to a machine
set up to build their drivers. The depth-unit conversions, the registered-
depth fallback logic, the packet-pipeline selection, the frame formats --
all of it is "this is what the docs say the API looks like", not "this is
what came back from a sensor on my desk".

As of 0.2.0-wip there is a real automated test suite
(tests/test_kinect.py, 25 tests, run with pytest), but it runs against
hand-built stand-in modules (tests/fake_kinect_drivers.py) that mimic
what freenect and pylibfreenect2 are documented to do -- not against the
real driver libraries. It catches a real class of bug (two serious ones were
found and fixed this way -- see CHANGELOG.md's 0.2.0-wip entry) and proves
the app-side logic holds together under failure conditions a real Kinect is
known to produce (hangs, crashes, a stream that stalls). It does not
prove a real Kinect works, because nothing on either side of this code has
touched real hardware or a real driver build.

Treat the first real run on each platform as a debugging session, not a
victory lap.
If something doesn't import, doesn't detect the camera, or
returns a frame shaped wrong, that is the expected starting point, not a
surprise -- please open an issue with what you saw.

What has actually been tested

Area Status
The scan → align → mesh → save pipeline's logic (point clouds, ICP math, TSDF fusion, Poisson reconstruction) Ported from realsense-scanner's own camera-agnostic core, which was tested there -- but not re-run here, since there's no Kinect frame to feed it yet.
kinect_camera.py (Kinect v1 / v2 drivers) Passes tests/test_kinect.py (25 tests) against hand-built stand-ins for freenect/pylibfreenect2 that mimic their documented behaviour -- frame shapes/dtypes, depth-unit conversion, colour-buffer copying, un-mirroring, BGRX→RGB, pipeline fallback, and clear errors on every missing-driver/no-device/can't-open case tried. Not run against the real driver libraries or real hardware.
camera_bridge.py (child-process isolation + auto-restart) Passes the same suite's bridge tests against a fake camera made to crash, hang, answer with no picture, or simply not be there -- restart/backoff/give-up, heartbeat-vs-no-picture timeouts, range resend after a restart, and fast shutdown are all exercised. Not exercised against an actual libfreenect/libfreenect2 fault, since none has been observed firsthand.
The GUI (buttons, threading, the Tk event-queue plumbing) Exercised with the camera code unreachable (no hardware) -- menus and state transitions were checked by hand, not against a live feed.
whoareyousb.py / "Diagnose USB..." Tested on synthetic USB data only (see that project's own README); not against a real Kinect disconnect.

Run the test suite yourself with pip install pytest open3d (inside
kinect_env, or any Python the project's requirements.txt works in) then
pytest tests/. If something breaks on your camera, please open an issue.

Using it

  1. Start Camera. (The "Driver setup help..." button and --check-drivers tell you what's missing if this fails.)
  2. Start Scan, slowly turn the object or walk the camera round it. Stop Scan when done. Capture Frame takes a single frame instead.
  3. Build Mesh: Smooth surface (fuses depth images into one TSDF volume, recommended) or Watertight (Poisson; closes up unscanned gaps, for printing).
  4. Save Mesh... as STL/OBJ/PLY, in millimetres or metres.

What was simplified from realsense-scanner's alignment engine

QualityParams/quality_to_params and the meshing (build_fused_mesh,
build_mesh) are ported close to verbatim from realsense-scanner, since that
part of the pipeline only touches point clouds/meshes and is entirely
camera-agnostic. The frame-alignment step (ScanSession.register_and_merge)
is a deliberately simpler version:

  • Kept: sequential coarse-to-fine ICP (colored ICP from a motion- extrapolated and an identity starting guess, falling back to feature-based global registration when overlap is weak), chaining each new frame onto the most recently successfully aligned frame. A frame that doesn't line up at all is skipped rather than merged in the wrong place.
  • Dropped: loop closures, full pose-graph global optimisation, the tracked-camera-pose rescue/veto machinery (there's no picture-based tracking here at all), the frame-consistency outlier check, and per-pixel "areas that moved" masking.

This is a real loss of robustness on a long, drifty scan -- a scan that
loops back on itself won't get the loop-closure correction that removes
built-up drift, and there's no safety net for a frame that aligns "well" but
in the wrong place. It is also not yet exercised against anything but
synthetic reasoning, since there's no camera to test it with. It is the
deliberate trade-off this MVP makes for a much smaller, easier-to-debug
alignment path; see realsense-scanner's own realsense_scanner_app.py if you
want the fuller version to port back in later.

Explicitly out of scope for this MVP (not ported from realsense-scanner at
all): raw stream recording/replay, the clean-up passes in
stream_pipeline.py, sensor-check masking, infrared handling, and tracking-
based pose estimation.

Top comments (0)