Building an Orthodontic Progress App That Never Uploads Your Face

I recently shipped SmileProgress (Japanese name: 矯正ログ / Kyousei Log), an indie iOS app for people going through orthodontic treatment — braces, aligners, and everything in between. The core promise is unusual: you can track months or years of progress without ever putting a photo of your face on a server. Not mine. Not iCloud. Not anyone's.

This post is a technical write-up of the on-device anonymization pipeline that makes that promise possible. It uses only Apple's built-in frameworks — Vision, ImageIO, CoreImage — and stores everything in the user's local Documents directory.

If you're building anything privacy-sensitive on iOS and you want a working example that goes beyond "we hash your email," I hope this is useful.

Why the constraint matters

Orthodontic treatment takes 1–3 years. People want to see the change, but they almost never want to publish "open your mouth wide" photos of their own face to a cloud service. In the Japanese 矯正垢 (orthodontic accounts) community on Instagram and X, users routinely crop, blur, or paint over everything except the teeth before sharing.

So the design constraint was:

• The app records mouth photos for years, but never uploads the raw photo
• Only the mouth region — cropped, stripped of metadata — can be shared, and only when the user explicitly opts in
• No account. No sync server. No cloud storage bill I have to pay every month as an indie developer

All three requirements point at the same technical answer: do the anonymization on the device, right after capture.

The pipeline in one picture

The full flow inside the phone looks like this:

1. AVCaptureSession grabs a frame from the camera
2. VNDetectFaceLandmarksRequest finds the face and its landmarks
3. outerLips.normalizedPoints gives us the mouth contour
4. We compute a square crop with a 1.6× margin
5. CGImageDestination writes a fresh JPEG with no EXIF / GPS / TIFF metadata
6. PhotoStore.add(...) saves it into Documents/photos/ with a JSON sidecar

Nothing in that list touches the network.

Detecting only the mouth

VNDetectFaceLandmarksRequest returns all facial landmarks — eyes, nose, jaw, pupils, and (importantly for us) the outer lip contour.

import Vision

func detectMouth(in image: UIImage) async -> [CGPoint]? {
    guard let cgImage = image.cgImage else { return nil }

    return await withCheckedContinuation { continuation in
        let request = VNDetectFaceLandmarksRequest()
        request.revision = VNDetectFaceLandmarksRequestRevision3

        let handler = VNImageRequestHandler(
            cgImage: cgImage,
            orientation: image.cgImageOrientation
        )

        do {
            try handler.perform([request])
            guard let face = request.results?.first,
                  let outerLips = face.landmarks?.outerLips else {
                continuation.resume(returning: nil)
                return
            }
            continuation.resume(returning: outerLips.normalizedPoints)
        } catch {
            continuation.resume(returning: nil)
        }
    }
}

Two things are worth calling out in that snippet.

First, revision = VNDetectFaceLandmarksRequestRevision3 (available iOS 15+) is a huge accuracy jump over the older revisions. If you're serious about landmark work, don't rely on the default revision — pin the newest one that ships with your minimum target.

Second, outerLips.normalizedPoints is not in image coordinates. It's normalized to the detected face bounding box. Which brings us to the next surprise.

The coordinate system will trip you up

outerLips.normalizedPoints gives you (x, y) in the range 0..1, but that range is measured relative to the face bounding box. And Vision uses a bottom-left origin, while UIImage and CGImage use top-left. Convert both:

let faceBBox = face.boundingBox                 // normalized to full image, 0..1
let imgWidth = CGFloat(cgImage.width)
let imgHeight = CGFloat(cgImage.height)

func denormalize(_ p: CGPoint) -> CGPoint {
    let x = (faceBBox.origin.x + p.x * faceBBox.width) * imgWidth
    let y = (1.0 - (faceBBox.origin.y + p.y * faceBBox.height)) * imgHeight
    return CGPoint(x: x, y: y)
}

let lipPointsInImage = outerLips.normalizedPoints.map(denormalize)

The Y-axis flip is the classic Vision gotcha. Every couple of months I forget it and spend twenty minutes debugging a phantom rotation.

The 1.6× margin trick

Now we have the lip contour in image pixels. The naive move is: bounding-box the points and crop exactly that rectangle. That looks awful. A raw lip crop with no surrounding skin feels clinical and slightly uncanny.

I experimented with margins from 1.0× to 2.4× and settled on 1.6× — enough to include a strip of skin, chin, and philtrum, not so much that it starts leaking useful identifying features:

let xs = lipPointsInImage.map(\.x)
let ys = lipPointsInImage.map(\.y)
guard let minX = xs.min(), let maxX = xs.max(),
      let minY = ys.min(), let maxY = ys.max() else { return nil }

let lipBBox = CGRect(
    x: minX, y: minY,
    width: maxX - minX, height: maxY - minY
)

let center = CGPoint(x: lipBBox.midX, y: lipBBox.midY)
let side = lipBBox.width * 1.6
let squareRect = CGRect(
    x: center.x - side / 2,
    y: center.y - side / 2,
    width: side, height: side
)

Cropping to a square (not the lip aspect ratio) matters for a later feature — a Before/After slider that only works when frames share dimensions.

When the mouth is wide open, landmarks fly off

Real orthodontic photos are almost always taken with the mouth wide open to expose the teeth. Vision's landmark model was clearly trained on more neutral expressions, and when the jaw drops significantly, outerLips can return points that are 20–30% off the actual lip line.

The heuristic I ended up with is a corner-angle check: if the left and right corners of the lip contour form an angle that couldn't plausibly be a mouth on a roughly upright face, fall back to a different detector.

let leftCorner  = lipPointsInImage.min(by: { $0.x < $1.x })!
let rightCorner = lipPointsInImage.max(by: { $0.x < $1.x })!
let angle = atan2(
    rightCorner.y - leftCorner.y,
    rightCorner.x - leftCorner.x
)

if abs(angle) > .pi / 3 {
    return await fallbackDetection(cgImage)
}

The fallback path uses simple HSV color analysis — detecting the teeth (bright, slightly blue-shifted white) and the lip (red-shifted) directly in pixel space. It's lower quality than Vision, but it recovers cases where VNDetectFaceLandmarksRequest returns garbage or nothing at all.

Close-up shots have no face at all

The other reality check: users import old photos from their camera roll, and orthodontic patients tend to shoot super close — just the mouth, no face in frame. In that case VNDetectFaceLandmarksRequest returns zero results, and no amount of tweaking will fix it.

So the whole detector is a two-stage cascade:

static func detectMouth(in image: UIImage) async -> MouthLandmarks? {
    // 1. Vision (works when a full face is visible)
    if let landmarks = await detectViaVision(image) {
        return landmarks
    }
    // 2. Color-analysis fallback (works on close-up shots)
    return await detectViaColorAnalysis(image)
}

Stripping EXIF, GPS, and TIFF metadata

Anonymizing the pixels is only half the job. iPhone photos ship with EXIF (timestamp, exposure), TIFF (device model, software version), and GPS (home address, to be blunt) blocks embedded in the file. If you UIImage.jpegData(compressionQuality:) and write that, all of it gets carried along.

The correct move is to skip UIImage entirely and use ImageIO's CGImageDestination, then simply omit every metadata dictionary:

import ImageIO
import MobileCoreServices

func writeAnonymizedJPEG(cgImage: CGImage, to url: URL) throws {
    guard let destination = CGImageDestinationCreateWithURL(
        url as CFURL,
        kUTTypeJPEG,
        1,
        nil
    ) else {
        throw AnonymizationError.destinationCreationFailed
    }

    let options: [CFString: Any] = [
        kCGImageDestinationLossyCompressionQuality: 0.92
        // deliberately no Exif / GPS / TIFF dictionaries
    ]

    CGImageDestinationAddImage(
        destination,
        cgImage,
        options as CFDictionary
    )

    guard CGImageDestinationFinalize(destination) else {
        throw AnonymizationError.finalizeFailed
    }
}

I also added a self-test that reads the JPEG back and asserts every metadata dictionary is nil. In DEBUG builds, this runs after every save. It's cheap paranoia and it caught a real regression once when a refactor accidentally started passing the source properties through.

func assertNoMetadata(url: URL) throws {
    guard let source = CGImageSourceCreateWithURL(url as CFURL, nil),
          let props = CGImageSourceCopyPropertiesAtIndex(source, 0, nil)
            as? [CFString: Any] else {
        throw AnonymizationError.readbackFailed
    }
    assert(props[kCGImagePropertyExifDictionary] == nil)
    assert(props[kCGImagePropertyGPSDictionary]  == nil)
    assert(props[kCGImagePropertyTIFFDictionary] == nil)
}

The full pipeline

Chained together:

static func anonymizeAndSave(
    image: UIImage,
    to url: URL
) async throws {
    // 1. Normalize orientation so pixel space matches display space
    let normalized = image.fixedOrientation()

    // 2. Detect the mouth (Vision, then color-analysis fallback)
    guard let landmarks = await detectMouth(in: normalized) else {
        throw AnonymizationError.mouthNotDetected
    }

    // 3. Compute a square with 1.6× margin around the lip bbox
    let square = squareRect(from: landmarks, margin: 1.6)

    // 4. Crop
    guard let cgImage = normalized.cgImage,
          let cropped = cgImage.cropping(to: square) else {
        throw AnonymizationError.cropFailed
    }

    // 5. Write a fresh JPEG with no EXIF / GPS / TIFF
    try writeAnonymizedJPEG(cgImage: cropped, to: url)

    // 6. Verify (DEBUG only)
    #if DEBUG
    try assertNoMetadata(url: url)
    #endif
}

Six steps, all local. There is nowhere in this pipeline where the raw photo — the one with the face still in it — leaves the phone.

Why on-device is the right default for an indie

I'll be honest: I initially considered a Firebase / Supabase-backed version because it would have been faster to prototype. Two things pushed me back to on-device only.

Cost predictability. Orthodontic photos are chunky (multi-megabyte JPEGs) and each user might store hundreds or thousands over a treatment. As an indie developer, "photo storage that scales with your active users" is exactly the kind of open-ended monthly bill I do not want. On-device storage means my hosting costs are effectively my smileprogress.com marketing site.

Trust as a product feature. In the orthodontic community, "does this app leak my face photos?" is a live question. Every legacy app in the space has had at least one review complaining about the cloud requirement. Building the entire app around "we cannot upload your raw photos even if we wanted to" turned out to be the strongest positioning I've had for any indie project so far.

The tradeoff — and it is a real one — is that features requiring backend state (multi-device sync, remote backup, shareable galleries) are all harder. I address the shareable-gallery case by making the sharing action extremely explicit: user taps Share, gets a preview of exactly what will be sent, has to accept a Terms + Privacy modal, and only then the already-anonymized mouth crop is uploaded. The face never leaves the device at any point in that path.

Try SmileProgress

The app is called SmileProgress (Japanese: 矯正ログ) and it's shipping on the App Store today, initially Japan-first and expanding to other regions:

App Store: https://apps.apple.com/app/id6780383136

It's a one-time purchase — currently a launch price of ¥160 in Japan (regional equivalent in other App Store regions) until 2026-12-31 — with no subscription, no ads, and no cloud-backed features hidden behind a future paywall. Once you own the app, the record / anonymize / compare / anonymous-share flow is yours to keep.

If you're building something in a privacy-adjacent space and want to compare notes on on-device pipelines, App Attest for anonymous auth, or output: "export" static SSG for a companion gallery site, I'd love to hear from you in the comments. Individual developer, so I read everything.

Recap

• VNDetectFaceLandmarksRequest (revision 3) gives you outerLips.normalizedPoints, but they're normalized to the face bounding box and use a bottom-left origin
• A 1.6× margin around the lip bounding box strikes a decent balance between "clinical crop" and "leaks features"
• Wide-open mouths break Vision — cascade to a color-analysis fallback for close-ups
• CGImageDestination with no metadata dictionaries is the reliable way to strip EXIF/GPS/TIFF; a DEBUG self-test catches regressions
• Six steps, all on-device, zero network traffic for the raw photo

Thanks for reading. Happy shipping.