DEV Community: Daisuke Majima

A Swift library to run Segment Anything natively on iOS (SamKit)

Daisuke Majima — Tue, 02 Jun 2026 06:16:54 +0000

For a while I'd wanted to build a Swift Package that runs Meta's Segment Anything Model (SAM) on-device on iOS.

Cut out the object you tap
Cut out the object you box in
Cut out the object you specify by text

Any of these segments instantly, with all inference completing on-device. It even comes with ready-to-use UI components.

So I built it.

GitHub: https://github.com/john-rocky/SamKit

What it can do

Feature	Description
Point & Box	Tap for a point, drag for a box, then segment
Text Prompt	Type text like `"dog"` or `"red cup"` to detect and segment
Subject Lift	Long-press to lift an object out, Apple Photos–style; copy/save/share
Two backbones	MobileSAM (fast, 23MB) and SAM2 Tiny (accurate, 76MB)
Drop-in UI	Just embed the SwiftUI views as-is

Architecture

SAMKit/
├── SAMKit            # core inference engine (point/box)
├── SAMKitGrounding   # text detection (YOLO-World + CLIP)
└── SAMKitUI          # SwiftUI views (SamView / TextPromptView)

Split into three Swift Package products. Import only what you need.

Setup

1. Add the Swift Package

dependencies: [
    .package(url: "https://github.com/john-rocky/SamKit.git", from: "1.0.0")
]

2. Download the models

Get the .mlpackage files from Releases and add them to your Xcode project.

Model	Size	Use
MobileSAM	23 MB	point/box segmentation (required)
SAM2 Tiny	76 MB	higher-accuracy segmentation (optional)
Grounding (YOLO-World + CLIP)	148 MB	text detection (optional)

Usage

Point/box segmentation

The most basic use. Set an image and specify a point.

import SAMKit

// create a session (the model auto-loads from a bundled resource)
let session = try SamSession(
    model: .bundled(.mobileSam),
    config: .bestAvailable      // priority: Neural Engine > GPU > CPU
)

// encode the image (once; later predicts use the cache)
try session.setImage(cgImage)

// segment by point
let result = try session.predict(
    points: [SamPoint(x: 100, y: 200, label: .positive)]
)

// results
let mask = result.masks.first!
mask.cgImage   // segmentation mask image
mask.score     // IoU confidence score
mask.alpha     // alpha-channel data

You can also specify negative points (regions to exclude) and a bounding box:

let result = try session.predict(
    points: [
        SamPoint(x: 100, y: 200, label: .positive),   // point to include
        SamPoint(x: 300, y: 400, label: .negative)     // point to exclude
    ],
    box: SamBox(x0: 50, y0: 50, x1: 400, y1: 400)    // bounding box
)

Segment by text prompt

Combine SAM with text detection by YOLO-World + CLIP.

import SAMKit
import SAMKitGrounding

let session = try TextSegmentationSession(
    groundingModel: .bundled(),
    samModel: .bundled(.mobileSam)
)

try session.setImage(cgImage)

// search by text and segment
let result = try session.segment(query: "dog, cat")

result.detections   // detections (bounding box + label)
result.masks        // segmentation mask for each detection
result.scores       // confidence scores

Cutting out the object

You can generate a transparent PNG from the segmentation result.

// cut out from a single mask
let extracted = result.masks[0].extractObject(from: cgImage)
// → a CGImage with a transparent background

// composite cut-out from multiple masks
let combined = SamMask.extractObject(from: cgImage, masks: result.masks)

Embedding the SwiftUI views

You don't need to build the UI yourself. SAMKitUI includes ready-to-use views.

import SAMKitUI

// interactive segmentation by point/box
SamView(image: uiImage, model: try .bundled(.mobileSam))

// segmentation by text search
TextPromptView(image: uiImage, session: textSession)

These views include:

subject highlight after segmentation (dim background + subject at full brightness)
an animated glowing outline
long-press to lift the object → drag → Copy/Save/Share menu

How Subject Lift is implemented

A technical walkthrough of recreating Apple Photos' "lift the subject" feature.

1. Binarizing the mask

SAM's mask output is continuous sigmoid values, so convert it to a clean binary mask for display.

func binarizeMask(_ maskImage: CGImage) -> CGImage? {
    // get pixel data via CGContext
    let ctx = CGContext(data: nil, width: width, height: height, ...)
    ctx.draw(maskImage, in: rect)

    let pixels = ctx.data!.bindMemory(to: UInt8.self, capacity: width * height * 4)
    let threshold: UInt8 = 128  // 50% — SAM's standard threshold

    for i in 0..<(width * height) {
        let o = i * 4
        if pixels[o + 3] >= threshold {
            // fully opaque white
            pixels[o] = 255; pixels[o+1] = 255; pixels[o+2] = 255; pixels[o+3] = 255
        } else {
            // fully transparent
            pixels[o] = 0; pixels[o+1] = 0; pixels[o+2] = 0; pixels[o+3] = 0
        }
    }
    return ctx.makeImage()
}

At threshold 0 it picks up mask noise and cuts out most of the image. 128 (50%) is stable.

2. Generating the glowing outline

Extract the mask's contour with CGContext's shadow feature. Far faster than per-pixel dilation.

func generateOutline(from maskImage: CGImage) -> CGImage? {
    // Step 1: turn the mask into a solid-white silhouette
    ctx.draw(maskImage, in: rect)
    ctx.setBlendMode(.sourceIn)
    ctx.setFillColor(UIColor.white.cgColor)
    ctx.fill(rect)  // → white silhouette

    // Step 2: draw with a shadow, then erase the interior → only the contour remains
    outCtx.setShadow(offset: .zero, blur: glowRadius, color: UIColor.white.cgColor)
    outCtx.draw(whiteSilhouette, in: rect)   // shadow = the contour's glow

    outCtx.setBlendMode(.destinationOut)
    outCtx.draw(whiteSilhouette, in: rect)   // erase the interior → contour only
}

Key points:

setShadow makes the white glow (only two draws)
.destinationOut erases the interior, leaving only the outer glow
Far faster than a dilation loop (O(thickness² × pixels))

3. Shimmer animation

Use TimelineView and AngularGradient to make light travel around the contour.

TimelineView(.animation(minimumInterval: 1.0 / 30)) { timeline in
    let phase = timeline.date.timeIntervalSinceReferenceDate
        .truncatingRemainder(dividingBy: 2.5) / 2.5  // one lap in 2.5s

    ZStack {
        // soft glow (blurred cyan)
        outlineImage.colorMultiply(Color(red: 0.5, green: 0.85, blue: 1.0))
            .blur(radius: 5).opacity(0.8)

        // sharp outline
        outlineImage.colorMultiply(.white)

        // moving highlight
        outlineImage.colorMultiply(.white)
            .mask(
                AngularGradient(
                    colors: [.white, .white.opacity(0.5), .clear, .clear, ...],
                    center: .center,
                    startAngle: .degrees(phase * 360),
                    endAngle: .degrees(phase * 360 + 360)
                )
            )
    }
}

4. Unified gesture handler

Manage tap (add point), box drawing, and long-press lift all with a single DragGesture(minimumDistance: 0).

SwiftUI's onTapGesture + onLongPressGesture block each other, so I receive all touches in one gesture and classify them by time and movement.

DragGesture(minimumDistance: 0)
    .onChanged { value in
        // schedule a timer on the first touch
        if gestureStartTime == nil {
            gestureStartTime = Date()
            // decide long-press after 0.3s
            DispatchQueue.main.asyncAfter(deadline: .now() + 0.3) {
                guard gestureStartTime != nil, !isLifted, hasVisibleMasks else { return }
                let moved = hypot(lastTranslation.width, lastTranslation.height)
                guard moved < 15 else { return }  // if it moved, it's not a long-press
                handleLiftObject()  // start the lift!
            }
        }

        if isLifted {
            liftDragOffset = value.translation  // follow the drag
        }
    }
    .onEnded { value in
        if isLifted {
            // released → show menu
            showLiftMenu = true
        } else if elapsed < 0.3 && moved < 15 {
            // quick touch → add point
            addPoint(at: value.startLocation)
        }
    }

Classification logic:

Condition	Verdict
< 0.3s, < 15pt moved	tap → add point
≥ 0.3s, < 15pt moved	long-press → start lift
≥ 10pt moved (box mode)	drag → draw box
movement while lifted	lift-drag → move object

5. Subject highlight

Rather than overlaying a colored mask after segmentation, dim the background and show only the subject at its original brightness.

// darken the background
Color.black.opacity(0.25)

// show only the subject at the original image's brightness
Image(uiImage: image)
    .mask(Image(uiImage: UIImage(cgImage: binaryMask)))

This makes the transition to long-press lift natural (the dimming just deepens 0.25 → 0.4).

Performance

Image encoding: once per image; later predicts reuse the cache
Inference: accelerated on Neural Engine / GPU (FP16)
Outline generation: only two CGContext-shadow draws; no pixel loop
Networking: none. Fully on-device

Summary

With SAMKit you can add segmentation to an iOS app in a few lines.

// an interactive segmentation UI in one line
SamView(image: uiImage, model: try .bundled(.mobileSam))

Experiences like Subject Lift are built in too, so you can bring an Apple Photos–like UX into your own app immediately.

GitHub: https://github.com/john-rocky/SamKit

Feedback and issues welcome!

Originally published in Japanese on Qiita. GitHub / X

Type 'dog' to detect a dog: running YOLO-World on iPhone

Daisuke Majima — Tue, 02 Jun 2026 06:16:52 +0000

What it does

Type text like "person, red car, coffee cup" and it detects those objects in the camera view in real time. No class list needed. You can specify any words you like, as many as you like.

This is YOLO-World's "Open-Vocabulary Detection." Presented at CVPR 2024, it's a fundamentally different approach from the conventional "fixed 80 classes" YOLO.

How it works

text input ──→ CLIP Text Encoder ──→ text features [1,80,512]
                                            │
camera feed ──→ YOLO-World Detector ────────┤──→ boxes [1,4,8400]
                                            └──→ scores [1,80,8400]
                                                     │
                                                 NMS + Filter ──→ bounding boxes

A dual-wield of CLIP's language understanding and YOLO's detection speed. It converts text into vectors and detects via the matching score against features extracted from the image.

Changing the query text only re-runs the CLIP encoder; camera-frame inference uses only the visual detector. No heavy recompute runs every time the text changes.

Preparing the CoreML models

Download (ready to use)

Download 3 files from the release assets of the CoreML-Models repository:

File	Size	Role
yoloworld_detector.mlpackage	25 MB	YOLO-World V2-S (image → boxes+scores)
clip_text_encoder.mlpackage	121 MB	CLIP ViT-B/32 (text → embedding)
clip_vocab.json	1.6 MB	BPE tokenizer vocabulary

Convert it yourself

pip install ultralytics open_clip_torch coremltools==8.1
python convert_models.py --size s  # s/m/l/x

The conversion script does:

Unwrap YOLO-World V2's Detect head — output boxes [1,4,8400] and scores [1,NC,8400] directly
Convert CLIP's text encoder standalone — patch MultiheadAttention to be CoreML-compatible
Export the BPE vocab as JSON — for the Swift-side tokenizer

iOS implementation

Architecture overview

TextGroundingDetector (ObservableObject)
├── visualModel: MLModel    — YOLO-World detector
├── textEncoder: MLModel    — CLIP text encoder
├── tokenizer: CLIPTokenizer — BPE tokenizer
└── cachedTxtFeats: MLMultiArray — text-feature cache

Encoding text

Run only when the user changes the query; the result is cached.

func updateQueries(_ queryString: String) {
    let queries = queryString.split(separator: ",")
        .map { $0.trimmingCharacters(in: .whitespaces) }

    // tokenize each query → CLIP encoder → 512-dim vector
    let txtFeats = try MLMultiArray(shape: [1, 80, 512], dataType: .float32)

    for (i, query) in queries.prefix(80).enumerated() {
        let tokens = tokenizer.tokenize(query)
        // ... textEncoder.prediction() via MLDictionaryFeatureProvider ...
        // L2-normalize the result and store into txtFeats[i]
    }
    cachedTxtFeats = txtFeats
}

Key points:

Up to 80 queries can be detected at once
L2 normalization is important — CLIP outputs live in a normalized cosine-similarity space
Fast normalization with Accelerate via vDSP_svesq + vDSP_vsmul

Image preprocessing

YOLO-World requires letterbox preprocessing (keep aspect ratio + padding):

func preprocessImage(_ cgImage: CGImage) throws -> MLMultiArray {
    let scale = Float(640) / Float(max(imgW, imgH))
    let scaledW = Int(Float(imgW) * scale)
    let scaledH = Int(Float(imgH) * scale)
    let padX = (640 - scaledW) / 2
    let padY = (640 - scaledH) / 2

    // draw onto a 640x640 canvas padded with gray (0.5)
    ctx.setFillColor(gray: 0.5, alpha: 1.0)
    ctx.fill(CGRect(x: 0, y: 0, width: 640, height: 640))
    ctx.draw(cgImage, in: CGRect(x: padX, y: padY, width: scaledW, height: scaledH))

    // RGBA → CHW Float32 [0,1]
    for i in 0..<(640*640) {
        dst[0 * hw + i] = Float(src[i * 4 + 0]) / 255  // R
        dst[1 * hw + i] = Float(src[i * 4 + 1]) / 255  // G
        dst[2 * hw + i] = Float(src[i * 4 + 2]) / 255  // B
    }
}

You can't use .scaleFill — the coordinates shift by the letterbox padding, so you have to subtract the padding back out of the output coordinates.

Inference and post-processing

let input = try MLDictionaryFeatureProvider(dictionary: [
    "image": tensor,
    "txt_feats": cachedTxtFeats,  // cached text features
])
let output = try visualModel.prediction(from: input)

let boxes = output.featureValue(for: "boxes")!.multiArrayValue!   // [1,4,8400]
let scores = output.featureValue(for: "scores")!.multiArrayValue! // [1,NC,8400]

for qi in 0..<queryCount {
    for anchor in 0..<8400 {
        let score = scores[qi * 8400 + anchor]
        guard score >= threshold else { continue }

        let cx = boxes[0 * 8400 + anchor]
        let cy = boxes[1 * 8400 + anchor]
        let bw = boxes[2 * 8400 + anchor]
        let bh = boxes[3 * 8400 + anchor]

        // remove padding and convert to normalized coordinates
        let nx = (cx - bw/2 - padX) / (imgW * scale)
        let ny = (cy - bh/2 - padY) / (imgH * scale)
    }
}

The output scores are sigmoid values already computed by the BNContrastiveHead, so you can use them directly as confidence.

NMS

Apply NMS per query (per-class):

allDets.sort { $0.confidence > $1.confidence }
var kept: [Int] = []
for i in allDets.indices {
    var suppress = false
    for ki in kept {
        if allDets[i].classIndex == allDets[ki].classIndex
            && iou(allDets[i].rect, allDets[ki].rect) > 0.5 {
            suppress = true; break
        }
    }
    if !suppress { kept.append(i) }
}

BPE tokenizer (Swift)

You need to implement CLIP's tokenizer in Swift. Load the BPE merge rules and vocabulary from clip_vocab.json:

class CLIPTokenizer {
    let contextLength: Int  // 77
    private let encoder: [String: Int]
    private let bpeRanks: [(String, String): Int]

    func tokenize(_ text: String) -> [Int] {
        var tokens = [encoder["<|startoftext|>"]!]
        // lowercase text → split into characters → BPE merge → token IDs
        // ...
        tokens.append(encoder["<|endoftext|>"]!)
        // pad to contextLength (77)
        return tokens + Array(repeating: 0, count: contextLength - tokens.count)
    }
}

Compared with ordinary YOLO

	YOLO-World (Open-Vocabulary)	YOLO26 (fixed classes)
Detection target	any text	fixed COCO 80 classes
Model setup	Detector + CLIP Encoder + Vocab	one model only
Total size	~148 MB	~18 MB
NMS	implemented app-side	none (End-to-End)
Use for	flexible detection / search / grounding	general object detection
Speed	a bit slower (CLIP overhead)	fastest

Practical scenarios

Search by "red sneakers" — visual search in an e-commerce app
Detect "cracks" — infrastructure inspection
Detect "dog, cat, hamster" simultaneously — pet tracking
Let users freely specify what to detect — deploy without customization

With fixed-class YOLO you had to collect a dataset and retrain to detect "cracks." With YOLO-World you just change the text.

Sample app

A complete sample app is in sample_apps/YOLOWorldDemo/ of the CoreML-Models repository.

3 modes: camera / photo / video
freely change the query in a text field
real-time filtering with a confidence slider
download the models from release assets and drag into Xcode

Conversion tips

Use coremltools 8.1 (9.0 has a bug)
You need to patch torch.nn.MultiheadAttention.forward — CoreML can't convert the default PyTorch MHA well; monkey-patch it to call F.multi_head_attention_forward directly
Use YOLO-World V2 (faster and more accurate than V1)
compute_precision=ct.precision.FLOAT16 halves the model size

Summary

YOLO-World delivers intuitive, powerful object detection where you "specify what you want to detect by text." Run it on the iPhone's Neural Engine and it works server-free, offline, with low latency.

When to use which:

Speed-first, COCO 80 classes is enough → YOLO26
Want to flexibly change targets → YOLO-World

References

Originally published in Japanese on Qiita. GitHub / X

Real-time relighting of Gaussian Splatting reflections on iPhone (Metal)

Daisuke Majima — Tue, 02 Jun 2026 06:11:50 +0000

I built a Metal viewer on iPhone that re-lights an already-captured 3D scene with any lighting you like. Swap or rotate the HDR environment map, and the object's reflections follow in real time, with that environment also drawn into the background.

Why relighting is valuable (the practical, commercial meaning)

Ordinary Gaussian Splatting has the lighting from capture time baked in. So if you place the captured object somewhere else, the highlights and shadows clash with the new surroundings and look fake. Relighting strips that light off and returns it to material, so you can place the captured object under any lighting. This matters in practice:

E-commerce / product visuals: capture a product once and show it under any light — showroom, outdoors, the customer's room (AR). Strong for "texture sells" goods like furniture, cars, jewelry, sneakers.
Film / virtual production: place a real capture into a new scene, consistent with that scene's lighting (HDRI / LED wall). No reshoot.
AR / spatial computing: an object placed in a real room only blends in once it's lit by the room's light. Relighting is a precondition for realistic AR placement.
Games / real-time 3D: instead of a baked-in fixed look, you get a photoreal asset that reacts to dynamic in-game light (day/night, moving lights).
Cost: an asset captured in minutes becomes "usable under real production lighting," like manual modeling + material authoring that takes days.

In short, relighting turns a captured 3D into an actually usable asset. This article is a record of bringing it from desktop research (CUDA-assumed) to a real-time Metal implementation on iPhone.

The first half collects background knowledge; the second half covers the implementation and four bugs I hit. I define jargon as it appears, so you can follow even without a Gaussian Splatting / PBR background.

Repo: https://github.com/john-rocky/MetalGaussianSplatRelighting

Background

1. What is 3D Gaussian Splatting

A method that reconstructs a 3D scene from a set of photos and renders it in real time. The scene is represented as a huge set of translucent ellipsoids called splats. Each splat has "position, shape & orientation (rotation), color, opacity." Rendering projects each splat to the screen as an ellipse and alpha-composites them front-to-back in depth order. Color changes with viewing angle (view-dependent).

Key point: ordinary Gaussian Splatting holds the appearance (color) itself, with the capture-time lighting baked in. So you can't change the lighting afterward.

2. Lighting and relighting

Ordinary GS directly learns "the color of that spot photographed under that light." So lighting is fixed.

Relightable GS thinks differently. Per splat, it holds not "color" but a decomposed material:

Albedo: the base color of the material itself, with lighting removed
Normal: the direction the surface faces
Roughness: surface micro-roughness
Reflectance: strength of specular reflection

With the material, you can recompute the color on the fly under any environment light. That's relighting. The Ref-Gaussian I used learns this material decomposition.

That's enough if you get "splats hold material, and we want to re-light them in a new environment." The rest, #3–#5, just answer one question:

Given that material and the environment, how do we compute the color of one pixel?

3. Reflection is split into two and added

Light hitting a surface returns in two ways:

Diffuse: returns light evenly in all directions → you see the albedo color itself. No reflections.
Specular: returns only in a specific direction (the reflection of the incidence) → the environment is reflected.

So color = diffuse + specular. The splat's reflectance decides the blend (how strong the specular is). And roughness decides how blurry the specular is:

low roughness → the environment reflects crisply, like a mirror
high roughness → blurry; the bright parts of the environment just appear as a vague blob

This matters later. In fact, a glossy car (roughness ~0.2) just shows the environment lighting as a blurry white blob that moves, without resolving beams or window shapes. It's not "no reflection," it's a blurry reflection, and that's physically correct. Drop the roughness way down and the same car becomes mirror-like, clearly reflecting the room.

4. How to hold the environment (light source)

Instead of placing point bulbs, use a 360° image as the light = a list of what color light comes from each direction.

HDR: an image that can hold brightness above 1 (needed because windows and lights are orders of magnitude brighter than paper)
equirect / cubemap: names for the storage format of that 360° image. Both contents are "direction → light color."

Swapping the environment map = swapping the lighting = relighting.

5. Getting diffuse and specular from the environment (split-sum)

We want to compute #3's "diffuse" and "specular" from #4's environment map. Done naively, you integrate the environment per pixel — heavy every frame. So UE4's split-sum precomputes two images:

For diffuse (irradiance): the environment averaged over all directions → one lookup in the normal direction gives diffuse light.
For specular (prefiltered): the environment blurred per roughness level (stored progressively in mips) → one lookup in the reflection direction gives specular light (blur level = roughness).

At runtime you just sample these two textures. Replacing a heavy integral with "look up a pre-blurred image" is the heart of split-sum. (Auxiliary: a small table that fine-tunes specular strength by angle — the BRDF LUT — is also precomputed.)

I implemented this precompute kernel in Metal.

6. Deferred shading (a splat-specific issue)

Splats are translucent and overlap, so neighboring splats' normals vary and get noisy. If you shade each splat individually and then composite, that noise comes straight through.

So change the order: first accumulate each splat's material (color, normal, roughness, etc.) into a screen buffer (G-buffer) and blend = average, then shade once per pixel. Computing after normals are averaged reduces noise. In Metal, use tile memory (imageblock) to keep this buffer on the GPU and process it fast.

7. Normals and coordinate systems

Normal: the unit vector of the surface direction. The reflection direction depends on it, so if it's off, all reflections are off. It's reconstructed from the splat's orientation (rotation quaternion).
Up-axis convention: there's Y-up (many viewers) and Z-up (Blender-family). A mismatch between data and viewer tips the object over (→ bug 3).

Implementation: the shading equations

With the background in place, read the equations Ref-Gaussian's deferred surfel renderer (render_surfel) computes per pixel:

F0       = 0.04*(1 - reflectance) + albedo * reflectance
specular = prefiltered(reflect(V, N), roughness) * (F0 * fg.x + fg.y)
final    = (1 - reflectance) * base_color + specular

reflect(V, N): view direction V reflected about normal N. Look up prefiltered (#5) here = the environment reflected in the specular.
fg: a lookup into the BRDF LUT (#5).
final: uses base_color directly for diffuse, and only computes specular from the environment and adds it (= #3's "diffuse + specular"). This matters in bug 2.

Whole pipeline:

Ref-Gaussian .ply --> load --> per-splat material (normal, roughness, reflectance, albedo)
                                       |
HDR env --> IBL precompute --> prefiltered (specular) + irradiance (diffuse) + BRDF LUT
                                       |
                         +-------------+--------------+
                         v                            v
                 G-buffer pass               postprocess pass
        (blend color/normal/material      (per-pixel split-sum IBL
          into tile memory = #6)            + skybox compositing)

— sounds clean. Until you run it. Now the real story: four bugs.

Bug 1: the normal map is a rainbow sandstorm

The shading was patchy and flickering. The "Normal" debug view (normals visualized as color) was rainbow noise, not a smooth gradient.

My first thought — "2D-surfel normals are just inherently noisy" — was wrong, and I nearly wasted a stack of device builds on it.

What saved me was discipline: first draw the normals offline and verify. Compositing each splat's geometric normal (#7: reconstructed from the rotation quaternion and flipped to face the camera) with a small numpy script gave a smooth result (median gradient 0.006). So the data was correct and the renderer was buggy.

Culprit: at load time MetalSplatter reorders splats for cache efficiency (Morton order, sortByLocality). It reorders the splat buffer and the SH-coefficient buffer, but the material buffer (which holds the normals) I'd added later was a separate buffer and wasn't reordered.

So after sorting, splats[i] corresponded to materials[some other j], and every splat held someone else's normal. Color (the view-dependent color of #1) was in an already-reordered buffer, so it stayed consistent, and only the normals and specular looked broken — which made it hard to isolate.

The fix was one line:

materialsBuffer.values.reorderInPlace(fromSourceIndices: sorted)

Lesson: when you bolt a "per-element parallel buffer" onto someone else's pipeline, fix every place the source data gets reordered.

Bug 2: I was re-lighting the diffuse with environment light (don't)

Even after fixing normals, the body was a "watery," patchy yellow.

I'd written the diffuse term by the textbook as albedo × irradiance (multiplying by #5's diffuse image). But Ref-Gaussian's equation uses base_color directly for diffuse — only the specular is relit. I was painting a pattern onto the cream-colored body with my own irradiance. Worse, I was tinting the specular F0 with the view-dependent color (capture-time reflections baked in) instead of the learned albedo.

Matching the reference equation exactly fixed it. Lesson: before improvising "correct" PBR, read the reference implementation's source and match it line by line.

Bug 3: the car is on its side (Z-up vs Y-up)

When I trained and loaded a reflective car, it rendered 90° on its side.

Instead of guessing, I measured the point cloud's bounding box: the shortest axis was Z (= height), the longest was Y (= length), and the dark tire splats were on the −Z side. The data was Z-up (#7, Blender-family). The viewer assumed Y-up. The 90° offset that didn't show on a round helmet was suddenly exposed by the car.

Adding a Z-up→Y-up correction (−90° about X) to the camera fixed the car. Then a second head sprouted: now the background environment was 90° off. equirect (#4) assumes Y-up, but the skybox rays and reflections are computed in the scene's Z-up frame.

The fix: convert the sample direction for the environment into the environment's Y-up frame, and rotate the slider about the scene's up-axis:

environmentRotation = Rx(-90°) * Rz(slider angle)

The skybox and reflections sample with the same matrix, so they always match. I verified the mapping numerically before building, and validated the skybox itself by an offline render that reconstructs rays from the inverse view-projection. That also caught an old top-bottom flip (double-flipped) I'd previously baked into the HDR.

Bug 4: a "successful" build that runs old code

After the orientation fix, a report came in: "the car is still on its side." The offline render had already proven the math correct, so the running binary must be stale — but why?

I'd been type-checking with xcodebuild -destination platform=macOS. That only compiles the #if os(macOS) branch and Mac architectures. When I built for the iOS simulator, existing code revealed a compile error: I was assigning Float16(x).bitPattern into a [UInt16] array.

arm64 (device): native Float16 exists, bitPattern is UInt16 → compiles
the simulator's x86_64 slice: Float16 falls back, bitPattern is UInt32 → type error

The iOS build was failing, so the device kept running the previous binary. Holding [Float16] directly fixed it.

Lesson: type-check for the platform you ship to. A macOS-only xcodebuild will happily lie about your iOS app. And "nothing changes on device" is almost always a sign the binary isn't new — suspect that before re-debugging your logic.

The methodology that actually worked

What all four bugs share: establish ground truth before judging your output. Early on, my offline numpy preview was overly smooth and misled me. What worked:

Run the reference renderer (Ref-Gaussian's eval.py or training-time visualizations) on the same asset and compare. If the reference is clean and yours is dirty, it's a bug in your renderer.
Reproduce the transforms (normals, skybox rays) exactly offline and look at them before building for device.
Verify on a clean synthetic asset (a hand-made chrome sphere) to separate "renderer bug" from "asset bug."

Every time I skipped this and settled for "looks fine," I was wrong.

Aside: is the look "correct"?

After finishing, I felt "the car's reflections are dull, it doesn't look like it's reflecting the environment." That's not a bug — it's the material's nature. To be clear:

The trained car is roughness ~0.2 = semi-gloss. Semi-gloss reflects the environment blurrily (not a mirror). So bright lighting appearing as a blurry blob is correct. Any renderer, lit by the same environment, shows it equally dull. Even the official renderer's output (ground truth) shows a clean reconstruction (sharp car, smooth normals); the material is fine.
Turning off the app's "Use trained material" and lowering roughness lets you override all splats to a uniform specular, clearly reflecting the room. That's flashy but not present on a real car. ON = the learned real material, OFF = a manual override.
Note that material decomposition has an inherent ambiguity: a blue car's albedo can decompose as yellowish (splitting blue into "blue light + yellow material"). This is normal in inverse rendering; the rendered result itself is clean, so the harm is small, but it's not a "physically perfect decomposition."

So "dull look = the material behaving correctly," not an error in the iOS renderer.

Result

A reflective Ref-Gaussian scene, relit in real time on iPhone: switch and rotate the HDR environment and the reflections and skybox follow. The base is MetalSplatter, the relighting model is Ref-Gaussian, and split-sum IBL is from UE4.

Code, demo, details: https://github.com/john-rocky/MetalGaussianSplatRelighting

Next: light the splats from the actual environment via ARKit's environment probe — place a relightable object in your own room and reflect the room.

Implemented in Swift + Metal / iOS. Credits: MetalSplatter (Sean Cier, MIT), Ref-Gaussian, HDRIs from Poly Haven (CC0). Originally published in Japanese on Qiita.

Adding UI to Google Colab: forms, sliders, buttons and more

Daisuke Majima — Tue, 02 Jun 2026 06:11:49 +0000

Adding UI to Colab

You can show UI in a Google Colaboratory notebook. Input forms, buttons, and so on are handy when other people use your notebook. A form's value is reflected into the cell's variable.

Live Colab notebook sample

Cell title

#@title cell title

Input form

You can reflect the form's content into a variable in the cell.

variable = "the form is reflected into the variable" #@param {type:"string"}
# for a number: #@param {type:"number"}

Select box

dropdown = 'value' #@param ["1st option", "2nd option", "3rd option"] {allow-input: true}

Date input

date_input = '2018-03-22' #@param {type:"date"}

Slider

number_slider = 0.1 #@param {type:"slider", min:-1, max:1, step:0.1}

Checkbox

boolean_checkbox = True #@param {type:"boolean"}

Markdown

#@markdown ---
#@markdown #Big
#@markdown ###Middle
#@markdown #####Little
#@markdown ---

A button via the DOM

from IPython.display import display, Javascript
from google.colab import output
from google.colab.output import eval_js

js = Javascript('''
            async function load_image() {
                const div = document.createElement('div');
                var button = document.createElement('button');
                var log = document.createElement('div');

                button.textContent = "button";
                button.onclick = function(){
                    log.innerHTML = "Button Clicked.";
                }
                div.appendChild(button)
                div.appendChild(log)

                document.querySelector("#output-area").appendChild(div);
                return
                } ''')

display(js)
eval_js('load_image()')

Originally published in Japanese on Qiita. I build apps with Core ML and write about machine learning. GitHub / X

Make a 3D model on iPhone just by taking photos (RealityKit Photogrammetry)

Daisuke Majima — Tue, 02 Jun 2026 06:06:47 +0000

Make a realistic 3D model just by taking photos

Using Apple's tool, you can easily create 3D models.

A turtle, and the captured turtle.

I want realistic 3D models

If you have plenty of usable 3D models, you can use them in AR and game apps. But (in my experience) freely usable 3D models for AR/games are surprisingly scarce — on download sites, the nice content is often paid.

Making 3D models seems hard

And making them yourself — modeling and converting in 3D software — seems difficult.

With Apple's RealityKit, just taking photos makes a model

In 2021 Apple released a tool that builds a 3D model just from photos. It's fairly easy and produces realistic objects. It recognizes the salient object and cleanly separates it from the background and floor.

Method

1. Take photos

With a handheld camera (iPhone is fine), photograph the thing you want in 3D from every direction.

I simply put a stuffed animal on the carpet and shot 360° from the side and from above, covering it like a hemisphere. The conditions:

in as bright a place as possible
with the whole subject as large in frame as possible
shooting frequently so consecutive photos overlap by 70%+

I took 200 photos and gathered them into a folder on my Mac.

2. Download Apple's tool

You can get it from the developer site:

https://developer.apple.com/documentation/realitykit/creating_a_photogrammetry_command-line_app/

Build it in Xcode and a HelloPhotogrammetry file appears in the product folder; open it in Finder to find its location.

3. Run

<path to HelloPhotogrammetry> <input image folder> <output file path, with .usdz>

* Absolute paths from root are required.

This generates a USDZ model. With 200 photos it took about 20 minutes. Click the link below on an iPhone to try the turtle USDZ model in AR:

https://firebasestorage.googleapis.com/v0/b/sincere-nirvana-292404.appspot.com/o/model.usdz?alt=media&token=96083418-ed43-435c-bce4-6fed085bbd7b

Toward richer 3D content

Some people upload models made this way to 3D-model sharing sites. It'd be great if 3D models and content grew richer as lots of people scan lots of things.

Originally published in Japanese on Qiita. I build apps with Core ML and ARKit and write about ML/AR. GitHub / X

Japanese OCR on iPhone with the Vision framework

Daisuke Majima — Tue, 02 Jun 2026 06:06:46 +0000

Easy on-device text recognition

If you can recognize text on iPhone, you can build handy things like transcribing whiteboards or reading signs.

A 2022 update made Japanese available

Since iOS 16 (2022), Japanese text recognition is possible — using only the built-in framework. The accuracy is quite good; personally I think it's usable in production for many apps.

How to use it

Use Vision's VNRecognizeTextRequest. Set recognitionLanguages to "ja". Requires macOS 13, Xcode 14, iOS 16 or later.

let request = VNRecognizeTextRequest()
request.recognitionLanguages = ["ja"] // specify Japanese
let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer)
do {
    try handler.perform([request])
} catch let error {
    print(error)
}
guard let observations = request.results as? [VNTextObservation] else { return }
for observation in observations {
    let box = observation.boundingBox // bounding box of the position
    let topCandidate = observation.topCandidates(1)
    if let recognizedText = topCandidate.first?.string { // recognized text
        print(recognizedText)
    }
}

That's it. Feed it an image or a camera frame and it recognizes the text.

Originally published in Japanese on Qiita. I build apps with Core ML and write about machine learning. GitHub / X

How I became a freelance engineer earning 1M yen/month from zero experience

Daisuke Majima — Tue, 02 Jun 2026 05:56:38 +0000

I think there were three keys to going from zero experience to making a living as a freelancer:

Make time
Grasp the big picture
Put yourself out there

Let me go through each.

1. Make time

When deep snow piles up, you want to go out and play, right? Build a snowman, dig a kamakura, have a snowball fight. It's the same: when you have room in your schedule, the urge to experiment shows up. Fail once and you can go "okay, let me try it this way" — trial and error. Like a kid rolling around in the snow. You get to fail. Wasted time matters. Heck, curling up under a kotatsu is fine too, lol.

I'd dabbled in programming in the gaps of my old job, but it never came together. When you fail once in the cracks of a day job, there's no "next try."

So I became unemployed. Did I immediately throw myself into programming because I had time? Not really, lol. For a while I watched Netflix and went for walks. Gradually that got old, and the programming I picked up out of "isn't there something to do?" started to get fun. I think that's about the pace at which self-motivation is born.

That said, making a big block of free time is hard for most people, so a rule like "make one hour of space a day" is fine too. Once you start, habitual momentum builds.

2. Grasp the big picture

When things aren't going well, I think it's usually when you can't see the whole picture. A kid who can't do a pull-over on the bar will try the first kick, but can't picture how that kick connects to the rest of the motion or where it sits in the whole. So they just keep kicking higher and higher. People who can't escape the beginner course can tackle one element but don't know where it fits in the whole. Learning that you can't connect to anything means nothing.

To get good at something, I think it's important to feel the whole picture, even vaguely. With the big picture in place, you can connect the element in front of you organically.

When you start learning to program and do some "copy the code" practice, you don't know how to actually use it. Classic beginner stuff. "1+1 prints like this" — "but how do I get from here to building an app??" I was the same. I took online courses, bought O'Reilly books.

How did I break out? I took Stanford's course — CS193p, the iOS course. After taking it, I could build apps. It covers the knowledge you need to build an app: "you'll really understand everything you need." Being a top university, it's top-tier clear. It's structured so you feel the landmarks needed to build an app and how to connect them, hands-on. I genuinely became able to build.

Finding something that shows you the whole picture as one set really matters. Like a carpenter's apprenticeship — at first you just plane wood until the shavings come out clean, but the planing itself isn't the point; being next to the master and watching them do the job from 1 to 10 is. Imitating people is important for grasping the whole.

Once you understand the whole of "building an app," then whenever you want to make something, you know which knowledge you're missing. After that you just pick up the needed pieces from the web and drop them into the big picture. The course's final project was "build your own app." I decided to add some cutting-edge tech to what I'd learned, hunted for interesting frameworks online, and tried building an app using machine learning. In the end, whether it's ML or iOS or anything, if you search the web there's documentation, prose, and code, and if you read it properly (as long as you have the big picture), you can understand it. Knowing the big picture and where the information lives massively expands what you can do.

Back to CS193p: all the lectures are free on YouTube. English-wise, YouTube's English captions got me through. Watching the whole of "Breaking Bad" in English helped too, lol. Understanding English lets you read docs in the original, which is handy. Though Google Translate is excellent now, so English is much less of a barrier.

3. Put yourself out there

Like the cat in the box, until you're observed, to others you may as well not exist. Ability alone doesn't bring work. You have to break out of the box, stick your head out, and go "meow." An efficient way to shout "I'm here!" is to publish information online.

Write technical articles, publish the information that is your ability. If those articles reach a client's eyes, work comes. You have to connect yourself with the people who'll pay. Doing that suddenly turned programming into money. I publish technical articles on Qiita, write the same articles on the English service Medium, and put code on GitHub. Companies at home and abroad saw them and sent development requests, and as I worked through them I passed ¥1M/month in three months.

Before that, there was about half a year where I earned ¥0. I uploaded personal apps to the App Store hoping to hit it with ad revenue, releasing about ten — an app that computes facial similarity, an AR virtual-background app, an app that strips ML edits off photos, an app that animifies photos. I'd undergone a chimeric, monster-cat evolution inside the box. Scary, right?

But my information never reached the people who'd actually pay. In my case, the payers weren't consumers. Marketing that properly reaches people who can pay matters.

A big part of starting to broadcast was joining a community. There was a "broadcasting course" inside it, taught by someone who'd written tons of technical articles and books, and they clearly explained the whole arc of how that leads to work. Like the carpenter's apprenticeship — being shown the whole path to landing work was huge. They held one-on-one consultation events, gave me a chance to talk with a company's recruiter, pushed me to go freelance, and proofread my articles, which became a real asset. They even taught me how to negotiate with clients. I happened to have time, so I compiled a "best quotes" of community members and helped edit a YouTube channel — that probably earned me some goodwill too. Again, something I could only do because I had time.

Make plenty of time
Grasp the big picture
Put yourself out there

These feel usable for things beyond "making money with programming," too.

Originally published in Japanese on Qiita. I build apps with Core ML and ARKit and write about ML/AR. GitHub / X

On-device LLM on iPhone: which runtime is fastest? MLX vs llama.cpp vs LiteRT-LM vs CoreML

Daisuke Majima — Tue, 02 Jun 2026 05:56:37 +0000

I want to run an LLM on iPhone.
But there are several runtimes and it's not obvious which to pick.

And I couldn't find many head-to-head benchmarks.

Runtime	In a nutshell
MLX	Apple charging into the on-device-LLM scene and pushing hard.
llama.cpp	The mature, battle-tested community standard for local LLMs.
LiteRT-LM	Gemma-4 only, but Google's heavyweight, finally deployed.
CoreML-LLM	Lets you use the Apple Neural Engine, which the GPU/Metal-dominated LLM world tends to overlook. I built it — can it even compete...?

Fine, let's just do it. On an iPhone 17 Pro (A19 Pro), I ran the same model on four on-device inference runtimes and measured decode speed and memory.

The conclusion:

"For local LLMs on iPhone, MLX by default."
"For Gemma 4 specifically, LiteRT-LM is unbeatable."

Conclusion first

Decode speed:
Qwen 3.5 2B is fastest on MLX (61 tok/s).
Gemma 4 E2B is a decisive win for LiteRT-LM (55 tok/s).
Memory:
CoreML / ANE (Apple Neural Engine) wins by a landslide. It runs Qwen 3.5 2B in just 241 MB (about 1/5 of MLX). Slowest on speed, though. Nice effort, CoreML.
Use-case recommendations at the end.

Test conditions

Item	Detail
Device	iPhone 17 Pro (A19 Pro / iOS 26.4.2)
Runtimes	MLX Swift / llama.cpp / LiteRT-LM / CoreML(ANE)
Models	Gemma 4 E2B, Qwen 3.5 2B (both ~4-bit)
Task	short-chat (128-token generation)
Aggregation	median of 3 cold runs
Metrics	decode tok/s (higher = better), peak memory MB (lower = better)

Result 1: decode speed (tok/s, higher is better)

Runtime	Gemma 4 E2B	Qwen 3.5 2B
🔴 LiteRT-LM	55.4 🏆	— (Gemma only)
🟣 MLX-Swift	47.5	61.2 🏆
🔵 llama.cpp	37.8	39.1
🟠 CoreML/ANE	33.4	27.9

For Gemma 4 E2B, LiteRT-LM dominates. It's Google's on-device runtime, running Gemma in its own .litertlm (INT4 QAT) format on the GPU — first-party model × first-party runtime optimization paying off. The Swift API was in development for ages; nice work, whoever shipped it.

Meanwhile for Qwen 3.5 2B, MLX is fastest (61 tok/s). Apple is clearly competing seriously on local LLMs. (LiteRT-LM's catalog is Gemma-only (.litertlm), so it doesn't compete on Qwen.)

Result 2: peak memory (MB, lower is better)

Runtime	Gemma 4 E2B	Qwen 3.5 2B
🔴 LiteRT-LM	641 🏆	—
🟣 MLX-Swift	2,900	1,279
🔵 llama.cpp	3,156	1,479
🟠 CoreML/ANE	1,187	241 🏆

CoreML / ANE wins by a landslide — Qwen 3.5 2B in just 241 MB. That comes from a chunked-MLKV approach (CoreML-LLM's Qwen35MLKVGenerator) that chunks the weights and KV cache onto the ANE — about 1/5 of MLX (1,279) and llama.cpp (1,479).

If you want to run a 2B-class model on a memory-constrained iPhone, or avoid fighting your app and other features for memory, the ANE is a very strong option.

Fairness notes

CoreML/ANE: the ANE is designed for memory/power over throughput. The first load triggers ANE compilation, so load time is longer. Decode is approximated by number of generated pieces (≈ tokens).
LiteRT-LM: there's no max-output-tokens API, so it generates until EOS (≈ 458-token full response); the others cut off at 128. But decode is a rate, so the comparison still holds. Numbers come from LiteRT-LM's own benchmark counter (getBenchmarkInfo).
All are ~4-bit, but the quantization schemes differ slightly per runtime (MLX 4bit / GGUF Q4_K_M / LiteRT INT4-QAT / CoreML INT4-palettized / INT8).

Recommendations by use case

Just fast, general-purpose, lots of models → MLX Swift. Fastest on Qwen, easy from Swift, and mlx-community has tons of models. The first choice for local LLMs on Apple devices.
Gemma as fast as possible → LiteRT-LM. For the Gemma family, strongest on both speed and memory. Can't beat it.
Memory first (any device / coexisting with other features) → CoreML / ANE. 241 MB is exceptional. If you can tolerate the speed, it's the strongest for low memory and power.
Portability / run anywhere → llama.cpp. GGUF assets and every platform. Not flashy, but solid.

Method and reproducibility

Every run was executed headlessly from a Mac via devicectl (no on-device tapping); models were side-loaded from the Mac. The raw result JSONL and charts are in the repo. One line = "1 runtime × 1 model × 1 device" — PRs welcome:

👉 https://github.com/john-rocky/apple-silicon-llm-bench

I'll write a separate article on the behind-the-scenes of full measurement automation and the build fights (git-LFS / SwiftPM unsafe-flags / @preconcurrency, etc.).

Summary

On-device LLM on iPhone: "MLX / LiteRT-LM for speed, CoreML/ANE for memory."

Hope it helps your local-LLM development!

Originally published in Japanese on Qiita. I do mobile AI / CoreML / ARKit development and write about it. GitHub / X

A relightable Gaussian Splatting x AR product viewer on iPhone

Daisuke Majima — Tue, 02 Jun 2026 05:44:57 +0000

In online shopping, you can only imagine how a product actually feels.

"I won't really know until it arrives."
"Since I can't tell, I'll just go with whatever has good reviews."

A lot of purchases get put on hold like this.

What if customers could feel a product's texture through the screen?

This article shows a way to let customers see a product's texture on iPhone.

We deliver it right in front of the customer, in AR.

The value of AR commerce: real texture

The value of AR commerce is being able to bring a product into the customer's own space. This project goes one step further: it lights the product with the real light of the customer's room and reflects their environment in it.

Texture comes alive in the room's light: the environment's lighting is captured and the product is re-lit. Glossy surfaces reflect the surroundings; a dark room produces calm, subdued shading — the texture "reacts to the environment."
Placed on the floor at true scale: the floor is detected and the product is placed at real size. Walk around it and you instantly grasp its size and its look from every angle.
A real "it's actually there" feeling: a contact shadow at its base, plus occlusion (people or furniture in front hide the product), give a placement with no composited look.

The goal: let customers experience, in their own environment, "the texture you couldn't tell from photos."

Why Gaussian Splatting — how the model is made

First, you need to capture the product as 3D. Here we use 3D Gaussian Splatting (3DGS).

It reconstructs a realistic 3D model from photos/video taken from multiple viewpoints (no manual modeling).
It's good at view-dependent representation, where appearance changes with viewing angle — well-suited to reproducing gloss and highlights.
Being point-based, it can be rendered in real time even on a phone.

For this project I built the 3D with Ref-Gaussian, which learns materials (normal, roughness, reflectance, albedo) separately. That makes it possible to "strip off the light from capture time and re-light with the light of wherever you place it (relighting)." This is the decisive difference from ordinary photogrammetry, which bakes lighting in.

Implementation: real-time AR on iPhone with Metal

Rendering is hand-written in Metal (based on scier/MetalSplatter, with split-sum IBL relighting and deferred PBR added). AR is ARKit. Main features:

Relight & reflect with the room's light (feeding AREnvironmentProbeAnchor's environment cubemap into IBL)
True scale, floor placement, walk-around (plane detection + raycast, ARCamera 6DOF)
Contact shadow / depth occlusion (LiDAR sceneDepth)
Color / finish try-on (configurator): switch the same model's color and finish (matte / gloss / mirror / metallic), correctly re-lit by the room's light. Change the color and the white logo or pattern stays white (only the hue is replaced).
Turntable display: hands-free, slow 360° auto-rotation.

The technical key is that lighting is computed at render time. That's why texture reacts correctly, in real time, to "the customer's room light" and "the color the customer chose."

Applicable to many products

The more a product depends on texture and size, the better this lands.

Cars: body-paint reflections, showroom-style presentation, fit-check for a parking space.
Furniture / interior: gloss and color under the room's light, sense of size when placed.
Jewelry / watches: metal and glass reflections (products where texture differences map directly to price).
Appliances / kitchenware: the feel of stainless steel or glossy plastic.
Apparel accessories: the sheen of leather and patent.

In all of them, "texture that photos can't convey" and "a size you can't judge until you place it" are solved in the user's own environment.

Summary

For e-commerce's "size and texture don't come across" problem,
capture the real product with Gaussian Splatting, and
display it in AR on iPhone at true scale, under the customer's room light,
to create a "check the texture before you buy" experience that applies to many products.

The code (relightable GS viewer / Metal + ARKit) is public:
https://github.com/john-rocky/MetalGaussianSplatRelighting

Originally published in Japanese on Qiita. I build apps with machine learning and AR, and write about both. GitHub / X

Fine-tuning a VLM to build an on-device fashion-scoring app

Daisuke Majima — Tue, 02 Jun 2026 05:44:56 +0000

Scoring outfits with AI. Offline.

Can it be done?
Style is qualitative. There isn't a single answer. AI can give a generic answer, but can it answer something like fashion, where the criteria vary by culture?

There is a way.
This article is a record of building a fully offline fashion-scoring app on iPhone using a Visual LLM (VLM).

The approach

Use a closed system of evaluation criteria.

Every aesthetic or philosophical judgment has many schools of thought, and it's hard to produce an open answer that satisfies every possible criterion.

But within a single school — whether in fashion, sports, or specialized work — there are cases where the correct answer is determined inside a closed system.

For example, here I referenced the idea of "the balance between dressy and casual," popularized for a general audience by the Japanese men's-fashion influencer "MB," and treated "if the dressy-to-casual balance is close to 7:3, it looks stylish" as the axis, scoring input images on it. (This is my own interpretation from reading MB's blog and so on.)

Each item of an outfit — tops, bottoms, shoes — is scored against a somewhat systematized standard. An AI (LLM) can do this. And it does it quite well. Even ~1,000 training examples is enough. You don't need to learn every possible item; it extrapolates to unseen ones.

That's the real subject here. More than scoring fashion itself, the theme is how well-suited an LLM is to handling a "closed system."

Small models that fit on an iPhone are well-suited to this kind of domain-specific fine-tuning. With fewer parameters, training is cheap.

This approach works not just for fashion but for anything where the answer is established within a closed system of a given school — makeup, sports form, fortune-telling, and so on.

How it's built

Fine-tune by knowledge distillation:

Teacher = large model (Qwen3-VL-235B-A22B)
Student = small model (Qwen3-VL-2B)

Feed a theory document (~10KB: definitions of 5 axes + baseline tables + aggregation rules + output rules) to the large model as a prompt, and have it score the training images according to that document.

Only the large model can do this; the small model can't hold the entire theory document.

Using the set of (image input given to the large model, output the large model produced), fine-tune the small model.

Now the small model can produce output grounded in the theoretical system. It doesn't know the theory document, but it can perform the baked-in process.

For this one closed-domain evaluation alone, the small model can imitate the behavior of a model 10×–100× its size.

How it works

Input: image
Output: fixed-schema JSON label
Train Qwen3-VL 2B on (image, fixed question, JSON) triplets via LoRA fine-tuning (student)
Convert to CoreML -> iPhone -> fully offline scoring

Because it's "closed," ~800 images are enough. The mapping has low entropy, so if the teacher emits labels under consistent rules, even a small set lets the student reconstruct those rules.

The highest-leverage part is the "theory document"

The most influential file in this pipeline is neither the training script nor the model definition — it's the theory document (the instructions to the teacher).

Writing a genuine theory document is the one thing you can't skimp on.

Output schema (author's reconstruction)

The JSON I had the student emit looks roughly like this (an implementation structure, not text from any original source):

{
  "items": [
    {
      "category": "tops",
      "description": "white cotton dress shirt",
      "scores": { "color": 5, "silhouette": 4, "material": 4, "design": 4, "item_type": 4 },
      "item_dress_score": 4.2
    },
    {
      "category": "bottoms",
      "description": "black skinny trousers",
      "scores": { "color": 5, "silhouette": 4, "material": 4, "design": 4, "item_type": 4 },
      "item_dress_score": 4.3
    }
  ],
  "overall_dress_ratio": 0.71,
  "coordinate_silhouette": { "type": "I", "style_score": 4, "rationale": "..." },
  "target_ratio": 0.70,
  "verdict": "Near-ideal 7:3 for street wear. Dressy edges ahead slightly; clean enough.",
  "advice": "..."
}

Implementation stack and numbers

Role	Choice	Notes
Base model	`Qwen/Qwen3-VL-2B-Instruct`	fp16/int8 stable on Apple Silicon; shipped
(compared) alt base	`google/gemma-4-E2B-it`	schema collapse at int4; passed over for FT
Teacher labeler	Qwen3-VL-235B-A22B	reads the theory and judges JSON
Training	LoRA rank16 / alpha32, `language_model.*` only, vision frozen	~25 min on Colab A100
Conversion	coreml-llm Qwen3-VL stateful pipeline	MLState + slice_update KV
Device	iPhone 17 Pro (A19 ANE)	2.3GB int8 / ~24 tok/s

Training data was ~800–900 full-body outfit photos from Unsplash + Pexels (~750 used for training). One iteration (collect → label → train → convert → transfer) takes roughly 2.5 hours.

Closing: a "dedicated scorer" that fits in your pocket

Specialized knowledge that can be written as a closed system runs faster, cheaper, more consistently, and more privately when distilled wholesale into a 2B model on-device than when thrown at a giant API. If a giant general model is "an advisor who knows a little about everything," what I built here is a way to put "a scorer who has drilled one certification standard into their body" in your pocket. Scoring, assessment, certification, fixed-schema extraction — the world has surprisingly many "closed systems," and any of them might be bakeable to device size with the same pattern.

※ To repeat: this implementation is not supervised or endorsed by any specific individual or organization; it is an independent reconstruction of publicly known ideas for technical validation. The scores do not represent any definitive "correct answer."

Note

The idea this article builds on — "shifting the dressy-to-casual balance toward 7:3" — references a publicly and widely known idea in Japanese men's fashion. The scoring axes, JSON schema, prompt design, and aggregation rules here are my own reconstruction for technical validation, not a quotation or reproduction of any original text, figures, or images. This implementation is not an official, supervised, partnered, or endorsed app of any individual or organization, nor is it intended as an accurate explanation of the theory. It is purely a technical experiment in "how to internalize a subjective evaluation axis into an image-understanding model," and the scores do not constitute anyone's definitive judgment. The value of this article lies not in fashion theory itself but in the methodology of distilling a closed system into a small model.

Originally published in Japanese on Qiita. I build apps with machine learning and AR, and write about both. GitHub / X

The simplest possible SwiftUI MVVM

Daisuke Majima — Tue, 02 Jun 2026 05:38:07 +0000

"I keep hearing about MVVM, but I only half get it..."
"I really should start playing with SwiftUI, but the steps and where to put the code look complicated..."

If that's you, this article will give you the backbone of SwiftUI + MVVM design.

What is MVVM?

MVVM is a code design pattern whose main goal is to separate the Model from the View.

The Model is the actual substance of what the app does.
The View is how the app is presented to the user.

A ViewModel translates and relays the changes between them. That way the Model can keep the app's substance clear and single-sourced, and the View can present the Model's state to the user without delay.

Is MVVM required?

MVVM is not required to build an app with SwiftUI, but using it lets you take the declarative approach — "hand a bundle of content changes to the View and let the View figure out how to render it" — which makes writing apps smooth. (By contrast, telling the View "do this, now do that" on every update is the imperative style.)

Let's understand it with the simplest possible example

It's only human to feel "so what on earth do I actually write in a ViewModel?" and "SwiftUI keeps throwing new characters at me like @ObservedObject and @Published — scary." (I felt exactly that.) So I wrote a tiny case study that combines SwiftUI and MVVM.

We'll build the MVVM pattern with the bare-minimum Model, View and ViewModel.

The case study is a switch that toggles between dog 🐶 and cat 🐱 when tapped.

A simple switch that toggles on tap:

tap ⇄

(The background is green just for clarity.)

A simple MVVM

Writing the Model

The Model of this sample — the substance of this app — is switching between dog and cat.

Model.swift

import Foundation // the Model does NOT import SwiftUI

struct Model {

    enum Pet: String { // the case is either dog or cat
        case 🐶
        case 🐱
    }

    var pet: Pet = .🐶 // default is dog

    mutating func switchPet() { // toggle dog and cat
        if pet == .🐶 {
            pet = .🐱
        } else {
            pet = .🐶
        }
    }

}

The Model does not import SwiftUI, because it is the app's substance — independent of the UI.

The substance of this app is switching between dog and cat, so the Model consists of a pet variable (dog or cat) and a switchPet function that toggles them. (A struct uses a mutating func to mutate itself.)

That's the entire Model of our app.

Writing the View

The View renders the Model's pet as a Text view, and when the Text view is tapped, it switches the Model's pet.

ContentView.swift

import SwiftUI

struct ContentView: View {

    var body: some View {
        Text("reflect the Model's pet here")
            .padding()
            .onTapGesture {
                // switch the model's pet here
        }
    }

}

Our View is responsible for displaying the Model's content to the user and for accepting the user's tap.

If we ignore the MVVM pattern, we could hold the Model directly inside the View:

ContentView.swift (holding the Model directly)

import SwiftUI

struct ContentView: View {
    @State var model = Model()
     // @State lets you change view-state values and reflect them instantly

    var body: some View {
        Text(model.pet.rawValue)
            .padding()
            .onTapGesture {
                model.switchPet()
        }
    }

}

Going further, you could of course hold the pet variable and the toggle function in the View itself:

ContentView.swift (holding pet and switchPet directly)

import SwiftUI

struct ContentView: View {

    enum Pet: String {
        case 🐶
        case 🐱
    }

    @State var pet: Pet = .🐶
     // @State lets you change view-state values and reflect them instantly

    mutating func switchPet() {
        if pet == .🐶 {
            pet = .🐱
        } else {
            pet = .🐶
        }
    }

    var body: some View {
        Text(pet.rawValue)
            .padding()
            .onTapGesture {
                switchPet()
        }
    }

}

If this pet variable only represents transient view state, maybe that's fine.

But then, for example, when you have many Views it becomes hard to keep the Model's state single-sourced. MVVM is about not doing that.

Writing the ViewModel

The ViewModel's job is to be the interpreter between View and Model: relaying the user's tap from the View to the Model, and relaying the Model's state back to the View.

ViewModel.swift

import Foundation
import SwiftUI

class ViewModel {
    var model: Model = Model() // holds the Model

    var pet: String {
        return model.pet.rawValue // return the Model's pet as the String the View needs
    }

    func switchPet() {
        model.switchPet() // call the Model's switchPet
    }
}

Accessing the ViewModel from the View

The View tells the ViewModel about the user's tap, the ViewModel calls the Model's toggle function, and the View reads the Model's pet value back through the ViewModel.

ContentView.swift

import SwiftUI

struct ContentView: View {
    var viewModel = ViewModel()

    var body: some View {
        ZStack {
            Text(viewModel.pet)
                .padding()
                .onTapGesture {
                    viewModel.switchPet()
                }
        }
    }
}

Run this and… the UI doesn't change.

To check, let's add a print to the Model's switchPet:

Model.swift

    mutating func switchPet() {
        if pet == .🐶 {
            pet = .🐱
        } else {
            pet = .🐶
        }
        print(pet)
    }

🐱
🐶
🐱
🐶

The Model's pet is toggling, but the UI isn't updating. In terms of the information flow above:

The View tells the ViewModel about the user's tap (✅ done)
The ViewModel calls the Model's toggle function (✅ done)
~~The View reads the Model's pet change back through the ViewModel~~ (❌ this part isn't arriving)

In MVVM, the ViewModel publishes Model changes to everyone, and the View subscribes to whatever information it cares about — that's how the View receives Model updates.

This is where SwiftUI's property wrappers come in.

The ViewModel publishes changes, and the View subscribes

ViewModel.swift

import Foundation
import SwiftUI

class ViewModel: ObservableObject { // conform to ObservableObject
    @Published var model: Model = Model() // mark it @Published

    var pet: String {
        return model.pet.rawValue
    }

    func switchPet() {
        model.switchPet()
    }
}

By conforming to ObservableObject, the ViewModel becomes observable and can broadcast information to the whole app (to anything willing to observe).

By adding @Published, the ViewModel (an ObservableObject) publishes to everyone the moment this Model changes.

Then the View subscribes to that published change.

ContentView.swift

import SwiftUI

struct ContentView: View {
    @ObservedObject var viewModel = ViewModel() // add @ObservedObject

    var body: some View {
        ZStack {
            Text(viewModel.pet)
                .padding()
                .onTapGesture {
                    viewModel.switchPet()
                }
        }
    }
}

By adding @ObservedObject to the viewModel property, whenever the ObservableObject ViewModel publishes a change, the View can instantly update the relevant UI from its body.

Now the "the View reads the Model's pet value through the ViewModel" part works, and tapping updates the UI.

Dog/cat updating on tap:

This is the simplest possible MVVM.
There's a lot more to it, but I think the basic building blocks of the pattern are all here.

Originally published in Japanese on Qiita. I build apps with Core ML and write about machine learning. GitHub / X

Prompt tips for realistic human images with Stable Diffusion

Daisuke Majima — Tue, 02 Jun 2026 05:38:06 +0000

Tips for generating good images

Stable Diffusion is the talk of the town, but generating the kind of high-quality, realistic portraits you often see on social media takes a bit of know-how. In this article I'll show, with examples, which words help you generate high-quality images.

The Web UI is the convenient way to try Stable Diffusion. You can learn how to use it here:

https://qiita.com/john-rocky/items/1b6dd780d38c63bb64cd

https://github.com/AUTOMATIC1111/stable-diffusion-webui

Just typing the text of the image you want isn't enough

Say you want an image of a girl. If you just type girl, you get something like this. It's not bad, but it looks a little CG-ish — we want something more photographic.

girl

Add words that evoke high quality

Now, pile in words that evoke high quality — best quality, high resolution, and so on — almost to an absurd degree. It might surprise you, but stuffing in lots of comma-separated words like this is the first tip.

girl, RAW photo, best quality, realistic, photo-realistic, best quality, masterpiece, an extremely delicate and beautiful, extremely detailed, 2k wallpaper, Amazing, finely detail, 8k wallpaper, huge filesize, ultra-detailed, highres, extremely detailed, realistic, 8K, Ultra-High Definition, highest quality, ultra high resolution, (realistic:1.4), High quality texture,

Now you get a crisp, realistic image like the one above. But it still looks a bit like a painting.

Add negative words

Add words you do not want in the image. This time, we want to remove the painterly feel and get closer to a photo, so we put painting-evoking words into the negative prompt.

Negative prompt

EasyNegative, paintings, sketches, (worst quality:2), (low quality:2), (normal quality:2), lowres, normal quality, ((monochrome)),

This gets you much closer to a photo. The negative prompt is really important.

Add words for fine detail

(extremely detailed eyes and face)

Add low-quality terms to the negative prompt

Drive the point home by adding words meaning the opposite of high quality to the negative prompt.

Negative prompt

worst quality, low quality, normal quality, jpegartifacts, signature, watermark, blurry, cropped, poorly draw, poorly draw, worst quality, low quality, lowres,

Weight specific words

You can give emphasis to a word by weighting it like this:

(detailed clothes:1.2)

Originally published in Japanese on Qiita. I build apps with machine learning and AR, and write about both. GitHub / X