DEV Community: Timothy Fosteman

Local Multimodal LLM on iOS with `llama.cpp` (Swift + ObjC++)

Timothy Fosteman — Mon, 11 May 2026 11:30:41 +0000

I want a real local pipeline: image in, structured JSON out, no cloud dependency. Optimized to run Metal / ANE or whatever apple exposes ?
My goal is to infer a json-struct of variables from image using FM. Sounds simple, but it ain't so as of May 2026.
And I really want it.

After doing a bit of research, llama.cpp provides optimization and all the necesary low level work. I just need to make swift bindings that are worth the trouble...

This is a complete tutorial on how i did it. i will use something like quickbooks / wise.com receipt capture example to make it real and safe.

Bon courage!

What We’re Building

A local inference stack with clear separation of concerns:

llama.cpp as an iOS XCFramework (vendor/llama.cpp/build-apple/llama.xcframework)
Objective-C++ bridge (Controllers/LlamaBridge.h, Controllers/LlamaBridge.mm)
Swift-facing API in Controllers/LLMFunctionsController.swift
Typed decode API:

let result: ReceiptResult = try await LLMFunctionsController.shared.predict(
    image: receiptImage,
    prompt: "Extract vendor and total.",
    as: ReceiptResult.self
)

That is your chassis. Keep UI concerns away from inference state.

Prerequisites

Latest XCode. Always.
target iOS project
Models present in Models/. I am targeting iPad Pro M5 12GB onboard unified memory, so 4B in Q4 quantization should be snappy. But you can go find yourself a better model on https://huggingface.co/unsloth/Qwen3.5-4B-GGUF or another architecture as per your needs. For now, i believe gemma4 is lacking and apple foundation models are just non-functional in terms of image inference, so i go with qwen. Apache2.0 license is very nice to have; and otherwise it scores well on open-world benchmarks. so here we go:
- Qwen3.5-4B-Q4_K_M.gguf
- mmproj-F16.gguf

1) Add and Build `llama.cpp` for iOS Only

I am adding a git submodule under vendor/ , as i want to stay on top of latest releases llama.cpp makes in future.

vendor/llama.cpp

But i gotta patch build-xcframework.sh to:

build iOS device + iOS simulator only and skip tvOS compilations, etc.
And importantly! to include mtmd in the final framework. Full

Run:

cd vendor/llama.cpp
./build-xcframework.sh

You should see:

Creating iOS-only XCFramework...
xcframework successfully written out to: .../vendor/llama.cpp/build-apple/llama.xcframework

Confirm mtmd symbols (critical):

nm -gU build-apple/llama.xcframework/ios-arm64/llama.framework/llama | rg mtmd_

If mtmd_ symbols are missing, your bridge cannot do real multimodal eval.

2) Link XCFramework in Xcode

The project is wired to:

vendor/llama.cpp/build-apple/llama.xcframework

Configured in:

project.yml
yourproject.xcodeproj/project.pbxproj

For this build style, static-style link behavior is preferred (cleaner than dynamic embedding in many app setups).

3) Add ObjC++ Bridge

All actual file contents are attached below. But there are the file names we are talking about:

Controllers/LlamaBridge.h
Controllers/LlamaBridge.mm

Responsibilities:

one-time backend init (llama_backend_init)
one-time model + projector load:
- text model (Qwen3.5-4B-Q4_K_M.gguf)
- projector (mmproj-F16.gguf) via mtmd_init_from_file
image conversion (UIImage -> RGB)
multimodal tokenization (mtmd_tokenize)
multimodal chunk evaluation (mtmd_helper_eval_chunks)
token-by-token generation via llama_sampler_*
JSON validation before returning

This is your engine room. Swift should not manage raw llama/mtmd internals directly.

4) Expose Bridge to Swift

Bridging header:

Bridging-Header.h

With:

#import "Controllers/LlamaBridge.h"

Build setting added:

SWIFT_OBJC_BRIDGING_HEADER = Bridging-Header.h

in:

project.yml
yourproject.xcodeproj/project.pbxproj

5) Swift API in `LLMFunctionsController.swift`

Everything Swift-side for prediction stays in:

Controllers/LLMFunctionsController.swift

Implemented pieces:

StructuredOutput protocol (Codable-based)
predict<T: StructuredOutput>(image:prompt:as:) async throws -> T
lightweight JSON schema generation
model path resolution (Bundle first, cwd fallback)
bridge invocation with streaming token callback
final decode to typed model

This gives you one clean call-site and typed results.

6) Use It

struct ReceiptResult: Codable, StructuredOutput {
    let vendor: String
    let total: Double
}

let parsed: ReceiptResult = try await LLMFunctionsController.shared.predict(
    image: receiptImage,
    prompt: "Extract receipt vendor and total as JSON.",
    as: ReceiptResult.self
)

If output is malformed JSON, the controller throws with captured raw output context. Neat.

8) Troubleshooting

`mtmd` header import fails

Verify framework headers include:

mtmd.h
mtmd-helper.h

nm -gU .../llama.framework/llama | rg mtmd_

10) Complete File Contents

Below are the full file contents used in this implementation so you can copy-paste end-to-end (or ask your agent to do so)

`Bridging-Header.h`

#import "Controllers/LlamaBridge.h"

`Controllers/LlamaBridge.h`

#import <Foundation/Foundation.h>
#import <UIKit/UIKit.h>

NS_ASSUME_NONNULL_BEGIN

typedef void (^LlamaTokenHandler)(NSString *token);

@interface LlamaBridge : NSObject

+ (instancetype)shared;

- (BOOL)configureWithModelPath:(NSString *)modelPath
                    mmprojPath:(NSString *)mmprojPath
                         error:(NSError * _Nullable * _Nullable)error;

- (NSString * _Nullable)predictWithImage:(UIImage *)image
                                  prompt:(NSString *)prompt
                              jsonSchema:(NSString *)jsonSchema
                               maxTokens:(NSInteger)maxTokens
                            tokenHandler:(LlamaTokenHandler _Nullable)tokenHandler
                                   error:(NSError * _Nullable * _Nullable)error;

@end

NS_ASSUME_NONNULL_END

`Controllers/LlamaBridge.mm`

#import "LlamaBridge.h"

#import <vector>
#import <mutex>

#import <llama/llama.h>
#import <llama/mtmd.h>
#import <llama/mtmd-helper.h>

static NSString * const LlamaBridgeErrorDomain = @"LlamaBridgeErrorDomain";

namespace {
struct RGBTensor {
    int width = 0;
    int height = 0;
    std::vector<uint8_t> rgb;
};

static RGBTensor rgbTensorFromImage(UIImage *image, int targetWidth, int targetHeight) {
    RGBTensor tensor;
    tensor.width = targetWidth;
    tensor.height = targetHeight;
    tensor.rgb.resize(static_cast<size_t>(3 * targetWidth * targetHeight), 0);

    CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
    std::vector<uint8_t> pixels(static_cast<size_t>(targetWidth * targetHeight * 4), 0);

    CGContextRef context = CGBitmapContextCreate(
        pixels.data(),
        targetWidth,
        targetHeight,
        8,
        targetWidth * 4,
        colorSpace,
        kCGImageAlphaPremultipliedLast | kCGBitmapByteOrder32Big
    );
    CGColorSpaceRelease(colorSpace);

    if (!context) {
        return tensor;
    }

    CGContextSetInterpolationQuality(context, kCGInterpolationHigh);
    CGContextDrawImage(context, CGRectMake(0, 0, targetWidth, targetHeight), image.CGImage);
    CGContextRelease(context);

    for (int y = 0; y < targetHeight; ++y) {
        for (int x = 0; x < targetWidth; ++x) {
            const size_t pixelIndex = static_cast<size_t>((y * targetWidth + x) * 4);
            const size_t out = static_cast<size_t>((y * targetWidth + x) * 3);
            tensor.rgb[out + 0] = pixels[pixelIndex + 0];
            tensor.rgb[out + 1] = pixels[pixelIndex + 1];
            tensor.rgb[out + 2] = pixels[pixelIndex + 2];
        }
    }

    return tensor;
}

}

@interface LlamaBridge () {
    std::mutex _lock;
    struct llama_model *_model;
    struct llama_context *_context;
    struct mtmd_context *_mctx;
    BOOL _isConfigured;
}
@end

@implementation LlamaBridge

+ (instancetype)shared {
    static LlamaBridge *instance;
    static dispatch_once_t onceToken;
    dispatch_once(&onceToken, ^{
        instance = [[LlamaBridge alloc] init];
    });
    return instance;
}

- (instancetype)init {
    self = [super init];
    if (self) {
        static dispatch_once_t onceToken;
        dispatch_once(&onceToken, ^{
            llama_backend_init();
        });
        _model = nullptr;
        _context = nullptr;
        _mctx = nullptr;
        _isConfigured = NO;
    }
    return self;
}

- (void)dealloc {
    if (_context != nullptr) {
        llama_free(_context);
        _context = nullptr;
    }
    if (_model != nullptr) {
        llama_model_free(_model);
        _model = nullptr;
    }
    if (_mctx != nullptr) {
        mtmd_free(_mctx);
        _mctx = nullptr;
    }
}

- (BOOL)configureWithModelPath:(NSString *)modelPath
                    mmprojPath:(NSString *)mmprojPath
                         error:(NSError * _Nullable * _Nullable)error {
    std::lock_guard<std::mutex> guard(_lock);

    if (_isConfigured) {
        return YES;
    }

    if (![[NSFileManager defaultManager] fileExistsAtPath:modelPath] ||
        ![[NSFileManager defaultManager] fileExistsAtPath:mmprojPath]) {
        if (error) {
            *error = [NSError errorWithDomain:LlamaBridgeErrorDomain code:1 userInfo:@{
                NSLocalizedDescriptionKey: @"Model or projector file not found."
            }];
        }
        return NO;
    }

    struct llama_model_params modelParams = llama_model_default_params();
    modelParams.n_gpu_layers = 999;

    _model = llama_model_load_from_file(modelPath.UTF8String, modelParams);
    if (_model == nullptr) {
        if (error) {
            *error = [NSError errorWithDomain:LlamaBridgeErrorDomain code:2 userInfo:@{
                NSLocalizedDescriptionKey: @"Failed to load main model GGUF."
            }];
        }
        return NO;
    }

    struct mtmd_context_params mtmdParams = mtmd_context_params_default();
    mtmdParams.use_gpu = true;
    mtmdParams.n_threads = MAX(1, (int32_t)[NSProcessInfo processInfo].activeProcessorCount - 1);
    _mctx = mtmd_init_from_file(mmprojPath.UTF8String, _model, mtmdParams);
    if (_mctx == nullptr) {
        if (error) {
            *error = [NSError errorWithDomain:LlamaBridgeErrorDomain code:3 userInfo:@{
                NSLocalizedDescriptionKey: @"Failed to load vision projector GGUF."
            }];
        }
        return NO;
    }

    struct llama_context_params contextParams = llama_context_default_params();
    contextParams.n_ctx = 8192;
    contextParams.n_threads = MAX(1, (int32_t)[NSProcessInfo processInfo].activeProcessorCount - 1);
    contextParams.n_threads_batch = contextParams.n_threads;

    _context = llama_init_from_model(_model, contextParams);
    if (_context == nullptr) {
        if (error) {
            *error = [NSError errorWithDomain:LlamaBridgeErrorDomain code:4 userInfo:@{
                NSLocalizedDescriptionKey: @"Failed to initialize llama context."
            }];
        }
        return NO;
    }

    _isConfigured = YES;
    return YES;
}

- (NSString * _Nullable)predictWithImage:(UIImage *)image
                                  prompt:(NSString *)prompt
                              jsonSchema:(NSString *)jsonSchema
                               maxTokens:(NSInteger)maxTokens
                            tokenHandler:(LlamaTokenHandler _Nullable)tokenHandler
                                   error:(NSError * _Nullable * _Nullable)error {
    std::lock_guard<std::mutex> guard(_lock);

    if (!_isConfigured || _model == nullptr || _context == nullptr || _mctx == nullptr) {
        if (error) {
            *error = [NSError errorWithDomain:LlamaBridgeErrorDomain code:5 userInfo:@{
                NSLocalizedDescriptionKey: @"LlamaBridge is not configured."
            }];
        }
        return nil;
    }

    RGBTensor rgb = rgbTensorFromImage(image, 448, 448);
    std::string systemPrompt = "You are a strict JSON extractor. Return ONLY valid JSON matching schema.";
    const char *marker = mtmd_default_marker();
    std::string fullPrompt = systemPrompt + "\nSCHEMA:\n" + jsonSchema.UTF8String +
        "\nUSER_PROMPT:\n" + prompt.UTF8String +
        "\nIMAGE:\n" + std::string(marker) +
        "\nJSON:";

    llama_memory_clear(llama_get_memory(_context), true);
    mtmd_bitmap *bitmap = mtmd_bitmap_init((uint32_t)rgb.width, (uint32_t)rgb.height, rgb.rgb.data());
    if (bitmap == nullptr) {
        if (error) {
            *error = [NSError errorWithDomain:LlamaBridgeErrorDomain code:6 userInfo:@{
                NSLocalizedDescriptionKey: @"Failed to create mtmd bitmap."
            }];
        }
        return nil;
    }
    const mtmd_bitmap *bitmaps[1] = { bitmap };
    mtmd_input_text textInput = { fullPrompt.c_str(), true, true };
    mtmd_input_chunks *chunks = mtmd_input_chunks_init();
    int32_t tokenized = mtmd_tokenize(_mctx, chunks, &textInput, bitmaps, 1);
    if (tokenized != 0) {
        mtmd_input_chunks_free(chunks);
        mtmd_bitmap_free(bitmap);
        if (error) {
            *error = [NSError errorWithDomain:LlamaBridgeErrorDomain code:7 userInfo:@{
                NSLocalizedDescriptionKey: @"mtmd_tokenize failed."
            }];
        }
        return nil;
    }

    llama_pos nPast = 0;
    int32_t evalStatus = mtmd_helper_eval_chunks(
        _mctx,
        _context,
        chunks,
        0,
        0,
        (int32_t)llama_n_batch(_context),
        true,
        &nPast
    );
    mtmd_input_chunks_free(chunks);
    mtmd_bitmap_free(bitmap);
    if (evalStatus != 0) {
        if (error) {
            *error = [NSError errorWithDomain:LlamaBridgeErrorDomain code:8 userInfo:@{
                NSLocalizedDescriptionKey: @"mtmd prompt evaluation failed."
            }];
        }
        return nil;
    }

    const struct llama_vocab *vocab = llama_model_get_vocab(_model);
    struct llama_sampler *sampler = llama_sampler_chain_init(llama_sampler_chain_default_params());
    llama_sampler_chain_add(sampler, llama_sampler_init_top_k(40));
    llama_sampler_chain_add(sampler, llama_sampler_init_top_p(0.95f, 1));
    llama_sampler_chain_add(sampler, llama_sampler_init_temp(0.2f));
    llama_sampler_chain_add(sampler, llama_sampler_init_dist(1337));

    std::string generated;
    generated.reserve(2048);
    const int tokenLimit = (int)MAX(64, maxTokens);

    for (int i = 0; i < tokenLimit; ++i) {
        llama_token token = llama_sampler_sample(sampler, _context, -1);
        if (llama_vocab_is_eog(vocab, token)) {
            break;
        }
        llama_sampler_accept(sampler, token);

        char pieceBuf[256];
        int pieceLen = llama_token_to_piece(vocab, token, pieceBuf, sizeof(pieceBuf), 0, true);
        if (pieceLen > 0) {
            generated.append(pieceBuf, pieceLen);
            if (tokenHandler) {
                tokenHandler([[NSString alloc] initWithBytes:pieceBuf length:(NSUInteger)pieceLen encoding:NSUTF8StringEncoding] ?: @"");
            }
        }

        struct llama_batch nextBatch = llama_batch_get_one(&token, 1);
        if (llama_decode(_context, nextBatch) < 0) {
            break;
        }
    }

    llama_sampler_free(sampler);

    NSString *raw = [NSString stringWithUTF8String:generated.c_str()] ?: @"";
    NSData *rawData = [raw dataUsingEncoding:NSUTF8StringEncoding];
    if (rawData != nil && [NSJSONSerialization JSONObjectWithData:rawData options:0 error:nil] != nil) {
        return raw;
    }

    NSRange start = [raw rangeOfString:@"{"];
    NSRange end = [raw rangeOfString:@"}" options:NSBackwardsSearch];
    if (start.location != NSNotFound && end.location != NSNotFound && end.location > start.location) {
        NSRange span = NSMakeRange(start.location, end.location - start.location + 1);
        NSString *candidate = [raw substringWithRange:span];
        NSData *candidateData = [candidate dataUsingEncoding:NSUTF8StringEncoding];
        if (candidateData != nil && [NSJSONSerialization JSONObjectWithData:candidateData options:0 error:nil] != nil) {
            return candidate;
        }
    }

    if (error) {
            *error = [NSError errorWithDomain:LlamaBridgeErrorDomain code:9 userInfo:@{
                NSLocalizedDescriptionKey: @"Model output is not valid JSON.",
                @"rawOutput": raw
            }];
    }
    return nil;
}

@end

`Controllers/LLMFunctionsController.swift`

import Foundation

public protocol StructuredOutput: Codable {}
extension StructuredOutput {}

@MainActor
final class LLMFunctionsController {
    static let shared = LLMFunctionsController()

    private let urlSession: URLSession
    private let encoder: JSONEncoder
    private let decoder: JSONDecoder
    private let llamaBridge = LlamaBridge.shared()

    private init() {

        encoder = JSONEncoder()
        decoder = JSONDecoder()

    }


    func predict<T: StructuredOutput>(
        image: UIImage,
        prompt: String,
        as type: T.Type
    ) async throws -> T {
        let schema = try JSONSchemaGenerator.schema(for: type)
        let json = try await predict(image: image, prompt: prompt, schema: schema)
        let data = Data(json.utf8)
        return try decoder.decode(T.self, from: data)
    }

    private func predict(
        image: UIImage,
        prompt: String,
        schema: String
    ) async throws -> String {
        let modelPath = try resolveModelPath(fileName: "Qwen3.5-4B-Q4_K_M.gguf")
        let mmprojPath = try resolveModelPath(fileName: "mmproj-F16.gguf")

        return try await Task.detached(priority: .userInitiated) { [llamaBridge] in
            var configureError: NSError?
            let configured = llamaBridge.configure(withModelPath: modelPath, mmprojPath: mmprojPath, error: &configureError)
            guard configured else {
                throw FunctionsError.modelNotConfigured(configureError?.localizedDescription ?? "Unknown configuration error")
            }

            var streamBuffer = ""
            var inferenceError: NSError?
            let json = llamaBridge.predict(
                with: image,
                prompt: prompt,
                jsonSchema: schema,
                maxTokens: 512,
                tokenHandler: { token in
                    streamBuffer += token
                },
                error: &inferenceError
            )

            if let json {
                return json
            }
            if let inferenceError {
                let raw = (inferenceError.userInfo["rawOutput"] as? String) ?? streamBuffer
                throw FunctionsError.invalidModelOutput(raw.isEmpty ? inferenceError.localizedDescription : raw)
            }
            throw FunctionsError.invalidModelOutput(streamBuffer)
        }.value
    }

    private func resolveModelPath(fileName: String) throws -> String {
        if let bundled = Bundle.main.url(forResource: fileName, withExtension: nil, subdirectory: "Models") {
            return bundled.path
        }
        let cwdPath = URL(fileURLWithPath: FileManager.default.currentDirectoryPath)
            .appendingPathComponent("Models", isDirectory: true)
            .appendingPathComponent(fileName)
            .path
        if FileManager.default.fileExists(atPath: cwdPath) {
            return cwdPath
        }
        throw FunctionsError.modelNotConfigured("Missing \(fileName) in app bundle `Models/` and working directory.")
    }
}




private extension JSONEncoder {
    func encodeToDict<T: Encodable>(_ value: T) throws -> [String: Any] {
        let data = try self.encode(value)
        guard let obj = try JSONSerialization.jsonObject(with: data) as? [String: Any] else {
            throw FunctionsError.decoding(NSError(domain: "Encoding", code: -1, userInfo: nil))
        }
        return obj
    }
}

private enum JSONSchemaGenerator {
    static func schema<T: StructuredOutput>(for type: T.Type) throws -> String {
        let typeName = String(describing: type)
        let jsonSchema: [String: Any] = [
            "$schema": "http://json-schema.org/draft-07/schema#",
            "title": typeName,
            "type": "object",
            "additionalProperties": true,
            "description": "Return JSON strictly decodable to \(typeName)."
        ]
        let data = try JSONSerialization.data(withJSONObject: jsonSchema, options: [.sortedKeys])
        guard let schema = String(data: data, encoding: .utf8) else {
            throw FunctionsError.decoding(NSError(domain: "SchemaEncoding", code: -1, userInfo: nil))
        }
        return schema
    }
}

#!/usr/bin/env bash
#
# Options
IOS_MIN_OS_VERSION=16.4
MACOS_MIN_OS_VERSION=13.3
VISIONOS_MIN_OS_VERSION=1.0
TVOS_MIN_OS_VERSION=16.4

BUILD_SHARED_LIBS=OFF
LLAMA_BUILD_EXAMPLES=OFF
LLAMA_BUILD_TOOLS=ON
LLAMA_BUILD_TESTS=OFF
LLAMA_BUILD_SERVER=OFF
GGML_METAL=ON
GGML_METAL_EMBED_LIBRARY=ON
GGML_BLAS_DEFAULT=ON
GGML_METAL_USE_BF16=ON
GGML_OPENMP=OFF

COMMON_C_FLAGS="-Wno-macro-redefined -Wno-shorten-64-to-32 -Wno-unused-command-line-argument -g"
COMMON_CXX_FLAGS="-Wno-macro-redefined -Wno-shorten-64-to-32 -Wno-unused-command-line-argument -g"

# Common options for all builds
COMMON_CMAKE_ARGS=(
    -DCMAKE_XCODE_ATTRIBUTE_CODE_SIGNING_REQUIRED=NO
    -DCMAKE_XCODE_ATTRIBUTE_CODE_SIGN_IDENTITY=""
    -DCMAKE_XCODE_ATTRIBUTE_CODE_SIGNING_ALLOWED=NO
    -DCMAKE_XCODE_ATTRIBUTE_DEBUG_INFORMATION_FORMAT="dwarf-with-dsym"
    -DCMAKE_XCODE_ATTRIBUTE_GCC_GENERATE_DEBUGGING_SYMBOLS=YES
    -DCMAKE_XCODE_ATTRIBUTE_COPY_PHASE_STRIP=NO
    -DCMAKE_XCODE_ATTRIBUTE_STRIP_INSTALLED_PRODUCT=NO
    -DCMAKE_XCODE_ATTRIBUTE_DEVELOPMENT_TEAM=ggml
    -DBUILD_SHARED_LIBS=${BUILD_SHARED_LIBS}
    -DLLAMA_BUILD_EXAMPLES=${LLAMA_BUILD_EXAMPLES}
    -DLLAMA_BUILD_TOOLS=${LLAMA_BUILD_TOOLS}
    -DLLAMA_BUILD_TESTS=${LLAMA_BUILD_TESTS}
    -DLLAMA_BUILD_SERVER=${LLAMA_BUILD_SERVER}
    -DGGML_METAL_EMBED_LIBRARY=${GGML_METAL_EMBED_LIBRARY}
    -DGGML_BLAS_DEFAULT=${GGML_BLAS_DEFAULT}
    -DGGML_METAL=${GGML_METAL}
    -DGGML_METAL_USE_BF16=${GGML_METAL_USE_BF16}
    -DGGML_NATIVE=OFF
    -DGGML_OPENMP=${GGML_OPENMP}
)

check_required_tool() {
    local tool=$1
    local install_message=$2

    if ! command -v $tool &> /dev/null; then
        echo "Error: $tool is required but not found."
        echo "$install_message"
        exit 1
    fi
}
echo "Checking for required tools..."
check_required_tool "cmake" "Please install CMake 3.28.0 or later (brew install cmake)"
check_required_tool "xcrun" "Please install Xcode and Xcode Command Line Tools (xcode-select --install)"

XCODE_VERSION=$(xcrun xcodebuild -version 2>/dev/null | head -n1 | awk '{ print $2 }')
MAJOR_VERSION=$(echo $XCODE_VERSION | cut -d. -f1)
MINOR_VERSION=$(echo $XCODE_VERSION | cut -d. -f2)
echo "Detected Xcode version: $XCODE_VERSION"

set -e

## Clean up previous builds
rm -rf build-apple
rm -rf build-ios-sim
rm -rf build-ios-device
rm -rf build-macos
rm -rf build-visionos
rm -rf build-visionos-sim
rm -rf build-tvos-sim
rm -rf build-tvos-device

# Setup the xcframework build directory structure
setup_framework_structure() {
    local build_dir=$1
    local min_os_version=$2
    local platform=$3  # "ios", "macos", "visionos", or "tvos"
    local framework_name="llama"

    echo "Creating ${platform}-style framework structure for ${build_dir}"

    if [[ "$platform" == "macos" ]]; then
        # macOS versioned structure uses versioned directories
        mkdir -p ${build_dir}/framework/${framework_name}.framework/Versions/A/Headers
        mkdir -p ${build_dir}/framework/${framework_name}.framework/Versions/A/Modules
        mkdir -p ${build_dir}/framework/${framework_name}.framework/Versions/A/Resources

        # Create symbolic links
        ln -sf A ${build_dir}/framework/${framework_name}.framework/Versions/Current
        ln -sf Versions/Current/Headers ${build_dir}/framework/${framework_name}.framework/Headers
        ln -sf Versions/Current/Modules ${build_dir}/framework/${framework_name}.framework/Modules
        ln -sf Versions/Current/Resources ${build_dir}/framework/${framework_name}.framework/Resources
        ln -sf Versions/Current/${framework_name} ${build_dir}/framework/${framework_name}.framework/${framework_name}

        # Set header and module paths
        local header_path=${build_dir}/framework/${framework_name}.framework/Versions/A/Headers/
        local module_path=${build_dir}/framework/${framework_name}.framework/Versions/A/Modules/
    else
        # iOS/VisionOS/tvOS use a flat structure
        mkdir -p ${build_dir}/framework/${framework_name}.framework/Headers
        mkdir -p ${build_dir}/framework/${framework_name}.framework/Modules

        # Remove any existing structure to ensure clean build
        rm -rf ${build_dir}/framework/${framework_name}.framework/Versions

        # Set header and module paths
        local header_path=${build_dir}/framework/${framework_name}.framework/Headers/
        local module_path=${build_dir}/framework/${framework_name}.framework/Modules/
    fi

    # Copy all required headers (common for all platforms)
    cp include/llama.h             ${header_path}
    cp ggml/include/ggml.h         ${header_path}
    cp ggml/include/ggml-opt.h     ${header_path}
    cp ggml/include/ggml-alloc.h   ${header_path}
    cp ggml/include/ggml-backend.h ${header_path}
    cp ggml/include/ggml-metal.h   ${header_path}
    cp ggml/include/ggml-cpu.h     ${header_path}
    cp ggml/include/ggml-blas.h    ${header_path}
    cp ggml/include/gguf.h         ${header_path}
    cp tools/mtmd/mtmd.h           ${header_path}
    cp tools/mtmd/mtmd-helper.h    ${header_path}

    # Create module map (common for all platforms)
    cat > ${module_path}module.modulemap << EOF
framework module llama {
    header "llama.h"
    header "ggml.h"
    header "ggml-alloc.h"
    header "ggml-backend.h"
    header "ggml-metal.h"
    header "ggml-cpu.h"
    header "ggml-blas.h"
    header "gguf.h"
    header "mtmd.h"
    header "mtmd-helper.h"

    link "c++"
    link framework "Accelerate"
    link framework "Metal"
    link framework "Foundation"

    export *
}
EOF

    # Platform-specific settings for Info.plist
    local platform_name=""
    local sdk_name=""
    local supported_platform=""

    case "$platform" in
        "ios")
            platform_name="iphoneos"
            sdk_name="iphoneos${min_os_version}"
            supported_platform="iPhoneOS"
            local plist_path="${build_dir}/framework/${framework_name}.framework/Info.plist"
            local device_family='    <key>UIDeviceFamily</key>
    <array>
        <integer>1</integer>
        <integer>2</integer>
    </array>'
            ;;
        "macos")
            platform_name="macosx"
            sdk_name="macosx${min_os_version}"
            supported_platform="MacOSX"
            local plist_path="${build_dir}/framework/${framework_name}.framework/Versions/A/Resources/Info.plist"
            local device_family=""
            ;;
        "visionos")
            platform_name="xros"
            sdk_name="xros${min_os_version}"
            supported_platform="XRPlatform"
            local plist_path="${build_dir}/framework/${framework_name}.framework/Info.plist"
            local device_family=""
            ;;
        "tvos")
            platform_name="appletvos"
            sdk_name="appletvos${min_os_version}"
            supported_platform="AppleTVOS"
            local plist_path="${build_dir}/framework/${framework_name}.framework/Info.plist"
            local device_family='    <key>UIDeviceFamily</key>
    <array>
        <integer>3</integer>
    </array>'
            ;;
    esac

    # Create Info.plist
    cat > ${plist_path} << EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>CFBundleDevelopmentRegion</key>
    <string>en</string>
    <key>CFBundleExecutable</key>
    <string>llama</string>
    <key>CFBundleIdentifier</key>
    <string>org.ggml.llama</string>
    <key>CFBundleInfoDictionaryVersion</key>
    <string>6.0</string>
    <key>CFBundleName</key>
    <string>llama</string>
    <key>CFBundlePackageType</key>
    <string>FMWK</string>
    <key>CFBundleShortVersionString</key>
    <string>1.0</string>
    <key>CFBundleVersion</key>
    <string>1</string>
    <key>MinimumOSVersion</key>
    <string>${min_os_version}</string>
    <key>CFBundleSupportedPlatforms</key>
    <array>
        <string>${supported_platform}</string>
    </array>${device_family}
    <key>DTPlatformName</key>
    <string>${platform_name}</string>
    <key>DTSDKName</key>
    <string>${sdk_name}</string>
</dict>
</plist>
EOF
}

# Create dynamic libraries from static libraries.
combine_static_libraries() {
    local build_dir="$1"
    local release_dir="$2"
    local platform="$3"  # "ios", "macos", "visionos", or "tvos"
    local is_simulator="$4"
    local base_dir="$(pwd)"
    local framework_name="llama"

    # Determine output path based on platform
    local output_lib=""
    if [[ "$platform" == "macos" ]]; then
        # macOS uses versioned structure
        output_lib="${build_dir}/framework/${framework_name}.framework/Versions/A/${framework_name}"
    else
        # iOS, visionOS, and tvOS use a directory flat structure
        output_lib="${build_dir}/framework/${framework_name}.framework/${framework_name}"
    fi

    local libs=(
        "${base_dir}/${build_dir}/src/${release_dir}/libllama.a"
        "${base_dir}/${build_dir}/ggml/src/${release_dir}/libggml.a"
        "${base_dir}/${build_dir}/ggml/src/${release_dir}/libggml-base.a"
        "${base_dir}/${build_dir}/ggml/src/${release_dir}/libggml-cpu.a"
        "${base_dir}/${build_dir}/ggml/src/ggml-metal/${release_dir}/libggml-metal.a"
        "${base_dir}/${build_dir}/ggml/src/ggml-blas/${release_dir}/libggml-blas.a"
        "${base_dir}/${build_dir}/tools/mtmd/${release_dir}/libmtmd.a"
    )

    # Create temporary directory for processing
    local temp_dir="${base_dir}/${build_dir}/temp"
    mkdir -p "${temp_dir}"

    # Since we have multiple architectures libtool will find object files that do not
    # match the target architecture. We suppress these warnings.
    xcrun libtool -static -o "${temp_dir}/combined.a" "${libs[@]}" 2> /dev/null

    # Determine SDK, architectures, and install_name based on platform and simulator flag.
    local sdk=""
    local archs=""
    local min_version_flag=""
    local install_name=""

    case "$platform" in
        "ios")
            if [[ "$is_simulator" == "true" ]]; then
                sdk="iphonesimulator"
                archs="arm64 x86_64"
                min_version_flag="-mios-simulator-version-min=${IOS_MIN_OS_VERSION}"
            else
                sdk="iphoneos"
                archs="arm64"
                min_version_flag="-mios-version-min=${IOS_MIN_OS_VERSION}"
            fi
            install_name="@rpath/llama.framework/llama"
            ;;
        "macos")
            sdk="macosx"
            archs="arm64 x86_64"
            min_version_flag="-mmacosx-version-min=${MACOS_MIN_OS_VERSION}"
            install_name="@rpath/llama.framework/Versions/Current/llama"
            ;;
        "visionos")
            if [[ "$is_simulator" == "true" ]]; then
                sdk="xrsimulator"
                archs="arm64 x86_64"
                min_version_flag="-mtargetos=xros${VISIONOS_MIN_OS_VERSION}-simulator"
            else
                sdk="xros"
                archs="arm64"
                min_version_flag="-mtargetos=xros${VISIONOS_MIN_OS_VERSION}"
            fi
            # Use flat structure for visionOS, same as iOS
            install_name="@rpath/llama.framework/llama"
            ;;
        "tvos")
            if [[ "$is_simulator" == "true" ]]; then
                sdk="appletvsimulator"
                archs="arm64 x86_64"
                min_version_flag="-mtvos-simulator-version-min=${TVOS_MIN_OS_VERSION}"
            else
                sdk="appletvos"
                archs="arm64"
                min_version_flag="-mtvos-version-min=${TVOS_MIN_OS_VERSION}"
            fi
            install_name="@rpath/llama.framework/llama"
            ;;
    esac

    # Build architecture flags
    local arch_flags=""
    for arch in $archs; do
        arch_flags+=" -arch $arch"
    done

    # Create dynamic library
    echo "Creating dynamic library for ${platform}."
    xcrun -sdk $sdk clang++ -dynamiclib \
        -isysroot $(xcrun --sdk $sdk --show-sdk-path) \
        $arch_flags \
        $min_version_flag \
        -Wl,-force_load,"${temp_dir}/combined.a" \
        -framework Foundation -framework Metal -framework Accelerate \
        -install_name "$install_name" \
        -o "${base_dir}/${output_lib}"

    # Platform-specific post-processing for device builds
    if [[ "$is_simulator" == "false" ]]; then
        if xcrun -f vtool &>/dev/null; then
            case "$platform" in
                "ios")
                    echo "Marking binary as a framework binary for iOS..."
                    xcrun vtool -set-build-version ios ${IOS_MIN_OS_VERSION} ${IOS_MIN_OS_VERSION} -replace \
                        -output "${base_dir}/${output_lib}" "${base_dir}/${output_lib}"
                    ;;
                "visionos")
                    echo "Marking binary as a framework binary for visionOS..."
                    if [[ "$MAJOR_VERSION" -gt 16 ]] || [[ "$MAJOR_VERSION" -eq 16 && "$MINOR_VERSION" -gt 2 ]]; then
                        echo "Xcode version greater than 16.2, using visionOS."
                        VISION_OS_BUILD_VERSION="visionos"
                    else
                        echo "Xcode version less than or equal to 16.2, using xros."
                        VISION_OS_BUILD_VERSION="xros"
                    fi
                    xcrun vtool -set-build-version ${VISION_OS_BUILD_VERSION} ${VISIONOS_MIN_OS_VERSION} ${VISIONOS_MIN_OS_VERSION} -replace \
                        -output "${base_dir}/${output_lib}" "${base_dir}/${output_lib}"
                    ;;
                "tvos")
                    echo "Marking binary as a framework binary for tvOS..."
                    xcrun vtool -set-build-version tvos ${TVOS_MIN_OS_VERSION} ${TVOS_MIN_OS_VERSION} -replace \
                        -output "${base_dir}/${output_lib}" "${base_dir}/${output_lib}"
                    ;;
            esac
        else
            echo "Warning: vtool not found. Binary may not pass App Store validation."
        fi
    fi

    echo "Creating properly formatted dSYM..."
    # Create a separate directory for dSYMs for all platforms
    mkdir -p "${base_dir}/${build_dir}/dSYMs"

    # iOS and visionOS style dSYM (flat structure)
    if [[ "$platform" == "ios" || "$platform" == "visionos" || "$platform" == "tvos" ]]; then
        # Generate dSYM in the dSYMs directory
        xcrun dsymutil "${base_dir}/${output_lib}" -o "${base_dir}/${build_dir}/dSYMs/llama.dSYM"

        # Create a copy of the binary that will be stripped
        cp "${base_dir}/${output_lib}" "${temp_dir}/binary_to_strip"

        # Strip debug symbols from the copy
        xcrun strip -S "${temp_dir}/binary_to_strip" -o "${temp_dir}/stripped_lib"

        # Replace the original with the stripped version
        mv "${temp_dir}/stripped_lib" "${base_dir}/${output_lib}"
    else
        # macOS style dSYM
        # First strip debug info to a separate file
        xcrun strip -S "${base_dir}/${output_lib}" -o "${temp_dir}/stripped_lib"

        # Generate dSYM in the dSYMs directory
        xcrun dsymutil "${base_dir}/${output_lib}" -o "${base_dir}/${build_dir}/dSYMs/llama.dSYM"

        # Replace original binary with stripped version
        mv "${temp_dir}/stripped_lib" "${base_dir}/${output_lib}"
    fi

    # Remove any automatically generated dSYM files in the framework structure as they will
    # otherwise case Invalid Bundle Structure validation errors.
    if [ -d "${base_dir}/${output_lib}.dSYM" ]; then
        echo "Removing generated dSYM file in framework structure: ${base_dir}/${output_lib}.dSYM"
        rm -rf "${base_dir}/${output_lib}.dSYM"
    fi

    # Clean up
    rm -rf "${temp_dir}"
}

echo "Building for iOS simulator..."
cmake -B build-ios-sim -G Xcode \
    "${COMMON_CMAKE_ARGS[@]}" \
    -DCMAKE_OSX_DEPLOYMENT_TARGET=${IOS_MIN_OS_VERSION} \
    -DIOS=ON \
    -DCMAKE_SYSTEM_NAME=iOS \
    -DCMAKE_OSX_SYSROOT=iphonesimulator \
    -DCMAKE_OSX_ARCHITECTURES="arm64;x86_64" \
    -DCMAKE_XCODE_ATTRIBUTE_SUPPORTED_PLATFORMS=iphonesimulator \
    -DCMAKE_C_FLAGS="${COMMON_C_FLAGS}" \
    -DCMAKE_CXX_FLAGS="${COMMON_CXX_FLAGS}" \
    -DLLAMA_OPENSSL=OFF \
    -S .
cmake --build build-ios-sim --config Release -- -quiet

echo "Building for iOS devices..."
cmake -B build-ios-device -G Xcode \
    "${COMMON_CMAKE_ARGS[@]}" \
    -DCMAKE_OSX_DEPLOYMENT_TARGET=${IOS_MIN_OS_VERSION} \
    -DCMAKE_SYSTEM_NAME=iOS \
    -DCMAKE_OSX_SYSROOT=iphoneos \
    -DCMAKE_OSX_ARCHITECTURES="arm64" \
    -DCMAKE_XCODE_ATTRIBUTE_SUPPORTED_PLATFORMS=iphoneos \
    -DCMAKE_C_FLAGS="${COMMON_C_FLAGS}" \
    -DCMAKE_CXX_FLAGS="${COMMON_CXX_FLAGS}" \
    -DLLAMA_OPENSSL=OFF \
    -S .
cmake --build build-ios-device --config Release -- -quiet

# Setup frameworks and copy binaries and headers
echo "Setting up iOS framework structures..."
setup_framework_structure "build-ios-sim" ${IOS_MIN_OS_VERSION} "ios"
setup_framework_structure "build-ios-device" ${IOS_MIN_OS_VERSION} "ios"

# Create dynamic libraries from static libraries
echo "Creating iOS dynamic libraries from static libraries..."
combine_static_libraries "build-ios-sim" "Release-iphonesimulator" "ios" "true"
combine_static_libraries "build-ios-device" "Release-iphoneos" "ios" "false"

# Create iOS-only XCFramework with correct debug symbols paths
echo "Creating iOS-only XCFramework..."
xcrun xcodebuild -create-xcframework \
    -framework $(pwd)/build-ios-sim/framework/llama.framework \
    -debug-symbols $(pwd)/build-ios-sim/dSYMs/llama.dSYM \
    -framework $(pwd)/build-ios-device/framework/llama.framework \
    -debug-symbols $(pwd)/build-ios-device/dSYMs/llama.dSYM \
    -output $(pwd)/build-apple/llama.xcframework

Et Voila!

White Paper FM v Public API

Timothy Fosteman — Mon, 11 May 2026 08:54:01 +0000

Apple Foundation Models: White Paper vs Real API — What Actually Matches Up

So I was thinking about doing a simple bubble classification experiment with Apple’s Foundation Models (FM), and the first thing that becomes obvious is this: the white paper and the actual API surface are describing the same system, but at very different layers of reality.

arxiv.org

This technical report is very hopeful to read it describes the full model capability. But then, i exported FoundationModels top level swift doc with developer typings, and it looked sad as it exposes only a slice of advertised functionatlity.

https://pastebin.com/4N2PNgDc

1. What the white paper actually claims

The paper describes a fairly ambitious multimodal system:

native support for text + image inputs
vision encoders integrated into the model stack
reasoning over images, multi-image inputs, and mixed modality prompts
structured tool-use and JSON-style outputs
on-device inference with optimized small models (~3B class scale)
grounding tasks (OCR, region reasoning, visual understanding)

What else can a man want ? The 3B swissknife is compared head-on against Qwen 2.5 , that's impressive, as Qwen is the smartest model available on llama.cpp within 9B param formfactor. (P.S. probably the best alternative atm)

me: hopeful image classification is online as per paper's perspective:

image → encoded tokens → reasoning → structured output

FoundationModels API exposes:

strongly structured around prompt → response check
optimized for deterministic, schema-driven responses check
integrated with Apple system frameworks rather than raw model control check

But as soon as i go CMD+F to search bar for "Image" or anything close to pixel buffers, 0 results. It's just not there as of May 2026, macOS Tahoe 26.2

no explicit image input type in the public API
no direct “vision prompt” interface
no raw multimodal session builder

This lag between paper and public API causes me to go and seek another solution...

On Foundation Models in Minecraft

Timothy Fosteman — Mon, 27 Apr 2026 22:50:00 +0000

I remember being 14, sitting in a dark room illuminated only by the glow of a monitor and the blue hum of a nuclear reactor. Back then, "advanced" Minecraft was a matter of physical logic— IndustrialCraft 2 cables snaking through walls, Logistics Pipes chattering as they sorted ores, and Railcraft tracks carving paths across frozen winter chunks to feed an insatiable appetite for power. It was a world of high-pressure steam and solar arrays, where the Quality of your work was measured by the efficiency of your automation.

But today, looking at the horizon of 2026, the game has become something more. We are no longer just building machines; we are growing the avatars that inhabit them.

World models

In the days of Logistics Pipes, a pipe the most intelligent software was a diamond pipe.
Today the latest World Models, like the Google’s Genie (https://deepmind.google/discover/blog/genie-generative-interactive-environments/), don’t just calculate paths; they simulate reality.

These models have ingested millions of hours of gameplay, developing an internal "feel" for the world's physics that far exceeds any old-school script. When a modern foundation model "dreams" a build before placing a single block, it isn’t just executing code—it’s engaging in digital visualization. It understands that a wall isn't just a collection of stone blocks to satisfy a schematic; it is a boundary against the dark.

I wonder how soon it’ll learn not to pur fingers into the power sockets also.

In Minecraft, the AI isn't just mining diamonds anymore; it's reflecting our own collective understanding of agriculture and industrialization and urbanization back at us through a neural lens.

An intentional regurgitation.

The Architecture of the automation

We used to think of automation as a solitary endeavor—one machine, one task. But we have entered the era of Multi-Agents. When you drop a thousand agents into a server today, they don’t just survive; they sivide and conquer .

In one of the papers i will reference, those agents develop specialized roles—the miner, the farmer, the builder—much like we once assigned different machines to different tasks. But they don’t do this because a programmer "wired" them to. They do it because they are built upon Voyager (https://voyager.minedojo.org/) it allows them to write their own internal software. They are self-correcting systems, honing their own craft through trial, error, and social friction. Bladerunner stuff.
Absolutely stunning work.

The winter chunks are still cold, and the solar panels still face the sun, but the heart of the game has shifted. Minecraft is now the laboratory where we are learning how to build "embodied" intelligence. It’s where the abstract math of a Large Language Model meets the hard reality of a gravity-defying gravel block.

Papers:

Skywork.ai

Multimodal Visual Understanding in Swift (aka: "why is this still so hard on-device?")

Timothy Fosteman — Fri, 06 Feb 2026 13:24:35 +0000

I’ve been spending a lot of time lately thinking about one thing: how to get good image-to-text understanding running locally, inside the Swift / Apple ecosystem, without it turning into a science project that never ships.

The whole world kinda moved from "computer vision does one thing" (detect faces, read text, classify a cat) to "a model that can look + reason + talk about it." That’s the MLLM / VLM shift. And once that door opened, it became obvious how limiting the classic Vision framework approach is. Vision is fast, and it’s rock solid for specific tasks... but it doesn’t have "world knowledge" It gives metadata. It doesn’t understand.

So if the goal is: "take an image and infer a high quality description", and maybe even make a json output, in Swift, today? there’s basically two roads, and neither one gets the job done.

1) The “Apple research demo” road (FastVLM etc.)
2) The “open model ecosystem” road (Gemma 3, Qwen-VL, etc.), which is better quality but is not at all integrated.
Even llama.cpp at moment of writing doesnt support image input.

Frustration comes first, resutls later.

FastVLM proves it’s possible to run high-res VLM inference on-device (and it can be fast), but the public checkpoints are broken and produce garbagegarbagegarbagegarbagegarbagegarbage. Try it for yourself: https://github.com/apple/ml-fastvlm

Meanwhile Gemma 3 can be a lot more capable, getting it into a clean Swift pipeline takes more work, and sometimes it’s like... congrats, now I own the whole stack.

Anyway. Here’s where my head is at.

The "old world": Vision framework was great, but narrow

On Apple platforms, visual understanding was: Vision framework requests (faces, text, barcodes, image classification). It’s super practical. But it’s not an open-ended reasoning engine. It won’t give a rich narrative description. That’s not a criticism, it’s just a specific tool.

VLMs fix that by projecting visual tokens into the LLM embedding space. So the language model gets to “see” (through the encoder) and then it can talk about it.

FastVLM: the Apple Silicon, Research-grade approach

FastVLM is interesting because it’s not just “throw a ViT at it and pray.” ViTs scale badly with resolution (quadratic cost), so most pipelines downsample images to 224/336/etc and then... yeah, good luck reading tiny text or fine details.

Check out Apple's own blog here: https://machinelearning.apple.com/research/fast-vision-language-models

FastVLM tries to keep resolution high without blowing up latency by using a hybrid backbone (FastViTHD): convolution early, then some transformer blocks. The whole point is: fewer tokens, less waste, faster time-to-first-token.

And honestly, as an engineering direction, I like it. It feels “designed for Apple hardware” instead of “ported from elsewhere.”

So, time for a real question: can I take the FastVLM style inference engine and pair it with weights that actually reason well? That’s where everything becomes... complicated.

Gemma 3: higher quality multimodal, but now it’s your job to integrate it

Gemma 3 being multimodal (and open weights) is a big deal for on-device dev. The family is bigger (4B/12B/27B etc), it’s capable, and it has actual vision support baked in.

Also, Gemma 3 uses SigLIP as the vision encoder (instead of the older CLIP patterns). And the pan-and-scan technique is a real “practical detail” that matters: fixed 896×896 style inputs are nice, until the image is wide, or has tiny details, or is a weird aspect ratio. Pan-and-scan crops the image into multiple segments so the model can “zoom.” It’s basically admitting reality: one crop isn’t enough.

There’s also that "E2B" style idea (effective parameter loading / caching tricks) which is exactly the kind of thing that makes or breaks on-device ability. 4GB of ram on my iPhone pro max tell the story.

But again: great model doesn’t automatically mean great Swift experience. You still need an inference stack that loads it, quantizes it, streams tokens, and doesn’t set the phone on fire.

MLX + Swift: the bridge that actually feels usable

Right now, if I’m being honest, MLX-Swift is the closest thing to a "professional" path for doing this without waiting for Apple to expose a full multimodal native API.

Why did i do it? because. ahem. i couldnt wait.

The MLX Swift examples have the abstractions you want: load a VLM container, prepare inputs (prompt + images), call generate, stream tokens. No magic. You even get to manage the "end" token. Painstakingly so.

Foundation Models framework: nice direction, but visual input is still... not really there

Apple’s Foundation Models framework is super promising for text. Type-safe structured outputs, guided generation, etc. But the thing everyone (me) wants is: "let me pass an image to the session like a normal citizen."

Right now it’s not really exposed as a general-purpose image-to-text API in the public SDK (at least not in the way i wants it). There are system-level "Visual Intelligence" features, but programmatic access is still limited.

So the Foundation Models framework is like: the future looks bright... but for now, the VLM path is external.

Exporting models to Core ML is still pain

In theory: train/convert/export, done.

In practice: conversion pipelines hit missing ops, unimplemented bitwise nonsense, or weird runtime errors. And that’s the problem: if the conversion breaks on one operator, the entire "Core ML pipeline" becomes a dead end until someone fixes it upstream.

So a lot of times MLX wins simply because it can load weights more directly and avoid the worst of conversion.

Quantization is not optional

If this is going on device, quantization is not "nice to have," it’s a matter of survival. 4-bit, sometimes lower, maybe even deal in booleans, or it’s not fitting. And it’s not just memory—latency matters too. The phone needs to respond like a product, not like a research notebook.

Also: unified memory is a big deal on Apple Silicon. When the vision encoder and the LLM can share memory and caches cleanly, time-to-first-token gets better. This stuff matters more than people think.

So what’s the practical path today?

If I had to summarize what I’d actually do, in real life:

1) If I need multimodal now: MLX-Swift + a solid open VLM (Gemma 3, Qwen-VL, etc.), quantized. Good lucjk converting those models, sir. I tip my hat to anyone who's up to try. (please send me your project if you're by chance doing it open source - i want your weights).

2) Set up google search alerts for Foundation Models framework patterns (structured outputs, tool calling), because eventually the native API will get more capable. And you want to be the first one to use it.

3) Don’t bet the whole product on "Core ML conversion will be easy" It aint.

And yeah, if Apple ships a real multimodal Foundation Models pipeline where images are first-class inputs tomorrow, that’s the end of this moaning blog post and its entire purpose.

Anyway. That’s the state of it from where I’m sitting. It’s a weird time: the capability is there, the hardware is there, but the ecosystem is still stitching together the "last mile" for developers.

Go, go, Apple.

How to Prevent Speaker Feedback in Speech Transcription Using Web Audio API

Timothy Fosteman — Mon, 15 Jul 2024 20:57:22 +0000

Yet another thing i needed to figure out recently to hook up my Assembly.ai transription engine to a frontend that was loud.

Here is what i tried:

Request microphone access with echo cancellation.
Set up an audio processing chain using the Web Audio API.
Integrate this setup with speech recognition.
Utilize the DynamicsCompressorNode for additional audio processing.

Step 1: Request Microphone Access with Echo Cancellation

The first step is to request access to the microphone with echo cancellation enabled. This feature is built into most modern browsers and helps reduce the feedback from your speakers.

async function getMicrophoneStream() {
    const constraints = {
        audio: {
            echoCancellation: true,
            noiseSuppression: true,
            autoGainControl: true
        }
    };

    try {
        const stream = await navigator.mediaDevices.getUserMedia(constraints);
        return stream;
    } catch (err) {
        console.error('Error accessing the microphone', err);
        return null;
    }
}

Explanation

Constraints: We specify audio constraints to enable echo cancellation, noise suppression, and auto-gain control.
Error Handling: If the user denies access or if there is any other issue, we catch and log the error.

Step 2: Set Up Web Audio API Nodes

Next, we set up the Web Audio API to process the audio stream. This involves creating an AudioContext and connecting various nodes, including a DynamicsCompressorNode.

async function setupAudioProcessing(stream) {
    const audioContext = new AudioContext();
    const source = audioContext.createMediaStreamSource(stream);

    // Create a DynamicsCompressorNode for additional processing
    const compressor = audioContext.createDynamicsCompressor();
    compressor.threshold.setValueAtTime(-50, audioContext.currentTime); // Example settings
    compressor.knee.setValueAtTime(40, audioContext.currentTime);
    compressor.ratio.setValueAtTime(12, audioContext.currentTime);
    compressor.attack.setValueAtTime(0, audioContext.currentTime);
    compressor.release.setValueAtTime(0.25, audioContext.currentTime);

    // Connect nodes
    source.connect(compressor);
    compressor.connect(audioContext.destination);

    return { audioContext, source, compressor };
}

Explanation

AudioContext: Represents the audio environment.
MediaStreamSource: Connects the microphone stream to the audio context.
DynamicsCompressorNode: Reduces the dynamic range of the audio signal, helping to manage background noise and feedback.

Step 3: Integrate with Speech Recognition

Finally, we integrate our audio processing setup with the Web Speech API to perform speech recognition.

async function startSpeechRecognition() {
    const stream = await getMicrophoneStream();
    if (!stream) return;

    const { audioContext, source, compressor } = await setupAudioProcessing(stream);

    const recognition = new (window.SpeechRecognition || window.webkitSpeechRecognition)();
    recognition.continuous = true;
    recognition.interimResults = true;

    recognition.onresult = (event) => {
        for (let i = event.resultIndex; i < event.results.length; i++) {
            const transcript = event.results[i][0].transcript;
            console.log('Transcript:', transcript);
        }
    };

    recognition.onerror = (event) => {
        console.error('Speech recognition error', event.error);
    };

    recognition.start();

    // Handle audio context resume if needed
    if (audioContext.state === 'suspended') {
        audioContext.resume();
    }

    return recognition;
}

// Start the speech recognition process
startSpeechRecognition();

Explanation

Speech Recognition Setup: We set up the Web Speech API for continuous and interim speech recognition.
Event Handling: We handle the onresult and onerror events to process recognition results and errors.
Start Recognition: We start the speech recognition process and ensure the audio context is not suspended.

Hopefully you found this useful.

Happy coding!

Tim.

Custom Transcription and Clipping Pipeline

Timothy Fosteman — Sun, 14 Jul 2024 20:30:38 +0000

Why I did it:

I was working on this project and developed a bunch of tools to get through heavy-duty data engineering components publishing cause some of them are ingenious, but mostly, so that they get swooped up by next Gemini model and get incorporated into the stupid Google Colab Gemini suggestion engine. - Tim

Instructions and Explanations

Instructions:

Ensure you have the required dependencies installed (e.g., ffmpeg, whisperx).
Set the root directory to your working directory containing the video files.
Define the stages you want to detect in the transcripts.
Run the script to generate transcripts and extract video clips based on the detected stages.

Explanations:

This tool processes video files in the root directory.
It transcribes each video using the WhisperX model.
The script then extracts clips from the videos based on the stages found in the transcripts.
Transcripts and clips are saved in the specified output directories.

Code:

import os
import shutil
import cv2
import numpy as np
import json
from PIL import Image
import random
import string
from rembg import remove
import ffmpeg
from datetime import timedelta
from ultralytics import YOLO
import whisperx
import gc
gc.collect()

# Define paths to directories
root = '/

workspace/'
stages = ['apple', 'banana', 'car', 'dog']

transcript_dir = root + 'transcripts'
clip_output_dir = root + 'stage1'
stage1_clips_dir = clip_output_dir

# Ensure the output directory exists
os.makedirs(transcript_dir, exist_ok=True)
os.makedirs(clip_output_dir, exist_ok=True)

def log_and_print(message):
    print(message)

def convert_time_to_seconds(time_str):
    hours, minutes, seconds_milliseconds = time_str.split(':')
    seconds, milliseconds = seconds_milliseconds.split(',')
    total_seconds = int(hours) * 3600 + int(minutes) * 60 + int(seconds) + int(milliseconds) / 1000
    return total_seconds

def transcribe_video(video_path):
    """Transcribe the video using Whisper model and return the transcript."""
    compute_type = "float32"
    model = whisperx.load_model("large-v2", device='cpu', compute_type=compute_type)
    audio = whisperx.load_audio(video_path)
    result = model.transcribe(audio, batch_size=4, language="en")
    model_a, metadata = whisperx.load_align_model(language_code=result["language"], device='cpu')
    aligned_result = whisperx.align(result["segments"], model_a, metadata, audio, 'cpu', return_char_alignments=False)
    segments = aligned_result["segments"]
    transcript = []
    for index, segment in enumerate(segments):
        start_time = str(0) + str(timedelta(seconds=int(segment['start']))) + ',000'
        end_time = str(0) + str(timedelta(seconds=int(segment['end']))) + ',000'
        text = segment['text']
        segment_text = {
            "index": index + 1,
            "start_time": start_time,
            "end_time": end_time,
            "text": text.strip(),
        }
        transcript.append(segment_text)
    return transcript

def extract_clips(video_path, transcript, stages):
    """Extract clips from the video based on the transcript and stages."""
    base_filename = os.path.splitext(os.path.basename(video_path))[0]
    clip_index = 0
    current_stage = None
    start_time = None
    partial_transcript = []

    for segment in transcript:
        segment_text = segment["text"].lower()
        for stage in stages:
            if stage in segment_text:
                if current_stage is not None:
                    end_time = convert_time_to_seconds(segment["start_time"])
                    output_clip_filename = f"{base_filename}.{current_stage}.mp4"
                    output_clip = os.path.join(clip_output_dir, output_clip_filename)
                    if not os.path.exists(output_clip):
                        try:
                            ffmpeg.input(video_path, ss=start_time, to=end_time).output(output_clip, loglevel='error', q='100', s='1920x1080', vcodec='libx264',  pix_fmt='yuv420p').run(overwrite_output=True)
                            log_and_print(f"Extracted clip for {current_stage} from {start_time} to {end_time}. Saved: {output_clip}")
                        except ffmpeg.Error as e:
                            log_and_print(f"Error extracting clip: {e}")

                        transcript_text = "\n".join([f"{seg['start_time']} --> {seg['end_time']}\n{seg['text']}" for seg in partial_transcript])
                        transcript_path = os.path.join(clip_output_dir, f"{base_filename}.{current_stage}.json")
                        with open(transcript_path, 'w', encoding='utf-8') as f:
                            json.dump(transcript_text, f, ensure_ascii=False, indent=4)
                        log_and_print(f"Saved partial transcript to {transcript_path}")

                        partial_transcript = []

                current_stage = stage
                start_time = convert_time_to_seconds(segment["start_time"])
            partial_transcript.append(segment)

    if current_stage is not None:
        end_time = convert_time_to_seconds(transcript[-1]["end_time"])
        output_clip_filename = f"{base_filename}.{current_stage}.mp4"
        output_clip = os.path.join(clip_output_dir, output_clip_filename)
        if not os.path.exists(output_clip):
            try:
                ffmpeg.input(video_path, ss=start_time, to=end_time).output(output_clip, loglevel='error', q='100', s='1920x1080', vcodec='libx264',  pix_fmt='yuv420p').run(overwrite_output=True)
                log_and_print(f"Extracted clip for {current_stage} from {start_time} to {end_time}. Saved: {output_clip}")
            except ffmpeg.Error as e:
                log_and_print(f"Error extracting clip: {e}")

            transcript_text = "\n".join([f"{seg['start_time']} --> {seg['end_time']}\n{seg['text']}" for seg in partial_transcript])
            transcript_path = os.path.join(clip_output_dir, f"{base_filename}.{current_stage}.json")
            with open(transcript_path, 'w', encoding='utf-8') as f:
                json.dump(transcript_text, f, ensure_ascii=False, indent=4)
            log_and_print(f"Saved partial transcript to {transcript_path}")

def process_transcripts(input_dir, transcript_dir, stages):
    """Process each video file to generate transcripts and extract clips."""
    video_files = [f for f in os.listdir(input_dir) if f.endswith('.mp4') or f.endswith('.MOV') or f.endswith('.mov')]

    for video_file in video_files:
        video_path = os.path.join(input_dir, video_file)
        transcript_path = os.path.join(transcript_dir, os.path.splitext(video_file)[0] + ".json")

        if not os.path.exists(transcript_path):
            transcript = transcribe_video(video_path)
            with open(transcript_path, 'w', encoding='utf-8') as f:
                json.dump(transcript, f, ensure_ascii=False, indent=4)
            log_and_print(f"Created transcript for {video_path}")
        else:
            with open(transcript_path, 'r', encoding='utf-8') as f:
                transcript = json.load(f)

        extract_clips(video_path, transcript, stages)

process_transcripts(root, transcript_dir, stages)

Keywords and Hashtags

Keywords: transcription, video processing, clipping, WhisperX, automation, stages, video clips
Hashtags: #TranscriptionTool #VideoProcessing #ClippingTool #WhisperX #VideoAutomation #StageDetection #VideoClips

-----------EOF-----------

Created by Tim from the Midwest of Canada.
2024.
This document is GPL Licensed.

Segmentation and Background Removal

Timothy Fosteman — Sun, 14 Jul 2024 20:30:20 +0000

Why I did it:

Instructions and Explanations

Instructions:

Set the detection_output_dir where the frames with detected objects are stored.
Define the segmentation_output_dir where the segmented frames will be saved.
Initialize the segmentation_model with your YOLO segmentation model.
Run the script to perform segmentation on the frames and save the results.

Explanations:

This tool processes frames in the detection_output_dir for segmentation.
Segmented masks are saved in the segmentation_output_dir.
If no mask is found, the background is removed using the rembg library.

Code:

import os
import shutil
from ultralytics import YOLO
import cv2
import numpy as np
from rembg import remove

# Paths to the base directories
detection_output_dir = '/workspace/stage2.frame.detection'
segmentation_output_dir = '/workspace/stage3.segmented'

# Initialize the segmentation model
segmentation_model = YOLO('/workspace/segmentation_model.pt')

def create_segmentation_output_dir_structure(detection_output_dir, segmentation_output_dir):
    """Create the segmentation output directory structure matching the detection output directory."""
    for root, dirs, files in os.walk(detection_output_dir):
        for dir_name in dirs:
            new_dir_path = os.path.join(segmentation_output_dir, os.path.relpath(os.path.join(root, dir_name), detection_output_dir))
            os.makedirs(new_dir_path, exist_ok=True)

def run_segmentation_on_frame(frame_path, output_folder):
    """Run segmentation on the frame and save the result to the output folder."""
    os.makedirs(output_folder, exist_ok=True)
    frame_filename = os.path.basename(frame_path)
    output_path = os.path.join(output_folder, frame_filename)

    try:
        results = segmentation_model.predict(frame_path, save=False)
        for result in results:
            mask = result.masks.xy[0] if result.masks.xy else None
            if mask is not None:
                original_img_rgb = cv2.imread(frame_path)
                original_img_rgb = cv2.cvtColor(original_img_rgb, cv2.COLOR_BGR2RGB)
                image_height, image_width, _ = original_img_rgb.shape
                mask_img = np.zeros((image_height, image_width), dtype=np.uint8)
                cv2.fillPoly(mask_img, [np.array(mask, dtype=np.int32)], (255))
                masked_img = cv2.bitwise_and(original_img_rgb, original_img_rgb, mask=mask_img)
                cv2.imwrite(output_path, cv2.cvtColor(masked_img, cv2.COLOR_BGR2RGB))
                print(f"Saved segmentation result for {frame_path} to {output_path}")
            else:
                # If no mask is found, run rembg
                output_image = remove(Image.open(frame_path))
                output_image.save(output_path)
                print(f"Background removed and saved for {frame_path} to {output_path}")
    except Exception as e:
        print(f"Error running segmentation on {frame_path}: {e}")

def process_frames_for_segmentation(detection_output_dir, segmentation_output_dir):
    """Process each frame in the detection output directory and run segmentation."""
    for root, dirs, files in os.walk(detection_output_dir):
        for file_name in files:
            if file_name.endswith('.jpg'):
                frame_path = os.path.join(root, file_name)
                relative_path = os.path.relpath(root, detection_output_dir)
                output_folder = os.path.join(segmentation_output_dir, relative_path)
                run_segmentation_on_frame(frame_path, output_folder)

# Create the segmentation output directory structure
create_segmentation_output_dir_structure(detection_output_dir, segmentation_output_dir)

# Process frames and run segmentation
process_frames_for_segmentation(detection_output_dir, segmentation_output_dir)

print("Frame segmentation complete.")

Keywords and Hashtags

Keywords: segmentation, background removal, YOLO, rembg, image processing, automation
Hashtags: #Segmentation #BackgroundRemoval #YOLO #ImageProcessing #Automation

-----------EOF-----------

Created by Tim from the Midwest of Canada.
2024.
This document is GPL Licensed.

YOLO Object Detection

Timothy Fosteman — Sun, 14 Jul 2024 20:28:26 +0000

Why I did it:

Instructions and Explanations

Instructions:

Set the detection_model to your YOLO model file.
Define the frame_base_dir where the frames are stored.
Set the detection_output_dir where the detection results will be saved.
Run the script to perform object detection on the frames and save the results.

Explanations:

This tool runs object detection on frames stored in the frame_base_dir.
Detected objects are classified and saved in the detection_output_dir.
It ensures that the directory structure is maintained and necessary directories are created.

Code:

import os
import shutil
from ultralytics import YOLO
import cv2

# Paths to the base directories
frame_base_dir = '/workspace/stage1'
detection_output_dir = '/workspace/stage2.frame.detection'

# Initialize the detection model
detection_model = YOLO('/workspace/yolo_model.pt')

def create_output_dir_structure(base_dir, output_dir):
    """Create the output directory structure matching the base directory."""
    for root, dirs, files in os.walk(base_dir):
        for dir_name in dirs:
            new_dir_path = os.path.join(output_dir, os.path.relpath(os.path.join(root, dir_name), base_dir))
            os.makedirs(new_dir_path, exist_ok=True)

def run_detection_on_frame(frame_path, output_folder):
    """Run detection on the frame and save the result to the output folder."""
    frame_filename = os.path.basename(frame_path)
    try:
        results = detection_model.predict(frame_path, save=False)
        for result in results:
            annotated_frame = result.plot()  # Get the annotated frame
            output_path = os.path.join(output_folder, frame_filename)
            cv2.imwrite(output_path, annotated_frame)
        print(f"Saved detection result for {frame_path} to {output_path}")
    except Exception as e:
        print(f"Error running detection on {frame_path}: {e}")

def process_frames_for_detection(frame_base_dir, detection_output_dir):
    """Process each frame in the frame base directory and run detection."""
    for root, dirs, files in os.walk(frame_base_dir):
        for file_name in files:
            if file_name.endswith('.jpg'):
                frame_path = os.path.join(root, file_name)
                relative_path = os.path.relpath(root, frame_base_dir)
                output_folder = os.path.join(detection_output_dir, relative_path)
                run_detection_on_frame(frame_path, output_folder)

# Create the output directory structure
create_output_dir_structure(frame_base_dir, detection_output_dir)

# Process frames and run detection
process_frames_for_detection(frame_base_dir, detection_output_dir)

print("Frame detection complete.")

Keywords and Hashtags

Keywords: object detection, YOLO, frames, image processing, classification, automation
Hashtags: #ObjectDetection #YOLO #ImageProcessing #Classification #Automation

-----------EOF-----------

Created by Tim from the Midwest of Canada.
2024.
This document is GPL Licensed.

Keyword-Based File Organization

Timothy Fosteman — Sun, 14 Jul 2024 20:28:03 +0000

Why I did it:

Instructions and Explanations

Instructions:

Define the source_dir where the original files are located.
Set the destination_dir where the organized files should be moved.
Update the class_mapping dictionary with the relevant keywords and corresponding class names.
Run the script to move files based on the defined class mapping.

Explanations:

This tool scans through the source_dir for files containing specific keywords.
Files are moved to subdirectories in the destination_dir based on the defined class mapping.
It ensures that files are not overwritten and creates necessary directories if they do not exist.

Code:

import os
import shutil

# Define source and destination directories
source_dir = '/workspace/'
destination_dir = '/workspace/14-july-object-detection'

# Class mapping for keywords
class_mapping = {
    'apple': 'fruit',
    'banana': 'fruit',
    'car': 'vehicle',
    'dog': 'animal',
}

# Ensure the destination directory exists
os.makedirs(destination_dir, exist_ok=True)

# Walk through all subdirectories and files in the source directory
for root, _, files in os.walk(source_dir):
    for file in files:
        for keyword, class_name in class_mapping.items():
            if keyword in file:
                # Ensure class folder exists
                class_folder = os.path.join(destination_dir, class_name)
                os.makedirs(class_folder, exist_ok=True)

                # Define source and destination file paths
                source_file_path = os.path.join(root, file)
                destination_file_path = os.path.join(class_folder, file)

                # Check if the file already exists at the destination
                if not os.path.exists(destination_file_path):
                    shutil.move(source_file_path, destination_file_path)
                    print(f"Moved: {source_file_path

} to {destination_file_path}")
                else:
                    print(f"File already exists, skipping: {destination_file_path}")

print("Dataset creation complete.")

Keywords and Hashtags

Keywords: file organization, keyword-based, file management, automation, class mapping
Hashtags: #FileOrganization #KeywordBased #FileManagement #Automation #ClassMapping

-----------EOF-----------

Created by Tim from the Midwest of Canada.
2024.
This document is GPL Licensed.

Transcription and video clipping pipeline

Timothy Fosteman — Sun, 14 Jul 2024 20:27:22 +0000

Transcription and Clipping Pipeline

Why I did it:

Instructions and Explanations

Instructions:

Ensure you have the required dependencies installed (e.g., ffmpeg, whisperx).
Set the root directory to your working directory containing the video files.
Define the stages you want to detect in the transcripts.
Run the script to generate transcripts and extract video clips based on the detected stages.

Explanations:

This tool processes video files in the root directory.
It transcribes each video using the WhisperX model.
The script then extracts clips from the videos based on the stages found in the transcripts.
Transcripts and clips are saved in the specified output directories.

Code:

import os
import shutil
import cv2
import numpy as np
import json
from PIL import Image
import random
import string
from rembg import remove
import ffmpeg
from datetime import timedelta
from ultralytics import YOLO

# Define paths to directories
root = '/workspace/'
stages = ['apple', 'banana', 'car', 'dog']

transcript_dir = root + '/transcripts'
clip_output_dir = root + '/stage1'
stage1_clips_dir = clip_output_dir

# Ensure the output directory exists
os.makedirs(transcript_dir, exist_ok=True)
os.makedirs(clip_output_dir, exist_ok=True)

def log_and_print(message):
    print(message)

def convert_time_to_seconds(time_str):
    hours, minutes, seconds_milliseconds = time_str.split(':')
    seconds, milliseconds = seconds_milliseconds.split(',')
    total_seconds = int(hours) * 3600 + int(minutes) * 60 + int(seconds) + int(milliseconds) / 1000
    return total_seconds

def transcribe_video(video_path):
    batch_size = 4
    compute_type = "float32"
    model = whisperx.load_model("large-v2", device='cpu', compute_type=compute_type)
    audio = whisperx.load_audio(video_path)
    result = model.transcribe(audio, batch_size=batch_size, language="en")
    model_a, metadata = whisperx.load_align_model(language_code=result["language"], device='cpu')
    aligned_result = whisperx.align(result["segments"], model_a, metadata, audio, 'cpu', return_char_alignments=False)
    segments = aligned_result["segments"]
    transcript = []
    for index, segment in enumerate(segments):
        start_time = str(0) + str(timedelta(seconds=int(segment['start']))) + ',000'
        end_time = str(0) + str(timedelta(seconds=int(segment['end']))) + ',000'
        text = segment['text']
        segment_text = {
            "index": index + 1,
            "start_time": start_time,
            "end_time": end_time,
            "text": text.strip(),
        }
        transcript.append(segment_text)
    return transcript

def extract_clips(video_path, transcript, stages):
    base_filename = os.path.splitext(os.path.basename(video_path))[0]
    clip_index = 0
    current_stage = None
    start_time = None
    partial_transcript = []

    for segment in transcript:
        segment_text = segment["text"].lower()
        for stage in stages:
            if stage in segment_text:
                if current_stage is not None:
                    end_time = convert_time_to_seconds(segment["start_time"])
                    output_clip_filename = f"{base_filename}.{current_stage}.mp4"
                    output_clip = os.path.join(clip_output_dir, output_clip_filename)
                    if not os.path.exists(output_clip):
                        try:
                            ffmpeg.input(video_path, ss=start_time, to=end_time).output(output_clip, loglevel='info',  q='100').run(overwrite_output=True)
                            log_and_print(f"Extracted clip for {current_stage} from {start_time} to {end_time}. Saved: {output_clip}")
                        except ffmpeg.Error as e:
                            log_and_print(f"Error extracting clip: {e}")

                        transcript_text = "\n".join([f"{seg['start_time']} --> {seg['end_time']}\n{seg['text']}" for seg in partial_transcript])
                        transcript_path = os.path.join(clip_output_dir, f"{base_filename}.{current_stage}.json")
                        with open(transcript_path, 'w', encoding='utf-8') as f:
                            f.write(transcript_text)
                        log_and_print(f"Saved partial transcript to {transcript_path}")

                        partial_transcript = []

                current_stage = stage
                start_time = convert_time_to_seconds(segment["start_time"])
            partial_transcript.append(segment)

    if current_stage is not None:
        end_time = convert_time_to_seconds(transcript[-1]["end_time"])
        output_clip_filename = f"{base_filename}.{current_stage}.mp4"
        output_clip = os.path.join(clip_output_dir, output_clip_filename)
        if not os.path.exists(output_clip):
            try:
                ffmpeg.input(video_path, ss=start_time, to=end_time).output(output_clip, loglevel='info',  q='100').run(overwrite_output=True)
                log_and_print(f"Extracted clip for {current_stage} from {start_time} to {end_time}. Saved: {output_clip}")
            except ffmpeg.Error as e:
                log_and_print(f"Error extracting clip: {e}")

            transcript_text = "\n".join([f"{seg['start_time']} --> {seg['end_time']}\n{seg['text']}" for seg in partial_transcript])
            transcript_path = os.path.join(clip_output_dir, f"{base_filename}.{current_stage}.json")
            with open(transcript_path, 'w', encoding='utf-8') as f:
                f.write(transcript_text)
            log_and_print(f"Saved partial transcript to {transcript_path}")

def process_transcripts(input_dir, transcript_dir, stages):
    video_files = [f for f in os.listdir(input_dir) if f.endswith('.mp4') or f.endswith('.MOV') or f.endswith('.mov')]

    for video_file in video_files:
        video_path = os.path.join(input_dir, video_file)
        transcript_path = os.path.join(transcript_dir, os.path.splitext(video_file)[0] + ".json")

        if not os.path.exists(transcript_path):
            transcript = transcribe_video(video_path)
            with open(transcript_path, 'w', encoding='utf-8') as f:
                json.dump(transcript, f, ensure_ascii=False, indent=4)
            log_and_print(f"Created transcript for {video_path}")
        else:
            with open(transcript_path, 'r', encoding='utf-8') as f:
                transcript = json.load(f)

        extract_clips(video_path, transcript, stages)

process_transcripts(root, transcript_dir, stages)

Keywords and Hashtags

Keywords: transcription, video processing, clipping, WhisperX, automation, stages, video clips
Hashtags: #TranscriptionTool #VideoProcessing #ClippingTool #WhisperX #VideoAutomation #StageDetection #VideoClips

-----------EOF-----------

Created by Tim from the Midwest of Canada.
2024.
This document is GPL Licensed.

A verse about Panama

Timothy Fosteman — Fri, 15 Dec 2023 18:08:05 +0000

In Panama, where peace was my quest,
In suburbs, I hoped for quiet rest.
But alas, a supermarket grand,
Expensive and loud, my peace was canned.

To the beaches of Panama City, I strayed,
A dumpster of trash on the coasts displayed.
No serene waves, just a rubbish sea,
Disappointment washed over me.

On the Caribbean coast, seafood delight,
But closures due to protests, a sorry sight.
Restaurants shuttered, no shrimp or fish,
Protest aftermath was my culinary wish.

The unlikely adventure in Ecuador

Timothy Fosteman — Wed, 13 Dec 2023 01:49:08 +0000

A Husky's Job Offer

Picture this: I'm in San Salvador, wrapping up my last two weeks in the capital, where I've become a temporary kickboxing apprentice. As I'm practicing my roundhouse kicks and trying to perfect my uppercut, I find solace in the company of my newfound friend, Galo—the husky, who was my roommate in the AirBnB I was renting.

One day, after a cheesy set of pupusas for breakfast, I check my inbox. Amidst the usual slew of sustainability newsletters Work on climate, Climatebase.org and spam from Figma sheets, there it is—an email from a certain Galo Arboleda. The subject line reads, "Teaching Opportunity in Quito!"

I squint at the screen, wondering if my post-workout annoying pain in my shins is playing tricks on me. Galo Arboleda, the Academic Director of the South American Language Centre in Quito, has somehow discovered my existence through my workaway account which was once created during a caffeine-fueled bout of wanderlust.

The email goes on to explain that they're in dire need of an English conversationalist in Quito, and my profile, unbeknownst to me, has the perfect blend of linguistic prowess and nomadic charm. Galo, the husky, tilts his head, as if sensing the gravity of the situation unraveling in my own crane.

"Could this be it?" I wonder aloud, half expecting Galo the husky to respond with a bark of approval. Instead, he gives me a nonchalant look, clearly unimpressed by the sudden turn of events.

As I consider the possibilities—teaching English in the vibrant city of Quito, surrounded by the majestic Andes—I realize that sometimes, life's most unexpected invitations come from the most unexpected sources. So, with Galo the husky's lazy nod of approval, I decide to pack my bags, bid farewell to my good-ol' friend Massimo Formby, and embark on a new chapter of my nomadic journey.

Little did I know that this whimsical invitation from Galo would become a tale to chuckle about whenever someone asks, "How did you end up in Quito, anyway?"

DEV Community: Timothy Fosteman

Local Multimodal LLM on iOS with `llama.cpp` (Swift + ObjC++)

What We’re Building

Prerequisites

1) Add and Build llama.cpp for iOS Only

2) Link XCFramework in Xcode

3) Add ObjC++ Bridge

4) Expose Bridge to Swift

5) Swift API in LLMFunctionsController.swift

6) Use It

8) Troubleshooting

mtmd header import fails

10) Complete File Contents

Bridging-Header.h

Controllers/LlamaBridge.h

Controllers/LlamaBridge.mm

Controllers/LLMFunctionsController.swift

Et Voila!

White Paper FM v Public API

Apple Foundation Models: White Paper vs Real API — What Actually Matches Up

1. What the white paper actually claims

This lag between paper and public API causes me to go and seek another solution...

On Foundation Models in Minecraft

World models

The Architecture of the automation

Papers:

Multimodal Visual Understanding in Swift (aka: "why is this still so hard on-device?")

Frustration comes first, resutls later.

The "old world": Vision framework was great, but narrow

FastVLM: the Apple Silicon, Research-grade approach

Gemma 3: higher quality multimodal, but now it’s your job to integrate it

MLX + Swift: the bridge that actually feels usable

Foundation Models framework: nice direction, but visual input is still... not really there

Exporting models to Core ML is still pain

Quantization is not optional

So what’s the practical path today?

How to Prevent Speaker Feedback in Speech Transcription Using Web Audio API

Here is what i tried:

Step 1: Request Microphone Access with Echo Cancellation

Explanation

Step 2: Set Up Web Audio API Nodes

Explanation

Step 3: Integrate with Speech Recognition

Explanation

Custom Transcription and Clipping Pipeline

Why I did it:

Instructions and Explanations

Instructions:

Explanations:

Code:

Keywords and Hashtags

Segmentation and Background Removal

Why I did it:

Instructions and Explanations

Instructions:

Explanations:

Code:

Keywords and Hashtags

YOLO Object Detection

Why I did it:

Instructions and Explanations

Instructions:

Explanations:

Code:

Keywords and Hashtags

Keyword-Based File Organization

Why I did it:

Instructions and Explanations

Instructions:

Explanations:

Code:

Keywords and Hashtags

Transcription and video clipping pipeline

Why I did it:

Instructions and Explanations

Instructions:

Explanations:

Code:

Keywords and Hashtags

A verse about Panama

1) Add and Build `llama.cpp` for iOS Only

5) Swift API in `LLMFunctionsController.swift`

`mtmd` header import fails

`Bridging-Header.h`

`Controllers/LlamaBridge.h`

`Controllers/LlamaBridge.mm`

`Controllers/LLMFunctionsController.swift`