DEV Community: Timothy Fosteman

Multimodal Visual Understanding in Swift (aka: "why is this still so hard on-device?")

Timothy Fosteman — Fri, 06 Feb 2026 13:24:35 +0000

I’ve been spending a lot of time lately thinking about one thing: how to get good image-to-text understanding running locally, inside the Swift / Apple ecosystem, without it turning into a science project that never ships.

The whole world kinda moved from "computer vision does one thing" (detect faces, read text, classify a cat) to "a model that can look + reason + talk about it." That’s the MLLM / VLM shift. And once that door opened, it became obvious how limiting the classic Vision framework approach is. Vision is fast, and it’s rock solid for specific tasks... but it doesn’t have "world knowledge" It gives metadata. It doesn’t understand.

So if the goal is: "take an image and infer a high quality description", and maybe even make a json output, in Swift, today? there’s basically two roads, and neither one gets the job done.

1) The “Apple research demo” road (FastVLM etc.)
2) The “open model ecosystem” road (Gemma 3, Qwen-VL, etc.), which is better quality but is not at all integrated.
Even llama.cpp at moment of writing doesnt support image input.

Frustration comes first, resutls later.

FastVLM proves it’s possible to run high-res VLM inference on-device (and it can be fast), but the public checkpoints are broken and produce garbagegarbagegarbagegarbagegarbagegarbage. Try it for yourself: https://github.com/apple/ml-fastvlm

Meanwhile Gemma 3 can be a lot more capable, getting it into a clean Swift pipeline takes more work, and sometimes it’s like... congrats, now I own the whole stack.

Anyway. Here’s where my head is at.

The "old world": Vision framework was great, but narrow

On Apple platforms, visual understanding was: Vision framework requests (faces, text, barcodes, image classification). It’s super practical. But it’s not an open-ended reasoning engine. It won’t give a rich narrative description. That’s not a criticism, it’s just a specific tool.

VLMs fix that by projecting visual tokens into the LLM embedding space. So the language model gets to “see” (through the encoder) and then it can talk about it.

FastVLM: the Apple Silicon, Research-grade approach

FastVLM is interesting because it’s not just “throw a ViT at it and pray.” ViTs scale badly with resolution (quadratic cost), so most pipelines downsample images to 224/336/etc and then... yeah, good luck reading tiny text or fine details.

Check out Apple's own blog here: https://machinelearning.apple.com/research/fast-vision-language-models

FastVLM tries to keep resolution high without blowing up latency by using a hybrid backbone (FastViTHD): convolution early, then some transformer blocks. The whole point is: fewer tokens, less waste, faster time-to-first-token.

And honestly, as an engineering direction, I like it. It feels “designed for Apple hardware” instead of “ported from elsewhere.”

So, time for a real question: can I take the FastVLM style inference engine and pair it with weights that actually reason well? That’s where everything becomes... complicated.

Gemma 3: higher quality multimodal, but now it’s your job to integrate it

Gemma 3 being multimodal (and open weights) is a big deal for on-device dev. The family is bigger (4B/12B/27B etc), it’s capable, and it has actual vision support baked in.

Also, Gemma 3 uses SigLIP as the vision encoder (instead of the older CLIP patterns). And the pan-and-scan technique is a real “practical detail” that matters: fixed 896×896 style inputs are nice, until the image is wide, or has tiny details, or is a weird aspect ratio. Pan-and-scan crops the image into multiple segments so the model can “zoom.” It’s basically admitting reality: one crop isn’t enough.

There’s also that "E2B" style idea (effective parameter loading / caching tricks) which is exactly the kind of thing that makes or breaks on-device ability. 4GB of ram on my iPhone pro max tell the story.

But again: great model doesn’t automatically mean great Swift experience. You still need an inference stack that loads it, quantizes it, streams tokens, and doesn’t set the phone on fire.

MLX + Swift: the bridge that actually feels usable

Right now, if I’m being honest, MLX-Swift is the closest thing to a "professional" path for doing this without waiting for Apple to expose a full multimodal native API.

Why did i do it? because. ahem. i couldnt wait.

The MLX Swift examples have the abstractions you want: load a VLM container, prepare inputs (prompt + images), call generate, stream tokens. No magic. You even get to manage the "end" token. Painstakingly so.

Foundation Models framework: nice direction, but visual input is still... not really there

Apple’s Foundation Models framework is super promising for text. Type-safe structured outputs, guided generation, etc. But the thing everyone (me) wants is: "let me pass an image to the session like a normal citizen."

Right now it’s not really exposed as a general-purpose image-to-text API in the public SDK (at least not in the way i wants it). There are system-level "Visual Intelligence" features, but programmatic access is still limited.

So the Foundation Models framework is like: the future looks bright... but for now, the VLM path is external.

Exporting models to Core ML is still pain

In theory: train/convert/export, done.

In practice: conversion pipelines hit missing ops, unimplemented bitwise nonsense, or weird runtime errors. And that’s the problem: if the conversion breaks on one operator, the entire "Core ML pipeline" becomes a dead end until someone fixes it upstream.

So a lot of times MLX wins simply because it can load weights more directly and avoid the worst of conversion.

Quantization is not optional

If this is going on device, quantization is not "nice to have," it’s a matter of survival. 4-bit, sometimes lower, maybe even deal in booleans, or it’s not fitting. And it’s not just memory—latency matters too. The phone needs to respond like a product, not like a research notebook.

Also: unified memory is a big deal on Apple Silicon. When the vision encoder and the LLM can share memory and caches cleanly, time-to-first-token gets better. This stuff matters more than people think.

So what’s the practical path today?

If I had to summarize what I’d actually do, in real life:

1) If I need multimodal now: MLX-Swift + a solid open VLM (Gemma 3, Qwen-VL, etc.), quantized. Good lucjk converting those models, sir. I tip my hat to anyone who's up to try. (please send me your project if you're by chance doing it open source - i want your weights).

2) Set up google search alerts for Foundation Models framework patterns (structured outputs, tool calling), because eventually the native API will get more capable. And you want to be the first one to use it.

3) Don’t bet the whole product on "Core ML conversion will be easy" It aint.

And yeah, if Apple ships a real multimodal Foundation Models pipeline where images are first-class inputs tomorrow, that’s the end of this moaning blog post and its entire purpose.

Anyway. That’s the state of it from where I’m sitting. It’s a weird time: the capability is there, the hardware is there, but the ecosystem is still stitching together the "last mile" for developers.

Go, go, Apple.

How to Prevent Speaker Feedback in Speech Transcription Using Web Audio API

Timothy Fosteman — Mon, 15 Jul 2024 20:57:22 +0000

Yet another thing i needed to figure out recently to hook up my Assembly.ai transription engine to a frontend that was loud.

Here is what i tried:

Request microphone access with echo cancellation.
Set up an audio processing chain using the Web Audio API.
Integrate this setup with speech recognition.
Utilize the DynamicsCompressorNode for additional audio processing.

Step 1: Request Microphone Access with Echo Cancellation

The first step is to request access to the microphone with echo cancellation enabled. This feature is built into most modern browsers and helps reduce the feedback from your speakers.

async function getMicrophoneStream() {
    const constraints = {
        audio: {
            echoCancellation: true,
            noiseSuppression: true,
            autoGainControl: true
        }
    };

    try {
        const stream = await navigator.mediaDevices.getUserMedia(constraints);
        return stream;
    } catch (err) {
        console.error('Error accessing the microphone', err);
        return null;
    }
}

Explanation

Constraints: We specify audio constraints to enable echo cancellation, noise suppression, and auto-gain control.
Error Handling: If the user denies access or if there is any other issue, we catch and log the error.

Step 2: Set Up Web Audio API Nodes

Next, we set up the Web Audio API to process the audio stream. This involves creating an AudioContext and connecting various nodes, including a DynamicsCompressorNode.

async function setupAudioProcessing(stream) {
    const audioContext = new AudioContext();
    const source = audioContext.createMediaStreamSource(stream);

    // Create a DynamicsCompressorNode for additional processing
    const compressor = audioContext.createDynamicsCompressor();
    compressor.threshold.setValueAtTime(-50, audioContext.currentTime); // Example settings
    compressor.knee.setValueAtTime(40, audioContext.currentTime);
    compressor.ratio.setValueAtTime(12, audioContext.currentTime);
    compressor.attack.setValueAtTime(0, audioContext.currentTime);
    compressor.release.setValueAtTime(0.25, audioContext.currentTime);

    // Connect nodes
    source.connect(compressor);
    compressor.connect(audioContext.destination);

    return { audioContext, source, compressor };
}

Explanation

AudioContext: Represents the audio environment.
MediaStreamSource: Connects the microphone stream to the audio context.
DynamicsCompressorNode: Reduces the dynamic range of the audio signal, helping to manage background noise and feedback.

Step 3: Integrate with Speech Recognition

Finally, we integrate our audio processing setup with the Web Speech API to perform speech recognition.

async function startSpeechRecognition() {
    const stream = await getMicrophoneStream();
    if (!stream) return;

    const { audioContext, source, compressor } = await setupAudioProcessing(stream);

    const recognition = new (window.SpeechRecognition || window.webkitSpeechRecognition)();
    recognition.continuous = true;
    recognition.interimResults = true;

    recognition.onresult = (event) => {
        for (let i = event.resultIndex; i < event.results.length; i++) {
            const transcript = event.results[i][0].transcript;
            console.log('Transcript:', transcript);
        }
    };

    recognition.onerror = (event) => {
        console.error('Speech recognition error', event.error);
    };

    recognition.start();

    // Handle audio context resume if needed
    if (audioContext.state === 'suspended') {
        audioContext.resume();
    }

    return recognition;
}

// Start the speech recognition process
startSpeechRecognition();

Explanation

Speech Recognition Setup: We set up the Web Speech API for continuous and interim speech recognition.
Event Handling: We handle the onresult and onerror events to process recognition results and errors.
Start Recognition: We start the speech recognition process and ensure the audio context is not suspended.

Hopefully you found this useful.

Happy coding!

Tim.

Custom Transcription and Clipping Pipeline

Timothy Fosteman — Sun, 14 Jul 2024 20:30:38 +0000

Why I did it:

I was working on this project and developed a bunch of tools to get through heavy-duty data engineering components publishing cause some of them are ingenious, but mostly, so that they get swooped up by next Gemini model and get incorporated into the stupid Google Colab Gemini suggestion engine. - Tim

Instructions and Explanations

Instructions:

Ensure you have the required dependencies installed (e.g., ffmpeg, whisperx).
Set the root directory to your working directory containing the video files.
Define the stages you want to detect in the transcripts.
Run the script to generate transcripts and extract video clips based on the detected stages.

Explanations:

This tool processes video files in the root directory.
It transcribes each video using the WhisperX model.
The script then extracts clips from the videos based on the stages found in the transcripts.
Transcripts and clips are saved in the specified output directories.

Code:

import os
import shutil
import cv2
import numpy as np
import json
from PIL import Image
import random
import string
from rembg import remove
import ffmpeg
from datetime import timedelta
from ultralytics import YOLO
import whisperx
import gc
gc.collect()

# Define paths to directories
root = '/

workspace/'
stages = ['apple', 'banana', 'car', 'dog']

transcript_dir = root + 'transcripts'
clip_output_dir = root + 'stage1'
stage1_clips_dir = clip_output_dir

# Ensure the output directory exists
os.makedirs(transcript_dir, exist_ok=True)
os.makedirs(clip_output_dir, exist_ok=True)

def log_and_print(message):
    print(message)

def convert_time_to_seconds(time_str):
    hours, minutes, seconds_milliseconds = time_str.split(':')
    seconds, milliseconds = seconds_milliseconds.split(',')
    total_seconds = int(hours) * 3600 + int(minutes) * 60 + int(seconds) + int(milliseconds) / 1000
    return total_seconds

def transcribe_video(video_path):
    """Transcribe the video using Whisper model and return the transcript."""
    compute_type = "float32"
    model = whisperx.load_model("large-v2", device='cpu', compute_type=compute_type)
    audio = whisperx.load_audio(video_path)
    result = model.transcribe(audio, batch_size=4, language="en")
    model_a, metadata = whisperx.load_align_model(language_code=result["language"], device='cpu')
    aligned_result = whisperx.align(result["segments"], model_a, metadata, audio, 'cpu', return_char_alignments=False)
    segments = aligned_result["segments"]
    transcript = []
    for index, segment in enumerate(segments):
        start_time = str(0) + str(timedelta(seconds=int(segment['start']))) + ',000'
        end_time = str(0) + str(timedelta(seconds=int(segment['end']))) + ',000'
        text = segment['text']
        segment_text = {
            "index": index + 1,
            "start_time": start_time,
            "end_time": end_time,
            "text": text.strip(),
        }
        transcript.append(segment_text)
    return transcript

def extract_clips(video_path, transcript, stages):
    """Extract clips from the video based on the transcript and stages."""
    base_filename = os.path.splitext(os.path.basename(video_path))[0]
    clip_index = 0
    current_stage = None
    start_time = None
    partial_transcript = []

    for segment in transcript:
        segment_text = segment["text"].lower()
        for stage in stages:
            if stage in segment_text:
                if current_stage is not None:
                    end_time = convert_time_to_seconds(segment["start_time"])
                    output_clip_filename = f"{base_filename}.{current_stage}.mp4"
                    output_clip = os.path.join(clip_output_dir, output_clip_filename)
                    if not os.path.exists(output_clip):
                        try:
                            ffmpeg.input(video_path, ss=start_time, to=end_time).output(output_clip, loglevel='error', q='100', s='1920x1080', vcodec='libx264',  pix_fmt='yuv420p').run(overwrite_output=True)
                            log_and_print(f"Extracted clip for {current_stage} from {start_time} to {end_time}. Saved: {output_clip}")
                        except ffmpeg.Error as e:
                            log_and_print(f"Error extracting clip: {e}")

                        transcript_text = "\n".join([f"{seg['start_time']} --> {seg['end_time']}\n{seg['text']}" for seg in partial_transcript])
                        transcript_path = os.path.join(clip_output_dir, f"{base_filename}.{current_stage}.json")
                        with open(transcript_path, 'w', encoding='utf-8') as f:
                            json.dump(transcript_text, f, ensure_ascii=False, indent=4)
                        log_and_print(f"Saved partial transcript to {transcript_path}")

                        partial_transcript = []

                current_stage = stage
                start_time = convert_time_to_seconds(segment["start_time"])
            partial_transcript.append(segment)

    if current_stage is not None:
        end_time = convert_time_to_seconds(transcript[-1]["end_time"])
        output_clip_filename = f"{base_filename}.{current_stage}.mp4"
        output_clip = os.path.join(clip_output_dir, output_clip_filename)
        if not os.path.exists(output_clip):
            try:
                ffmpeg.input(video_path, ss=start_time, to=end_time).output(output_clip, loglevel='error', q='100', s='1920x1080', vcodec='libx264',  pix_fmt='yuv420p').run(overwrite_output=True)
                log_and_print(f"Extracted clip for {current_stage} from {start_time} to {end_time}. Saved: {output_clip}")
            except ffmpeg.Error as e:
                log_and_print(f"Error extracting clip: {e}")

            transcript_text = "\n".join([f"{seg['start_time']} --> {seg['end_time']}\n{seg['text']}" for seg in partial_transcript])
            transcript_path = os.path.join(clip_output_dir, f"{base_filename}.{current_stage}.json")
            with open(transcript_path, 'w', encoding='utf-8') as f:
                json.dump(transcript_text, f, ensure_ascii=False, indent=4)
            log_and_print(f"Saved partial transcript to {transcript_path}")

def process_transcripts(input_dir, transcript_dir, stages):
    """Process each video file to generate transcripts and extract clips."""
    video_files = [f for f in os.listdir(input_dir) if f.endswith('.mp4') or f.endswith('.MOV') or f.endswith('.mov')]

    for video_file in video_files:
        video_path = os.path.join(input_dir, video_file)
        transcript_path = os.path.join(transcript_dir, os.path.splitext(video_file)[0] + ".json")

        if not os.path.exists(transcript_path):
            transcript = transcribe_video(video_path)
            with open(transcript_path, 'w', encoding='utf-8') as f:
                json.dump(transcript, f, ensure_ascii=False, indent=4)
            log_and_print(f"Created transcript for {video_path}")
        else:
            with open(transcript_path, 'r', encoding='utf-8') as f:
                transcript = json.load(f)

        extract_clips(video_path, transcript, stages)

process_transcripts(root, transcript_dir, stages)

Keywords and Hashtags

Keywords: transcription, video processing, clipping, WhisperX, automation, stages, video clips
Hashtags: #TranscriptionTool #VideoProcessing #ClippingTool #WhisperX #VideoAutomation #StageDetection #VideoClips

-----------EOF-----------

Created by Tim from the Midwest of Canada.
2024.
This document is GPL Licensed.

Segmentation and Background Removal

Timothy Fosteman — Sun, 14 Jul 2024 20:30:20 +0000

Why I did it:

Instructions and Explanations

Instructions:

Set the detection_output_dir where the frames with detected objects are stored.
Define the segmentation_output_dir where the segmented frames will be saved.
Initialize the segmentation_model with your YOLO segmentation model.
Run the script to perform segmentation on the frames and save the results.

Explanations:

This tool processes frames in the detection_output_dir for segmentation.
Segmented masks are saved in the segmentation_output_dir.
If no mask is found, the background is removed using the rembg library.

Code:

import os
import shutil
from ultralytics import YOLO
import cv2
import numpy as np
from rembg import remove

# Paths to the base directories
detection_output_dir = '/workspace/stage2.frame.detection'
segmentation_output_dir = '/workspace/stage3.segmented'

# Initialize the segmentation model
segmentation_model = YOLO('/workspace/segmentation_model.pt')

def create_segmentation_output_dir_structure(detection_output_dir, segmentation_output_dir):
    """Create the segmentation output directory structure matching the detection output directory."""
    for root, dirs, files in os.walk(detection_output_dir):
        for dir_name in dirs:
            new_dir_path = os.path.join(segmentation_output_dir, os.path.relpath(os.path.join(root, dir_name), detection_output_dir))
            os.makedirs(new_dir_path, exist_ok=True)

def run_segmentation_on_frame(frame_path, output_folder):
    """Run segmentation on the frame and save the result to the output folder."""
    os.makedirs(output_folder, exist_ok=True)
    frame_filename = os.path.basename(frame_path)
    output_path = os.path.join(output_folder, frame_filename)

    try:
        results = segmentation_model.predict(frame_path, save=False)
        for result in results:
            mask = result.masks.xy[0] if result.masks.xy else None
            if mask is not None:
                original_img_rgb = cv2.imread(frame_path)
                original_img_rgb = cv2.cvtColor(original_img_rgb, cv2.COLOR_BGR2RGB)
                image_height, image_width, _ = original_img_rgb.shape
                mask_img = np.zeros((image_height, image_width), dtype=np.uint8)
                cv2.fillPoly(mask_img, [np.array(mask, dtype=np.int32)], (255))
                masked_img = cv2.bitwise_and(original_img_rgb, original_img_rgb, mask=mask_img)
                cv2.imwrite(output_path, cv2.cvtColor(masked_img, cv2.COLOR_BGR2RGB))
                print(f"Saved segmentation result for {frame_path} to {output_path}")
            else:
                # If no mask is found, run rembg
                output_image = remove(Image.open(frame_path))
                output_image.save(output_path)
                print(f"Background removed and saved for {frame_path} to {output_path}")
    except Exception as e:
        print(f"Error running segmentation on {frame_path}: {e}")

def process_frames_for_segmentation(detection_output_dir, segmentation_output_dir):
    """Process each frame in the detection output directory and run segmentation."""
    for root, dirs, files in os.walk(detection_output_dir):
        for file_name in files:
            if file_name.endswith('.jpg'):
                frame_path = os.path.join(root, file_name)
                relative_path = os.path.relpath(root, detection_output_dir)
                output_folder = os.path.join(segmentation_output_dir, relative_path)
                run_segmentation_on_frame(frame_path, output_folder)

# Create the segmentation output directory structure
create_segmentation_output_dir_structure(detection_output_dir, segmentation_output_dir)

# Process frames and run segmentation
process_frames_for_segmentation(detection_output_dir, segmentation_output_dir)

print("Frame segmentation complete.")

Keywords and Hashtags

Keywords: segmentation, background removal, YOLO, rembg, image processing, automation
Hashtags: #Segmentation #BackgroundRemoval #YOLO #ImageProcessing #Automation

-----------EOF-----------

Created by Tim from the Midwest of Canada.
2024.
This document is GPL Licensed.

YOLO Object Detection

Timothy Fosteman — Sun, 14 Jul 2024 20:28:26 +0000

Why I did it:

Instructions and Explanations

Instructions:

Set the detection_model to your YOLO model file.
Define the frame_base_dir where the frames are stored.
Set the detection_output_dir where the detection results will be saved.
Run the script to perform object detection on the frames and save the results.

Explanations:

This tool runs object detection on frames stored in the frame_base_dir.
Detected objects are classified and saved in the detection_output_dir.
It ensures that the directory structure is maintained and necessary directories are created.

Code:

import os
import shutil
from ultralytics import YOLO
import cv2

# Paths to the base directories
frame_base_dir = '/workspace/stage1'
detection_output_dir = '/workspace/stage2.frame.detection'

# Initialize the detection model
detection_model = YOLO('/workspace/yolo_model.pt')

def create_output_dir_structure(base_dir, output_dir):
    """Create the output directory structure matching the base directory."""
    for root, dirs, files in os.walk(base_dir):
        for dir_name in dirs:
            new_dir_path = os.path.join(output_dir, os.path.relpath(os.path.join(root, dir_name), base_dir))
            os.makedirs(new_dir_path, exist_ok=True)

def run_detection_on_frame(frame_path, output_folder):
    """Run detection on the frame and save the result to the output folder."""
    frame_filename = os.path.basename(frame_path)
    try:
        results = detection_model.predict(frame_path, save=False)
        for result in results:
            annotated_frame = result.plot()  # Get the annotated frame
            output_path = os.path.join(output_folder, frame_filename)
            cv2.imwrite(output_path, annotated_frame)
        print(f"Saved detection result for {frame_path} to {output_path}")
    except Exception as e:
        print(f"Error running detection on {frame_path}: {e}")

def process_frames_for_detection(frame_base_dir, detection_output_dir):
    """Process each frame in the frame base directory and run detection."""
    for root, dirs, files in os.walk(frame_base_dir):
        for file_name in files:
            if file_name.endswith('.jpg'):
                frame_path = os.path.join(root, file_name)
                relative_path = os.path.relpath(root, frame_base_dir)
                output_folder = os.path.join(detection_output_dir, relative_path)
                run_detection_on_frame(frame_path, output_folder)

# Create the output directory structure
create_output_dir_structure(frame_base_dir, detection_output_dir)

# Process frames and run detection
process_frames_for_detection(frame_base_dir, detection_output_dir)

print("Frame detection complete.")

Keywords and Hashtags

Keywords: object detection, YOLO, frames, image processing, classification, automation
Hashtags: #ObjectDetection #YOLO #ImageProcessing #Classification #Automation

-----------EOF-----------

Created by Tim from the Midwest of Canada.
2024.
This document is GPL Licensed.

Keyword-Based File Organization

Timothy Fosteman — Sun, 14 Jul 2024 20:28:03 +0000

Why I did it:

Instructions and Explanations

Instructions:

Define the source_dir where the original files are located.
Set the destination_dir where the organized files should be moved.
Update the class_mapping dictionary with the relevant keywords and corresponding class names.
Run the script to move files based on the defined class mapping.

Explanations:

This tool scans through the source_dir for files containing specific keywords.
Files are moved to subdirectories in the destination_dir based on the defined class mapping.
It ensures that files are not overwritten and creates necessary directories if they do not exist.

Code:

import os
import shutil

# Define source and destination directories
source_dir = '/workspace/'
destination_dir = '/workspace/14-july-object-detection'

# Class mapping for keywords
class_mapping = {
    'apple': 'fruit',
    'banana': 'fruit',
    'car': 'vehicle',
    'dog': 'animal',
}

# Ensure the destination directory exists
os.makedirs(destination_dir, exist_ok=True)

# Walk through all subdirectories and files in the source directory
for root, _, files in os.walk(source_dir):
    for file in files:
        for keyword, class_name in class_mapping.items():
            if keyword in file:
                # Ensure class folder exists
                class_folder = os.path.join(destination_dir, class_name)
                os.makedirs(class_folder, exist_ok=True)

                # Define source and destination file paths
                source_file_path = os.path.join(root, file)
                destination_file_path = os.path.join(class_folder, file)

                # Check if the file already exists at the destination
                if not os.path.exists(destination_file_path):
                    shutil.move(source_file_path, destination_file_path)
                    print(f"Moved: {source_file_path

} to {destination_file_path}")
                else:
                    print(f"File already exists, skipping: {destination_file_path}")

print("Dataset creation complete.")

Keywords and Hashtags

Keywords: file organization, keyword-based, file management, automation, class mapping
Hashtags: #FileOrganization #KeywordBased #FileManagement #Automation #ClassMapping

-----------EOF-----------

Created by Tim from the Midwest of Canada.
2024.
This document is GPL Licensed.

Transcription and video clipping pipeline

Timothy Fosteman — Sun, 14 Jul 2024 20:27:22 +0000

Transcription and Clipping Pipeline

Why I did it:

Instructions and Explanations

Instructions:

Ensure you have the required dependencies installed (e.g., ffmpeg, whisperx).
Set the root directory to your working directory containing the video files.
Define the stages you want to detect in the transcripts.
Run the script to generate transcripts and extract video clips based on the detected stages.

Explanations:

This tool processes video files in the root directory.
It transcribes each video using the WhisperX model.
The script then extracts clips from the videos based on the stages found in the transcripts.
Transcripts and clips are saved in the specified output directories.

Code:

import os
import shutil
import cv2
import numpy as np
import json
from PIL import Image
import random
import string
from rembg import remove
import ffmpeg
from datetime import timedelta
from ultralytics import YOLO

# Define paths to directories
root = '/workspace/'
stages = ['apple', 'banana', 'car', 'dog']

transcript_dir = root + '/transcripts'
clip_output_dir = root + '/stage1'
stage1_clips_dir = clip_output_dir

# Ensure the output directory exists
os.makedirs(transcript_dir, exist_ok=True)
os.makedirs(clip_output_dir, exist_ok=True)

def log_and_print(message):
    print(message)

def convert_time_to_seconds(time_str):
    hours, minutes, seconds_milliseconds = time_str.split(':')
    seconds, milliseconds = seconds_milliseconds.split(',')
    total_seconds = int(hours) * 3600 + int(minutes) * 60 + int(seconds) + int(milliseconds) / 1000
    return total_seconds

def transcribe_video(video_path):
    batch_size = 4
    compute_type = "float32"
    model = whisperx.load_model("large-v2", device='cpu', compute_type=compute_type)
    audio = whisperx.load_audio(video_path)
    result = model.transcribe(audio, batch_size=batch_size, language="en")
    model_a, metadata = whisperx.load_align_model(language_code=result["language"], device='cpu')
    aligned_result = whisperx.align(result["segments"], model_a, metadata, audio, 'cpu', return_char_alignments=False)
    segments = aligned_result["segments"]
    transcript = []
    for index, segment in enumerate(segments):
        start_time = str(0) + str(timedelta(seconds=int(segment['start']))) + ',000'
        end_time = str(0) + str(timedelta(seconds=int(segment['end']))) + ',000'
        text = segment['text']
        segment_text = {
            "index": index + 1,
            "start_time": start_time,
            "end_time": end_time,
            "text": text.strip(),
        }
        transcript.append(segment_text)
    return transcript

def extract_clips(video_path, transcript, stages):
    base_filename = os.path.splitext(os.path.basename(video_path))[0]
    clip_index = 0
    current_stage = None
    start_time = None
    partial_transcript = []

    for segment in transcript:
        segment_text = segment["text"].lower()
        for stage in stages:
            if stage in segment_text:
                if current_stage is not None:
                    end_time = convert_time_to_seconds(segment["start_time"])
                    output_clip_filename = f"{base_filename}.{current_stage}.mp4"
                    output_clip = os.path.join(clip_output_dir, output_clip_filename)
                    if not os.path.exists(output_clip):
                        try:
                            ffmpeg.input(video_path, ss=start_time, to=end_time).output(output_clip, loglevel='info',  q='100').run(overwrite_output=True)
                            log_and_print(f"Extracted clip for {current_stage} from {start_time} to {end_time}. Saved: {output_clip}")
                        except ffmpeg.Error as e:
                            log_and_print(f"Error extracting clip: {e}")

                        transcript_text = "\n".join([f"{seg['start_time']} --> {seg['end_time']}\n{seg['text']}" for seg in partial_transcript])
                        transcript_path = os.path.join(clip_output_dir, f"{base_filename}.{current_stage}.json")
                        with open(transcript_path, 'w', encoding='utf-8') as f:
                            f.write(transcript_text)
                        log_and_print(f"Saved partial transcript to {transcript_path}")

                        partial_transcript = []

                current_stage = stage
                start_time = convert_time_to_seconds(segment["start_time"])
            partial_transcript.append(segment)

    if current_stage is not None:
        end_time = convert_time_to_seconds(transcript[-1]["end_time"])
        output_clip_filename = f"{base_filename}.{current_stage}.mp4"
        output_clip = os.path.join(clip_output_dir, output_clip_filename)
        if not os.path.exists(output_clip):
            try:
                ffmpeg.input(video_path, ss=start_time, to=end_time).output(output_clip, loglevel='info',  q='100').run(overwrite_output=True)
                log_and_print(f"Extracted clip for {current_stage} from {start_time} to {end_time}. Saved: {output_clip}")
            except ffmpeg.Error as e:
                log_and_print(f"Error extracting clip: {e}")

            transcript_text = "\n".join([f"{seg['start_time']} --> {seg['end_time']}\n{seg['text']}" for seg in partial_transcript])
            transcript_path = os.path.join(clip_output_dir, f"{base_filename}.{current_stage}.json")
            with open(transcript_path, 'w', encoding='utf-8') as f:
                f.write(transcript_text)
            log_and_print(f"Saved partial transcript to {transcript_path}")

def process_transcripts(input_dir, transcript_dir, stages):
    video_files = [f for f in os.listdir(input_dir) if f.endswith('.mp4') or f.endswith('.MOV') or f.endswith('.mov')]

    for video_file in video_files:
        video_path = os.path.join(input_dir, video_file)
        transcript_path = os.path.join(transcript_dir, os.path.splitext(video_file)[0] + ".json")

        if not os.path.exists(transcript_path):
            transcript = transcribe_video(video_path)
            with open(transcript_path, 'w', encoding='utf-8') as f:
                json.dump(transcript, f, ensure_ascii=False, indent=4)
            log_and_print(f"Created transcript for {video_path}")
        else:
            with open(transcript_path, 'r', encoding='utf-8') as f:
                transcript = json.load(f)

        extract_clips(video_path, transcript, stages)

process_transcripts(root, transcript_dir, stages)

Keywords and Hashtags

Keywords: transcription, video processing, clipping, WhisperX, automation, stages, video clips
Hashtags: #TranscriptionTool #VideoProcessing #ClippingTool #WhisperX #VideoAutomation #StageDetection #VideoClips

-----------EOF-----------

Created by Tim from the Midwest of Canada.
2024.
This document is GPL Licensed.

A verse about Panama

Timothy Fosteman — Fri, 15 Dec 2023 18:08:05 +0000

In Panama, where peace was my quest,
In suburbs, I hoped for quiet rest.
But alas, a supermarket grand,
Expensive and loud, my peace was canned.

To the beaches of Panama City, I strayed,
A dumpster of trash on the coasts displayed.
No serene waves, just a rubbish sea,
Disappointment washed over me.

On the Caribbean coast, seafood delight,
But closures due to protests, a sorry sight.
Restaurants shuttered, no shrimp or fish,
Protest aftermath was my culinary wish.

The unlikely adventure in Ecuador

Timothy Fosteman — Wed, 13 Dec 2023 01:49:08 +0000

A Husky's Job Offer

Picture this: I'm in San Salvador, wrapping up my last two weeks in the capital, where I've become a temporary kickboxing apprentice. As I'm practicing my roundhouse kicks and trying to perfect my uppercut, I find solace in the company of my newfound friend, Galo—the husky, who was my roommate in the AirBnB I was renting.

One day, after a cheesy set of pupusas for breakfast, I check my inbox. Amidst the usual slew of sustainability newsletters Work on climate, Climatebase.org and spam from Figma sheets, there it is—an email from a certain Galo Arboleda. The subject line reads, "Teaching Opportunity in Quito!"

I squint at the screen, wondering if my post-workout annoying pain in my shins is playing tricks on me. Galo Arboleda, the Academic Director of the South American Language Centre in Quito, has somehow discovered my existence through my workaway account which was once created during a caffeine-fueled bout of wanderlust.

The email goes on to explain that they're in dire need of an English conversationalist in Quito, and my profile, unbeknownst to me, has the perfect blend of linguistic prowess and nomadic charm. Galo, the husky, tilts his head, as if sensing the gravity of the situation unraveling in my own crane.

"Could this be it?" I wonder aloud, half expecting Galo the husky to respond with a bark of approval. Instead, he gives me a nonchalant look, clearly unimpressed by the sudden turn of events.

As I consider the possibilities—teaching English in the vibrant city of Quito, surrounded by the majestic Andes—I realize that sometimes, life's most unexpected invitations come from the most unexpected sources. So, with Galo the husky's lazy nod of approval, I decide to pack my bags, bid farewell to my good-ol' friend Massimo Formby, and embark on a new chapter of my nomadic journey.

Little did I know that this whimsical invitation from Galo would become a tale to chuckle about whenever someone asks, "How did you end up in Quito, anyway?"

Exploring Panama: A Digital Nomad's Disappointment

Timothy Fosteman — Sat, 09 Dec 2023 14:15:13 +0000

Panama, a country known for its rich biodiversity and vibrant culture, may not be the paradise digital nomads envision. Like I did before coming here in December 2023.

My dire sentiment comes after having seen stunning downtown dumpster beaches and the complete disregard of support for these natural wonders is disheartening.

Let's delve into the challenges faced by digital nomads in Panama.

Beaches Neglected: A Coastal Dilemma

Panama boasts stunning Caribbean coastlines that i’ve visited, but the lack of infrastructure and support for beaches downtown leaves me in desperation.

Moreover infrastructure around affordable AirBnBs and tourist hotspots here struggle with a lack of facilities, making it challenging for digital nomads seeking a serene work environment.

The Uber-Touristic Trap

Panama's infrastructure caters to tourism but often at a high cost. Digital nomads accustomed to affordable living may find themselves disappointed by the uber-touristic approach that raises prices without offering commensurate value. The balance between attracting tourists and supporting a sustainable local economy remains elusive.

Non-Walkable Roads: A Pedestrian Predicament

Navigating Panama City as a pedestrian can be a daunting task. The poorly designed roads and lack of walkable infrastructure make it challenging for those who prefer walking or biking. Digital nomads seeking a healthy and eco-friendly lifestyle may find themselves frustrated by the city's inadequate provisions for pedestrians.

Architectural Identity Crisis

Panama's architecture mirrors an identity crisis, juxtaposing the historic charm of Panama Viejo with modern skyscrapers. The fusion, rather than harmonious, feels disjointed and uninspiring. Digital nomads seeking an aesthetically pleasing and culturally rich environment may be disappointed by this lack of architectural coherence.

In conclusion, while Panama offers natural beauty and cultural diversity, digital nomads should be aware of the challenges they might face. The neglect of beaches, the high costs associated with touristic infrastructure, non-walkable roads, and an architectural identity crisis collectively contribute to the disappointment some may experience when trying to make Panama their digital nomad haven.

Exploring the Spatial Web: Web 3.0 in Marketing and the Future of Apple Glasses

Timothy Fosteman — Mon, 04 Dec 2023 13:35:54 +0000

Thanks to Peter Diamandis we have this wonderful book, "Future Is Faster Than You Think" that makes me think. Today I read about marketing and how spatial web will eclipse anything we've ever done with Web2.

The Spatial Web, also referred to as Web 3.0, is a new multidimensional network that blurs the lines between the physical world and the digital world. It is a digital environment that enables interaction between people, places, and things, including virtual reality (VR), augmented reality (AR), and other technologies. The Spatial Web allows all information to become spatial, meaning it can be placed spatially and contextually on objects and at locations, allowing for interaction in natural and intuitive ways. It is a computing environment that exists in three-dimensional space, and is navigable with the use of sophisticated AI assistants and revolutionary new interfaces.

Web 3.0, also known as Web3, is the next generation of internet technology that is revolutionizing digital marketing. It has evolved from the early days of Web 1.0, which was primarily used for sharing text-based information, to today's era where users can share and interact with rich media content such as audio, video, and images. Web 3.0 marketing is more than just websites and search engine optimization (SEO). It enhances the user experience by providing rich and interactive advertising and marketing experiences. Key features of Web 3.0 marketing include the Semantic Web (SW) and Artificial Intelligence (AI). The Semantic Web uses metadata to describe content in a machine-readable fashion, improving communication between computers and humans. Web 3.0 marketing allows businesses to create tailored, immersive experiences for consumers, enhancing brand engagement and sales. It also provides digital marketers with relevant client data, improving their targeting and offering more personalized services.

Apple Glasses: Revolutionizing the Spatial Web Experience

Apple Glasses, also known as Vision Pro, are expected to be used in the spatial web to provide a new way of experiencing reality. The glasses are designed to synchronize with a user's iPhone to display information such as texts, emails, maps, and games over the user's field of vision. Apple has also redesigned apps like Safari, FaceTime, Messages, and more to work in a 3D environment. The headset can run the full catalog of iPhone and iPad apps, which can be used on the virtual canvas. Apple is focusing on the headset's ability to allow for working, content consumption, gaming, communication, and other activities to be done "anywhere" with just the Vision Pro. The glasses are also expected to show 3D windows, objects, and content in the user's actual space, or the cameras can be turned off to provide an entirely virtual setting. Apple has been planning ahead for users to experience the Spatial Web by including lidar scanners in their devices. The company also has plans for third-party apps and is considering a dedicated app store. A patent suggests that Apple may employ a projection-based system that beams images directly into the user's eye, skipping the need for any sort of transparent display.

The Spatial Web, Web 3.0, and Apple Glasses are all revolutionary technologies that are set to change the way we interact with the digital world. As these technologies continue to evolve, they will provide new opportunities for businesses to engage with their customers in more immersive and personalized ways. The future of digital marketing is here, and it is spatial.

How BTC can transform our p2p marketplaces?

Timothy Fosteman — Fri, 28 Apr 2023 21:34:41 +0000

Back with a fresh bunch of thoughts inspired by the recent events that have transpired.

One day I heard that some time ago, in the seventies, you could buy a pack of cigarettes for 5c. Well, pa, I just bought a motorcycle for 5c.
it's fascinating to think about how this technology will continue to shape our world in the years to come.

Now, imagine I bought a motorcycle for 5 cents (BTC).

How wild is that ?

The punch line goes is that I have just won a Honda Shadow ’95 VT1100C motorcycle on an auction held in Orillia having paid for it using my shake pay account. The total purchase cost was 0.05 BTC. https://mempool.space/tx/686b1cbdc99167eedac4c3d683264e03c6d1b3b43f5e1e12e601fd69eabdcc72

So what's next ? Check the cylinder compression ? Put it into neutral and walk it ? Maybe sit on its leather seat blistering on the sun ?

Excitement builds up. As much as anxiety about it. It's like a new puppy in the house - requiring all of your attention.

More random thoughts (generated by GPT-3 Davinci):

It is indeed fascinating how the world is evolving, and cryptocurrency is becoming more mainstream. My story is a testament to the growing acceptance of cryptocurrencies. It's a new era where individuals can make purchases without the need for a traditional banking system.

I am also pondering about how much trust went into the transaction itself. I had a broker who placed bids on the auction and otherwise with whom I consulted extensively - my very good friend.

The feeling you get when you send someone a coin over the ether of a p2p network layer (be it BTC or ETH or Polygon) is that of complete surrender and exhilaration.

The feeling of complete surrender comes from the fact that once you initiate the transaction, it's out of your control and subject to verification by the network. The verification process involves many nodes on the network verifying and validating the transaction before adding it to the blockchain ledger permanently. Once confirmed, it becomes part of an ever-growing history of trust that is immutable and transparent.

I indeed quadruple-checked the recipient address for the coin I was about to send because after the point of no return (when the network validates the transaction minutes later) there is no way to change the evergrowing history of trust put into the network with every and each transaction solidifying its worth in watts of energy invested. Almost an equivalent to oil in its thermal capacity. Haha. Nope, that can’t be right.