DEV Community: Laurent Picard

Detecting and Editing Visual Objects with Gemini

Laurent Picard — Tue, 03 Mar 2026 17:00:11 +0000

A few notes before we start:

The complete source code for this article, including future updates, is available in this notebook under the Apache 2.0 license.
All new images in this article were generated with Gemini Nano Banana using the explored proof-of-concept. All source images are either in the public domain or free to use (reference links are provided in the code output).
You can experiment with Gemini models for free in Google AI Studio. For programmatic API access, please note that while a free tier is available for some models (i.e., you can perform object detection), image generation is a pay-as-you-go service.

✨ Overview

Traditional computer vision models are typically trained to detect a fixed set of object classes, like "person", "cat", or "car". If you want to detect something specific that wasn't in the training set, such as an "illustration" in a book photograph, you usually have to gather a dataset, label it manually, and train a custom model, which can take hours or even days.

In this exploration, we'll test a different approach using Gemini. We will leverage its spatial understanding capabilities to perform open-vocabulary object detection. This allows us to find objects based solely on a natural language description, without any training.

Once the visual objects are detected, we'll extract them and then use Gemini's image editing capabilities (specifically the Nano Banana models) to restore and creatively transform them.

🔥 Challenge

We are dealing with unstructured data: photos of books, magazines, and objects in the wild. These images present several difficulties for traditional computer vision:

Variety: The objects we want to find (illustrations, engravings, and any visuals in general) vary wildly in style and content.
Distortion: Pages are curved, photos are taken at angles, and lighting is uneven.
Noise: Old books have stains, paper grain, and text bleeding through from the other side.

Our challenge is to build a robust pipeline that can detect these objects despite the distortions, extract them cleanly, and edit them to look like high-quality digital assets… all using simple text prompts.

🏁 Setup

🐍 Python packages

We'll use the following packages:

google-genai: the Google Gen AI Python SDK lets us call Gemini with a few lines of code
pillow for image management
matplotlib for result visualization

We'll also use these packages (dependencies of google-genai):

pydantic for data management
tenacity for request management

pip install --quiet "google-genai>=1.64.0" "pillow>=11.3.0" "matplotlib>=3.10.0"

🔗 Gemini API

To use the Gemini API, we have two main options:

Via Vertex AI with a Google Cloud project
Via Google AI Studio with a Gemini API key

The Google Gen AI SDK provides a unified interface to these APIs, and we can use environment variables for the configuration. 🔽

🛠️ Option 1 - Gemini API via Vertex AI

Requirements:

A Google Cloud project
The Vertex AI API must be enabled for this project: ▶️ Enable the Vertex AI API

Gen AI SDK environment variables:

GOOGLE_GENAI_USE_VERTEXAI="True"
GOOGLE_CLOUD_PROJECT="<PROJECT_ID>"
GOOGLE_CLOUD_LOCATION="<LOCATION>"

💡 For preview models, the location must be set to global. For generally available models, we can choose the closest location among the Google model endpoint locations.

ℹ️ Learn more about setting up a project and a development environment.

🛠️ Option 2 - Gemini API via Google AI Studio

Requirement:

A Gemini API key

Gen AI SDK environment variables:

GOOGLE_GENAI_USE_VERTEXAI="False"
GOOGLE_API_KEY="<API_KEY>"

ℹ️ Learn more about getting a Gemini API key from Google AI Studio.

💡 You can store your environment configuration outside of the source code:

Environment	Method
IDE	`.env` file (or equivalent)
Colab	Colab Secrets (🗝️ icon in left panel, see code below)
Colab Enterprise	Google Cloud project and location are automatically defined
Vertex AI Workbench	Google Cloud project and location are automatically defined

Define the following environment detection functions. You can also define your configuration manually if needed. 🔽

# @title {display-mode: "form"}
import os
import sys
from collections.abc import Callable

from google import genai

# Manual setup (leave unchanged if setup is environment-defined)

# @markdown **Which API: Vertex AI or Google AI Studio?**
GOOGLE_GENAI_USE_VERTEXAI = True  # @param {type: "boolean"}

# @markdown **Option A - Google Cloud project [+location]**
GOOGLE_CLOUD_PROJECT = ""  # @param {type: "string"}
GOOGLE_CLOUD_LOCATION = "global"  # @param {type: "string"}

# @markdown **Option B - Google AI Studio API key**
GOOGLE_API_KEY = ""  # @param {type: "string"}


def check_environment() -> bool:
    check_colab_user_authentication()
    return check_manual_setup() or check_vertex_ai() or check_colab() or check_local()


def check_manual_setup() -> bool:
    return check_define_env_vars(
        GOOGLE_GENAI_USE_VERTEXAI,
        GOOGLE_CLOUD_PROJECT.strip(),  # Might have been pasted with line return
        GOOGLE_CLOUD_LOCATION,
        GOOGLE_API_KEY,
    )


def check_vertex_ai() -> bool:
    # Workbench and Colab Enterprise
    match os.getenv("VERTEX_PRODUCT", ""):
        case "WORKBENCH_INSTANCE":
            pass
        case "COLAB_ENTERPRISE":
            if not running_in_colab_env():
                return False
        case _:
            return False

    return check_define_env_vars(
        True,
        os.getenv("GOOGLE_CLOUD_PROJECT", ""),
        os.getenv("GOOGLE_CLOUD_REGION", ""),
        "",
    )


def check_colab() -> bool:
    if not running_in_colab_env():
        return False

    # Colab Enterprise was checked before, so this is Colab only
    from google.colab import auth as colab_auth  # type: ignore

    colab_auth.authenticate_user()

    # Use Colab Secrets (🗝️ icon in left panel) to store the environment variables
    # Secrets are private, visible only to you and the notebooks that you select
    # - Vertex AI: Store your settings as secrets
    # - Google AI: Directly import your Gemini API key from the UI
    vertexai, project, location, api_key = get_vars(get_colab_secret)

    return check_define_env_vars(vertexai, project, location, api_key)


def check_local() -> bool:
    vertexai, project, location, api_key = get_vars(os.getenv)

    return check_define_env_vars(vertexai, project, location, api_key)


def running_in_colab_env() -> bool:
    # Colab or Colab Enterprise
    return "google.colab" in sys.modules


def check_colab_user_authentication() -> None:
    if running_in_colab_env():
        from google.colab import auth as colab_auth  # type: ignore

        colab_auth.authenticate_user()


def get_colab_secret(secret_name: str, default: str) -> str:
    from google.colab import errors, userdata  # type: ignore

    try:
        return userdata.get(secret_name)
    except errors.SecretNotFoundError:
        return default


def disable_colab_cell_scrollbar() -> None:
    if running_in_colab_env():
        from google.colab import output  # type: ignore

        output.no_vertical_scroll()


def get_vars(getenv: Callable[[str, str], str]) -> tuple[bool, str, str, str]:
    # Limit getenv calls to the minimum (may trigger UI confirmation for secret access)
    vertexai_str = getenv("GOOGLE_GENAI_USE_VERTEXAI", "")
    if vertexai_str:
        vertexai = vertexai_str.lower() in ["true", "1"]
    else:
        vertexai = bool(getenv("GOOGLE_CLOUD_PROJECT", ""))

    project = getenv("GOOGLE_CLOUD_PROJECT", "") if vertexai else ""
    location = getenv("GOOGLE_CLOUD_LOCATION", "") if project else ""
    api_key = getenv("GOOGLE_API_KEY", "") if not project else ""

    return vertexai, project, location, api_key


def check_define_env_vars(
    vertexai: bool,
    project: str,
    location: str,
    api_key: str,
) -> bool:
    match (vertexai, bool(project), bool(location), bool(api_key)):
        case (True, True, _, _):
            # Vertex AI - Google Cloud project [+location]
            location = location or "global"
            define_env_vars(vertexai, project, location, "")
        case (True, False, _, True):
            # Vertex AI - API key
            define_env_vars(vertexai, "", "", api_key)
        case (False, _, _, True):
            # Google AI Studio - API key
            define_env_vars(vertexai, "", "", api_key)
        case _:
            return False

    return True


def define_env_vars(vertexai: bool, project: str, location: str, api_key: str) -> None:
    os.environ["GOOGLE_GENAI_USE_VERTEXAI"] = str(vertexai)
    os.environ["GOOGLE_CLOUD_PROJECT"] = project
    os.environ["GOOGLE_CLOUD_LOCATION"] = location
    os.environ["GOOGLE_API_KEY"] = api_key


def check_configuration(client: genai.Client) -> None:
    service = "Vertex AI" if client.vertexai else "Google AI Studio"
    print(f"✅ Using the {service} API", end="")

    if client._api_client.project:
        print(f' with project "{client._api_client.project[:7]}…"', end="")
        print(f' in location "{client._api_client.location}"')
    elif client._api_client.api_key:
        api_key = client._api_client.api_key
        print(f' with API key "{api_key[:5]}…{api_key[-5:]}"', end="")
        print(f" (in case of error, make sure it was created for {service})")


print("✅ Environment functions defined")

🤖 Gen AI SDK

To send Gemini requests, create a google.genai client:

from google import genai

check_environment()

client = genai.Client()

check_configuration(client)

🖼️ Image test suite

Let's define a list of images for our tests: 🔽

from dataclasses import dataclass
from enum import StrEnum

Url = str


class Source(StrEnum):
    incunable = "https://tile.loc.gov/image-services/iiif/service:rbc:rbc0001:2014:2014rosen0487:0165/full/pct:25/0/default.jpg"
    engravings = "https://tile.loc.gov/image-services/iiif/service:gdc:gdcscd:00:34:07:66:92:1:00340766921:0121/full/pct:50/0/default.jpg"
    museum_guidebook = "https://tile.loc.gov/image-services/iiif/service:rbc:rbc0001:2014:2014gen34181:0033/full/pct:75/0/default.jpg"
    denver_illustrated = "https://tile.loc.gov/image-services/iiif/service:gdc:gdclccn:rc:01:00:04:94:rc01000494:0051/full/pct:50/0/default.jpg"
    physics_textbook = "https://tile.loc.gov/image-services/iiif/service:gdc:gdcscd:00:03:64:87:31:8:00036487318:0103/full/pct:50/0/default.jpg"
    portrait_miniatures = "https://tile.loc.gov/image-services/iiif/service:rbc:rbc0001:2024:2024rosen013592v02:0249/full/pct:50/0/default.jpg"
    wizard_of_oz_drawings = "https://tile.loc.gov/image-services/iiif/service:rbc:rbc0001:2006:2006gen32405:0048/full/pct:25/0/default.jpg"
    paintings = "https://images.unsplash.com/photo-1714146681164-f26fed839692?h=1440"
    alice_drawing = "https://images.unsplash.com/photo-1630595011903-689853b04ee2?h=800"
    book = "https://images.unsplash.com/photo-1643451533573-ee364ba6e330?h=800"
    manual = "https://images.unsplash.com/photo-1623666936367-a100f62ba9b7?h=800"
    electronics = "https://images.unsplash.com/photo-1757397584789-8b2c5bfcdbc3?h=1440"


@dataclass
class SourceMetadata:
    title: str
    webpage_url: Url
    credit_line: str


LOC = "Library of Congress"
LOC_RARE_BOOKS = "Library of Congress, Rare Book and Special Collections Division"
LOC_MEETING_FRONTIERS = "Library of Congress, Meeting of Frontiers"

metadata_by_source: dict[Source, SourceMetadata] = {
    Source.incunable: SourceMetadata(
        "Vergaderinge der historien van Troy (1485)",
        "https://www.loc.gov/resource/rbc0001.2014rosen0487/?sp=165",
        LOC_RARE_BOOKS,
    ),
    Source.engravings: SourceMetadata(
        "Harper's illustrated catalogue (1847)",
        "https://www.loc.gov/resource/gdcscd.00340766921/?sp=121",
        LOC,
    ),
    Source.museum_guidebook: SourceMetadata(
        "Barnum's American Museum illustrated (1850)",
        "https://www.loc.gov/resource/rbc0001.2014gen34181/?sp=33",
        LOC_RARE_BOOKS,
    ),
    Source.denver_illustrated: SourceMetadata(
        "Denver illustrated (1893)",
        "https://www.loc.gov/resource/gdclccn.rc01000494/?sp=51",
        LOC_MEETING_FRONTIERS,
    ),
    Source.physics_textbook: SourceMetadata(
        "Lessons in physics (1916)",
        "https://www.loc.gov/resource/gdcscd.00036487318/?sp=103",
        LOC,
    ),
    Source.portrait_miniatures: SourceMetadata(
        "The history of portrait miniatures (1904)",
        "https://www.loc.gov/resource/rbc0001.2024rosen013592v02/?sp=249",
        LOC_RARE_BOOKS,
    ),
    Source.wizard_of_oz_drawings: SourceMetadata(
        "The wonderful Wizard of Oz (1899)",
        "https://www.loc.gov/resource/rbc0001.2006gen32405/?sp=48",
        LOC_RARE_BOOKS,
    ),
    Source.paintings: SourceMetadata(
        "Open book showing paintings by Vincent van Gogh",
        "https://unsplash.com/photos/9hD7qrxICag",
        "Photo by Trung Manh cong on Unsplash",
    ),
    Source.alice_drawing: SourceMetadata(
        "Open book showing an illustration and text from Alice's Adventures in Wonderland",
        "https://unsplash.com/photos/bewzr_Q9u2o",
        "Photo by Brett Jordan on Unsplash",
    ),
    Source.book: SourceMetadata(
        "Open book showing two botanical illustrations",
        "https://unsplash.com/photos/4IDqcNj827I",
        "Photo by Ranurte on Unsplash",
    ),
    Source.manual: SourceMetadata(
        "Open user manual for vintage camera",
        "https://unsplash.com/photos/aaFU96eYASk",
        "Photo by Annie Spratt on Unsplash",
    ),
    Source.electronics: SourceMetadata(
        "Circuit board with electronic components",
        "https://unsplash.com/photos/Aqa1pHQ57pw",
        "Photo by Albert Stoynov on Unsplash",
    ),
}

print("✅ Test images defined")

🧠 Gemini models

Gemini comes in different versions. We can currently use the following models:

For object detection: Gemini 2.5 or Gemini 3, each available in Flash or Pro versions.
For object editing: Gemini 2.5 Flash Image, Gemini 3 Pro Image, or Gemini 3.1 Flash Image (also known as Nano Banana 🍌, Nano Banana Pro 🍌, and Nano Banana 2 🍌).

🛠️ Helpers

Now, let's add core helper classes and functions: 🔽

from enum import auto
from pathlib import Path
from typing import Any, cast

import IPython.display
import matplotlib.pyplot as plt
import pydantic
import tenacity
from google.genai.errors import ClientError
from google.genai.types import (
    FinishReason,
    GenerateContentConfig,
    GenerateContentResponse,
    PIL_Image,
    ThinkingConfig,
    ThinkingLevel,
)


# Multimodal models with spatial understanding and structured outputs
class MultimodalModel(StrEnum):
    # Generally Available (GA)
    GEMINI_2_5_FLASH = "gemini-2.5-flash"
    GEMINI_2_5_PRO = "gemini-2.5-pro"
    # Preview
    GEMINI_3_FLASH_PREVIEW = "gemini-3-flash-preview"
    GEMINI_3_1_PRO_PREVIEW = "gemini-3.1-pro-preview"
    # Default model used for object detection
    DEFAULT = GEMINI_3_FLASH_PREVIEW


# Image generation and editing models (Nano Banana 🍌 models)
class ImageModel(StrEnum):
    # Generally Available (GA)
    GEMINI_2_5_FLASH_IMAGE = "gemini-2.5-flash-image"  # Nano Banana
    # Preview
    GEMINI_3_PRO_IMAGE_PREVIEW = "gemini-3-pro-image-preview"  # Nano Banana Pro
    GEMINI_3_1_FLASH_IMAGE_PREVIEW = "gemini-3.1-flash-image-preview"  # Nano Banana 2
    # Default model used for image editing
    DEFAULT = GEMINI_2_5_FLASH_IMAGE


Model = MultimodalModel | ImageModel


def generate_content(
    contents: list[Any],
    model: Model,
    config: GenerateContentConfig | None,
    should_display_response_info: bool = False,
) -> GenerateContentResponse | None:
    response = None
    client = check_client_for_model(model)

    for attempt in get_retrier():
        with attempt:
            response = client.models.generate_content(
                model=model.value,
                contents=contents,
                config=config,
            )
    if should_display_response_info:
        display_response_info(response, config)

    return response


def check_client_for_model(model: Model) -> genai.Client:
    if (
        model.value.endswith("-preview")
        and client.vertexai
        and client._api_client.location != "global"
    ):
        # Preview models are only available on the "global" location
        return genai.Client(location="global")

    return client


def display_response_info(
    response: GenerateContentResponse | None,
    config: GenerateContentConfig | None,
) -> None:
    if response is None:
        print("❌ No response")
        return

    if usage_metadata := response.usage_metadata:
        if usage_metadata.prompt_token_count:
            print(f"Input tokens   : {usage_metadata.prompt_token_count:9,d}")
        if usage_metadata.candidates_token_count:
            print(f"Output tokens  : {usage_metadata.candidates_token_count:9,d}")
        if usage_metadata.thoughts_token_count:
            print(f"Thoughts tokens: {usage_metadata.thoughts_token_count:9,d}")

    if (
        config is not None
        and config.response_mime_type == "application/json"
        and response.parsed is None
    ):
        print("❌ Could not parse the JSON response")
        return
    if not response.candidates:
        print("❌ No `response.candidates`")
        return
    if (finish_reason := response.candidates[0].finish_reason) != FinishReason.STOP:
        print(f"❌ {finish_reason = }")
    if not response.text:
        print("❌ No `response.text`")
        return


def generate_image(
    sources: list[PIL_Image],
    prompt: str,
    model: ImageModel,
    config: GenerateContentConfig | None = None,
) -> PIL_Image | None:
    contents = [*sources, prompt.strip()]

    response = generate_content(contents, model, config)

    return check_get_output_image_from_response(response)


def check_get_output_image_from_response(
    response: GenerateContentResponse | None,
) -> PIL_Image | None:
    if response is None:
        print("❌ No `response`")
        return None
    if not response.candidates:
        print("❌ No `response.candidates`")
        if response.prompt_feedback:
            if block_reason := response.prompt_feedback.block_reason:
                print(f"{block_reason = :s}")
            if block_reason_message := response.prompt_feedback.block_reason_message:
                print(f"{block_reason_message = }")
        return None
    if not (content := response.candidates[0].content):
        print("❌ No `response.candidates[0].content`")
        return None
    if not (parts := content.parts):
        print("❌ No `response.candidates[0].content.parts`")
        return None

    output_image: PIL_Image | None = None
    for part in parts:
        if part.text:
            display_markdown(part.text)
            continue
        sdk_image = part.as_image()
        assert sdk_image is not None
        output_image = sdk_image._pil_image
        assert output_image is not None
        break  # There should be a single image

    return output_image


def get_thinking_config(model: Model) -> ThinkingConfig | None:
    match model:
        case MultimodalModel.GEMINI_2_5_FLASH:
            return ThinkingConfig(thinking_budget=0)
        case MultimodalModel.GEMINI_2_5_PRO:
            return ThinkingConfig(thinking_budget=128, include_thoughts=False)
        case MultimodalModel.GEMINI_3_FLASH_PREVIEW:
            return ThinkingConfig(thinking_level=ThinkingLevel.MINIMAL)
        case MultimodalModel.GEMINI_3_1_PRO_PREVIEW:
            return ThinkingConfig(thinking_level=ThinkingLevel.LOW)
        case _:
            return None  # Default


def display_markdown(markdown: str) -> None:
    IPython.display.display(IPython.display.Markdown(markdown))


def display_image(image: PIL_Image) -> None:
    IPython.display.display(image)


def get_retrier() -> tenacity.Retrying:
    return tenacity.Retrying(
        stop=tenacity.stop_after_attempt(7),
        wait=tenacity.wait_incrementing(start=10, increment=1),
        retry=should_retry_request,
        reraise=True,
    )


def should_retry_request(retry_state: tenacity.RetryCallState) -> bool:
    if not retry_state.outcome:
        return False
    err = retry_state.outcome.exception()
    if not isinstance(err, ClientError):
        return False
    print(f"❌ ClientError {err.code}: {err.message}")

    retry = False
    match err.code:
        case 400 if err.message is not None and " try again " in err.message:
            # Workshop: first time access to Cloud Storage (service agent provisioning)
            retry = True
        case 429:
            # Workshop: temporary project with 1 QPM quota
            retry = True
    print(f"🔄 Retry: {retry}")

    return retry


print("✅ Helpers defined")

🔍 Detecting visual objects

To perform visual object detection, craft the prompt to indicate what you'd like to detect and how results should be returned. In the same request, it's possible to also extract additional information about each detected object. This can be virtually anything, from labels such as "furniture", "table", or "chair", to more precise classifications like "mammals" or "reptiles", or to contextual data such as captions, colors, shapes, etc.

For the next tests, we'll experiment with detecting illustrations within book photos. Here's a possible prompt:

OBJECT_DETECTION_PROMPT = """
Detect every illustration within the book photo and extract the following data for each:
- `box_2d`: Bounding box coordinates of the illustration only (ignoring any caption).
- `caption`: Verbatim caption or legend such as "Figure 1". Use "" if not found.
- `label`: Single-word label describing the illustration. Use "" if not found.
"""

Notes:

Bounding boxes are very useful for locating or extracting the detected objects.
Typically, for Gemini models, a box_2d bounding box represents coordinates normalized to a (0, 0, 1000, 1000) space for a (0, 0, width, height) input image.
We're also requesting to extract captions (metadata often present in reference books) and labels (dynamic metadata).

To automate response processing, it's convenient to define a Pydantic class that matches the prompt, such as:

class DetectedObject(pydantic.BaseModel):
    box_2d: list[int]
    caption: str
    label: str

DetectedObjects = list[DetectedObject]

Then, request a structured output with config fields response_mime_type and response_schema:

config = GenerateContentConfig(
    # …,
    response_mime_type="application/json",
    response_schema=DetectedObjects,
    # …,
)

This will generate a JSON response which the SDK can parse automatically, letting us directly use object instances:

detected_objects = cast(DetectedObjects, response.parsed)

Let's add a few object-detection-specific classes and functions: 🔽

import io
import urllib.request
from collections.abc import Iterator
from dataclasses import field
from datetime import datetime

import PIL.Image
from google.genai.types import Part, PartMediaResolutionLevel
from PIL.PngImagePlugin import PngInfo

OBJECT_DETECTION_PROMPT = """
Detect every illustration within the book photo and extract the following data for each:
- `box_2d`: Bounding box coordinates of the illustration only (ignoring any caption).
- `caption`: Verbatim caption or legend such as "Figure 1". Use "" if not found.
- `label`: Single-word label describing the illustration. Use "" if not found.
"""

# Margin added to detected/cropped objects, giving more context for a better understanding of spatial distortions
CROP_MARGIN_PX = 10

# Set to True to save each generated image
SAVE_GENERATED_IMAGES = False
OUTPUT_IMAGES_PATH = Path("./object_detection_and_editing")


# Matching class for structured output generation
class DetectedObject(pydantic.BaseModel):
    box_2d: list[int]
    caption: str
    label: str


# Misc data classes
InputImage = Path | Url
DetectedObjects = list[DetectedObject]
WorkflowStepImages = list[PIL_Image]


class WorkflowStep(StrEnum):
    SOURCE = auto()
    CROPPED = auto()
    RESTORED = auto()
    COLORIZED = auto()
    CINEMATIZED = auto()


@dataclass
class VisualObjectWorkflow:
    source_image: PIL_Image
    detected_objects: DetectedObjects
    images_by_step: dict[WorkflowStep, WorkflowStepImages] = field(default_factory=dict)

    def __post_init__(self) -> None:
        denormalize_bounding_boxes(self)


workflow_by_image: dict[InputImage, VisualObjectWorkflow] = {}


def denormalize_bounding_boxes(self: VisualObjectWorkflow) -> None:
    """Convert the box_2d coordinates.
    - Before: [y1, x1, y2, x2] normalized to 0-1000, as returned by Gemini
    - After:  [x1, y1, x2, y2] in source_image coordinates, as used in Pillow
    """

    def to_image_coord(coord: int, dim: int) -> int:
        return int(coord * dim / 1000 + 0.5)

    w, h = self.source_image.size
    for obj in self.detected_objects:
        y1, x1, y2, x2 = obj.box_2d
        x1, x2 = to_image_coord(x1, w), to_image_coord(x2, w)
        y1, y2 = to_image_coord(y1, h), to_image_coord(y2, h)
        obj.box_2d = [x1, y1, x2, y2]


def detect_objects(
    image: InputImage,
    prompt: str = OBJECT_DETECTION_PROMPT,
    model: MultimodalModel = MultimodalModel.DEFAULT,
    config: GenerateContentConfig | None = None,
    media_resolution: PartMediaResolutionLevel | None = None,
    display_results: bool = True,
) -> None:
    display_image_source_info(image)
    pil_image, content_part = get_pil_image_and_part(image, model, media_resolution)
    prompt = prompt.strip()
    contents = [content_part, prompt]
    config = config or get_object_detection_config(model)

    response = generate_content(contents, model, config)

    if response is not None and response.parsed is not None:
        detected_objects = cast(DetectedObjects, response.parsed)
    else:
        detected_objects = DetectedObjects()

    workflow = VisualObjectWorkflow(pil_image, detected_objects)
    workflow_by_image[image] = workflow
    add_cropped_objects(workflow, image, prompt)

    if display_results:
        display_detected_objects(workflow)


def get_pil_image_and_part(
    image: InputImage,
    model: MultimodalModel,
    media_resolution: PartMediaResolutionLevel | None,
) -> tuple[PIL_Image, Part]:
    if isinstance(image, Path):
        image_bytes = image.read_bytes()
    else:
        headers = {"User-Agent": "Mozilla/5.0"}
        req = urllib.request.Request(image, headers=headers)
        with urllib.request.urlopen(req, timeout=10) as response:
            image_bytes = response.read()

    pil_image = PIL.Image.open(io.BytesIO(image_bytes))
    mime_type = f"image/{pil_image.format.lower()}" if pil_image.format else "image/*"
    content_part = Part.from_bytes(
        data=image_bytes,
        mime_type=mime_type,
        media_resolution=media_resolution,
    )

    return pil_image, content_part


def get_object_detection_config(model: Model) -> GenerateContentConfig:
    # Low randomness for more determinism
    return GenerateContentConfig(
        temperature=0.0,
        top_p=0.0,
        seed=42,
        response_mime_type="application/json",
        response_schema=DetectedObjects,
        thinking_config=get_thinking_config(model),
    )


def add_cropped_objects(
    workflow: VisualObjectWorkflow,
    input: InputImage,
    prompt: str,
    crop_margin: int = CROP_MARGIN_PX,
) -> None:
    cropped_images: list[PIL_Image] = []
    obj_count = len(workflow.detected_objects)
    for obj_order, obj in enumerate(workflow.detected_objects, 1):
        cropped_image, _ = extract_object_image(workflow.source_image, obj, crop_margin)
        cropped_images.append(cropped_image)
        save_workflow_image(
            WorkflowStep.SOURCE,
            WorkflowStep.CROPPED,
            input,
            obj_order,
            obj_count,
            cropped_image,
            dict(prompt=prompt, crop_margin=str(crop_margin)),
        )
    workflow.images_by_step[WorkflowStep.CROPPED] = cropped_images


def extract_object_image(
    image: PIL_Image,
    obj: DetectedObject,
    margin: int = 0,
) -> tuple[PIL_Image, tuple[int, int, int, int]]:
    def clamp(coord: int, dim: int) -> int:
        return min(max(coord, 0), dim)

    x1, y1, x2, y2 = obj.box_2d
    w, h = image.size
    if margin != 0:
        x1, x2 = clamp(x1 - margin, w), clamp(x2 + margin, w)
        y1, y2 = clamp(y1 - margin, h), clamp(y2 + margin, h)

    box = (x1, y1, x2, y2)
    object_image = image.crop(box)

    return object_image, box


def save_workflow_image(
    source_step: WorkflowStep,
    target_step: WorkflowStep,
    input_image: InputImage,
    obj_order: int,
    obj_count: int,
    target_image: PIL_Image | None,
    image_info: dict[str, str] | None = None,
) -> None:
    if not SAVE_GENERATED_IMAGES or target_image is None:
        return
    OUTPUT_IMAGES_PATH.mkdir(parents=True, exist_ok=True)
    time_str = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
    try:
        filename = f"{Source(input_image).name}_"
    except ValueError:
        filename = ""
    filename += f"{obj_order}o{obj_count}_{source_step}_{target_step}_{time_str}.png"
    image_path = OUTPUT_IMAGES_PATH.joinpath(filename)
    params = {}
    if image_info:
        png_info = PngInfo()
        for k, v in image_info.items():
            png_info.add_text(k, v)
        params.update(pnginfo=png_info)
    target_image.save(image_path, **params)


# Matplotlib
FIGURE_FG_COLOR = "#F1F3F4"
FIGURE_BG_COLOR = "#202124"
EDGE_COLOR = "#80868B"
rcParams = {
    "figure.dpi": 300,
    "text.color": FIGURE_FG_COLOR,
    "figure.facecolor": FIGURE_BG_COLOR,
    "figure.edgecolor": FIGURE_FG_COLOR,
    "axes.titlecolor": FIGURE_FG_COLOR,
    "axes.edgecolor": EDGE_COLOR,
    "xtick.color": FIGURE_FG_COLOR,
    "ytick.color": FIGURE_FG_COLOR,
    "xtick.bottom": False,
    "xtick.top": False,
    "ytick.left": False,
    "ytick.right": False,
    "xtick.labelbottom": False,
    "ytick.labelleft": False,
}
plt.rcParams.update(rcParams)


def display_image_source_info(image: InputImage) -> None:
    def get_image_info_md() -> str:
        if image not in Source:
            return f"[[Source Image]({image})]"
        source = Source(image)
        metadata = metadata_by_source.get(source)
        if not metadata:
            return f"[[Source Image]({source.value})]"
        parts = [
            f"[Source Image]({source.value})",
            f"[Source Page]({metadata.webpage_url})",
            metadata.title,
            metadata.credit_line,
        ]
        separator = "•"
        inner_info = f" {separator} ".join(parts)
        return f"{separator} {inner_info} {separator}"

    def yield_md_rows() -> Iterator[str]:
        horizontal_line = "---"
        image_info = get_image_info_md()
        yield horizontal_line
        yield f"_{image_info}_"
        yield horizontal_line

    display_markdown(f"{chr(10)}{chr(10)}".join(yield_md_rows()))


def display_detected_objects(workflow: VisualObjectWorkflow) -> None:
    source_image = workflow.source_image
    detected_objects = PIL.Image.new("RGB", source_image.size, "white")
    for obj in workflow.detected_objects:
        obj_image, box = extract_object_image(source_image, obj)
        detected_objects.paste(obj_image, (box[0], box[1]))

    _, (ax1, ax2) = plt.subplots(1, 2, layout="compressed")
    ax1.imshow(source_image)
    ax2.imshow(detected_objects)

    disable_colab_cell_scrollbar()
    plt.show()


print("✅ Object detection helpers defined")

🧪 Let's start simple: can we detect the single illustration in this incunable from 1485?

detect_objects(Source.incunable)

• Source Image • Source Page • Vergaderinge der historien van Troy (1485) • Library of Congress, Rare Book and Special Collections Division •

💡 This works nicely. The bounding box is very precise, enclosing the hand-colored woodcut illustration very tightly.

🧪 Now, let's check the detection of the multiple visuals in this museum guidebook:

detect_objects(Source.museum_guidebook)

• Source Image • Source Page • Barnum's American Museum illustrated (1850) • Library of Congress, Rare Book and Special Collections Division •

💡 Remarks:

The bounding boxes are again very precise.

The results are perfect: there are no false positives and no false negatives.

The captions below the visuals are not enclosed within the bounding boxes, which was specifically requested. The bounding box granularity can be controlled by changing the prompt.

🧪 What about slightly warped visuals?

detect_objects(Source.paintings)

• Source Image • Source Page • Open book showing paintings by Vincent van Gogh • Photo by Trung Manh cong on Unsplash •

💡 This doesn't make a difference. Notice how the bottom-right painting is partially covered by the orange bookmark. We'll try to fix that in the restoration step.

🧪 What about the tilted visuals in this book about the architecture in Denver?

detect_objects(Source.denver_illustrated)

• Source Image • Source Page • Denver illustrated (1893) • Library of Congress, Meeting of Frontiers •

💡 Each visual is perfectly detected: spatial understanding covers tilted objects.

🧪 Finally, let's check the detection on this significantly warped book page from Alice's Adventures in Wonderland:

detect_objects(Source.alice_drawing)

• Source Image • Source Page • Open book showing an illustration and text from Alice's Adventures in Wonderland • Photo by Brett Jordan on Unsplash •

💡 Page curvature and other distortions don't prevent non-rectangular objects from being detected. In fact, spatial understanding works at the pixel level, which explains this precision for warped objects. If you'd like to work at a lower level, you can also ask for a "segmentation mask" in the prompt and you'll get a base64-encoded PNG (each pixel giving the 0-255 probability it belongs to the object within the bounding box). See the segmentation doc for more details.

🏷️ Text extraction and dynamic labeling

On top of localizing each object with its bounding box, our prompt requested to extract a verbatim caption and to assign a single-word label, when possible.

Let's add a simple function to display the detection data in a table: 🔽

from collections import defaultdict


def display_detection_data(source: Source, show_consolidated: bool = False) -> None:
    def string_with_visible_linebreaks(s: str) -> str:
        return f'''"{s.replace(chr(10), "↩️")}"'''

    def yield_md_rows_consolidated(workflow: VisualObjectWorkflow) -> Iterator[str]:
        yield "| label | count | captions |"
        yield "| :--- | ---: | :--- |"
        stats = defaultdict(list)
        for obj in workflow.detected_objects:
            stats[obj.label].append(string_with_visible_linebreaks(obj.caption))
        for label, captions in stats.items():
            count = len(captions)
            label_captions = " • ".join(sorted(captions))
            yield f"| {label} | {count} | {label_captions} |"

    def yield_md_rows_with_bbox(workflow: VisualObjectWorkflow) -> Iterator[str]:
        yield "| box_2d | label | caption |"
        yield "| :--- | :--- | :--- |"
        for obj in workflow.detected_objects:
            yield f"| {obj.box_2d} | {obj.label} | {string_with_visible_linebreaks(obj.caption)} |"

    workflow = workflow_by_image.get(source)
    if workflow is None:
        print(f'❌ No detection for source "{source.name}"')
        return
    md_rows = list(
        yield_md_rows_consolidated(workflow)
        if show_consolidated
        else yield_md_rows_with_bbox(workflow)
    )
    display_image_source_info(source)
    display_markdown(chr(10).join(md_rows))

In the museum guidebook, the dynamic labeling is precise according to the context, and the captions below each illustration are perfectly extracted:

display_detection_data(Source.museum_guidebook)

• Source Image • Source Page • Barnum's American Museum illustrated (1850) • Library of Congress, Rare Book and Special Collections Division •

box_2d	label	caption
[954, 629, 1338, 1166]	beetle	"The Horned Beetle."
[265, 984, 464, 1504]	armor	"Armor of a Man."
[737, 984, 915, 1328]	armor	"Horse Armor."
[1225, 1244, 1589, 1685]	beetle	"The Goliath Beetle."
[264, 1766, 431, 2006]	mask	"The Mask."
[937, 1769, 1260, 2087]	butterfly	"Painted Lady Butterfly."
[1325, 2170, 1581, 2468]	butterfly	"The Lady Butterfly."

In the book photo showing four paintings, this is perfect too:

display_detection_data(Source.paintings)

• Source Image • Source Page • Open book showing paintings by Vincent van Gogh • Photo by Trung Manh cong on Unsplash •

box_2d	label	caption
[378, 203, 837, 575]	painting	"Hái Ô-liu (Olive Picking), tháng 12 năm 1889, sơn dầu trên toan, 28 3/4 x 35 in. [73 x 89 cm]"
[913, 207, 1380, 563]	painting	"Hẻm núi Les Peiroulets (Les Peiroulets Ravine), tháng 10 năm 1889, sơn dầu trên toan, 28 3/4 x 36 1/4 in. [73 x 92 cm]"
[387, 596, 845, 978]	painting	"Trưa: Nghỉ ngơi (phỏng theo Millet) (Noon: Rest from Work [after Millet]), tháng 1 năm 1890, sơn dầu trên toan, 28 3/4 x 35 7/8 in. [73 x 91 cm]"
[921, 611, 1397, 982]	painting	"Hoa hạnh đào (Almond Blossom), tháng 2 năm 1890, sơn dầu trên toan, 28 3/8 x 36 1/4 in. [73 x 92 cm]"

In the Denver architecture book, the four captions are assigned to the correct illustrations, which was not an obvious task:

display_detection_data(Source.denver_illustrated)

• Source Image • Source Page • Denver illustrated (1893) • Library of Congress, Meeting of Frontiers •

box_2d	label	caption
[203, 224, 741, 839]	building	"ERNEST AND CRANMER BUILDING."
[743, 73, 1192, 758]	building	"PEOPLE'S BANK BUILDING."
[1185, 211, 1787, 865]	building	"BOSTON BUILDING."
[699, 754, 1238, 1203]	building	"COOPER BUILDING."

💡 If you have a closer look at the source image, it's hard to tell which caption belongs to which illustration at a glance. Most of us would need to think about it (and might be wrong). Asking Gemini reveals that the results are intentional and not pure luck: Deciphering vintage layouts can feel a bit like a puzzle, but there is usually a "reading-order" logic at play. In this specific case, the captions are arranged to correspond with the images in a clockwise or Z-pattern starting from the top left.

In the "Alice's Adventures in Wonderland" book page, there was a single illustration accompanying the story text. As expected, the caption is empty (i.e., no false positive):

display_detection_data(Source.alice_drawing)

• Source Image • Source Page • Open book showing an illustration and text from Alice's Adventures in Wonderland • Photo by Brett Jordan on Unsplash •

box_2d	label	caption
[111, 146, 1008, 593]	illustration	""

🔭 Generalizing object detection

We can use the same principles for other object types. We'll generally keep requesting bounding boxes to identify object positions within images. Without changing our current output structure (i.e., no code change), we can use captions and labels to extract different object metadata depending on the input type.

🧪 See how we can detect electronic components by adapting the prompt while keeping the exact same code and output structure:

ELECTRONIC_COMPONENT_DETECTION_PROMPT = """
Exhaustively detect all the individual electronic components in the image and provide the following data for each:
- `box_2d`: bounding box coordinates.
- `caption`: Verbatim alphanumeric text visible on the component (including original line breaks), or "" if no text is present.
- `label`: Specific type of component.
"""

detect_objects(
    Source.electronics,
    ELECTRONIC_COMPONENT_DETECTION_PROMPT,
    media_resolution=PartMediaResolutionLevel.MEDIA_RESOLUTION_ULTRA_HIGH,
)

• Source Image • Source Page • Circuit board with electronic components • Photo by Albert Stoynov on Unsplash •

💡 Remarks:

Large and tiny components are detected, thanks to the specific instruction "exhaustively detect…".

By using the ultra-high media resolution, we ensure more details are tokenized and the "P" component (a visual outlier) gets detected.

Here's a consolidated view of the detected components:

display_detection_data(Source.electronics, show_consolidated=True)

• Source Image • Source Page • Circuit board with electronic components • Photo by Albert Stoynov on Unsplash •

label	count	captions
integrated circuit	3	"49240↩️020S6K" • "8105↩️0:35" • "P4010↩️9NA0"
resistor	4	"" • "" • "105" • "R020"
inductor	1	"n1W"
diode	3	"K" • "L" • "P"
capacitor	6	"" • "" • "" • "" • "" • ""
transistor	1	"41"
connector	1	""

💡 Remarks:

Components are detected along with their text markings, despite the three different text orientations (upright, sideways, and upside down), the blur, and the photo noise.

We removed the degree of freedom for multi-line text by specifying the inclusion of "original line breaks" in the prompt: responses now consistently include the line breaks for the three integrated circuits (displayed with the ↩️ emoji for better visibility).

The last degree of freedom lies in the labeling. While most components have been properly labeled, it is unclear whether the "P" component is a diode, a resistor, or a fuse. Making the instructions more specific (e.g., listing the possible labels, using an enum for the label field in the Pydantic class, or providing guidelines and more details about the expected circuit boards) will make the prompt more "closed" and the results more deterministic and accurate. It's also possible to enable/update the thinking_config configuration, which will trigger a chain of thought before generating the final answer. In all the detections performed, our code used ThinkingLevel.MINIMAL, which didn't consume any thought tokens (with Gemini 3 Flash). Updating the parameter to ThinkingLevel.LOW, ThinkingLevel.MEDIUM, or ThinkingLevel.HIGH will use thought tokens and can lead to better outputs in complex cases.

This demonstrates the versatility of the approach. Without retraining a model, we switched from detecting 15th-century woodcuts and illustrations with vintage layouts to identifying modern electronics just by changing the prompt. Such detections, including caption and label metadata, could be used to auto-crop components for a parts catalog, verify assembly lines, or create interactive schematics… all without a single labeled training image.

🪄 Editing visual objects

Now that we can detect visual objects, we can envision an automation workflow to extract and reuse them. For this, we'll use Gemini 2.5 Flash Image (also known as Nano Banana 🍌) by default, a state-of-the-art image generation and editing model.

Our object editing functions will follow the same template, taking one step as input and generating an edited image for the output step. Let's define core helpers for this: 🔽

from typing import Protocol


class ObjectEditingFunction(Protocol):
    def __call__(
        self,
        image: InputImage,
        prompt: str | None = None,
        model: ImageModel | None = None,
        config: GenerateContentConfig | None = None,
        display_results: bool = True,
    ) -> None: ...


SourceTargetSteps = tuple[WorkflowStep, WorkflowStep]
registered_functions: dict[SourceTargetSteps, ObjectEditingFunction] = {}

DEFAULT_EDITING_CONFIG = GenerateContentConfig(response_modalities=["IMAGE"])
EMPTY_IMAGE = PIL.Image.new("1", (1, 1), "white")


def object_editing_function(
    default_prompt: str,
    source_step: WorkflowStep,
    target_step: WorkflowStep,
    default_model: ImageModel = ImageModel.DEFAULT,
    default_config: GenerateContentConfig = DEFAULT_EDITING_CONFIG,
) -> ObjectEditingFunction:
    def editing_function(
        image: InputImage,
        prompt: str | None = default_prompt,
        model: ImageModel | None = default_model,
        config: GenerateContentConfig | None = default_config,
        display_results: bool = True,
    ) -> None:
        workflow, source_images = get_workflow_and_step_images(image, source_step)
        if prompt is None:
            prompt = default_prompt
        prompt = prompt.strip()
        if model is None:
            model = default_model
        # Note: "config is None" is valid and will use the model endpoint default config

        target_images: list[PIL_Image] = []
        display_image_source_info(image)
        obj_count = len(source_images)
        for obj_order, source_image in enumerate(source_images, 1):
            target_image = generate_image([source_image], prompt, model, config)
            save_workflow_image(
                source_step,
                target_step,
                image,
                obj_order,
                obj_count,
                target_image,
                dict(prompt=prompt),
            )
            target_images.append(target_image if target_image else EMPTY_IMAGE)

        workflow.images_by_step[target_step] = target_images
        if display_results:
            display_sources_and_targets(workflow, source_step, target_step)

    registered_functions[(source_step, target_step)] = editing_function

    return editing_function


def get_workflow_and_step_images(
    image: InputImage,
    step: WorkflowStep,
) -> tuple[VisualObjectWorkflow, list[PIL_Image]]:
    # Objects detected?
    if image not in workflow_by_image:
        detect_objects(image, display_results=False)
    workflow = workflow_by_image.get(image)
    assert workflow is not None

    # Workflow step objects? (single level, could be extended to a dynamical graph)
    operation = (WorkflowStep.CROPPED, step)
    if step not in workflow.images_by_step and operation in registered_functions:
        source_function = registered_functions[operation]
        source_function(image, display_results=False)

    # Source images
    source_images = workflow.images_by_step.get(step)
    assert source_images is not None

    return workflow, source_images


def display_sources_and_targets(
    workflow: VisualObjectWorkflow,
    source_step: WorkflowStep,
    target_step: WorkflowStep,
) -> None:
    source_images = workflow.images_by_step[source_step]
    target_images = workflow.images_by_step[target_step]
    if not source_images:
        print("❌ No images to display")
        return

    fig = plt.figure(layout="compressed")
    if horizontal := (len(source_images) >= 2):
        rows, cols = 2, len(source_images)
    else:
        rows, cols = len(source_images), 2
    gs = fig.add_gridspec(rows, cols)

    for i, (source_image, target_image) in enumerate(
        zip(source_images, target_images, strict=True)
    ):
        for dim, image in enumerate([source_image, target_image]):
            grid_spec = gs[dim, i] if horizontal else gs[i, dim]
            ax = fig.add_subplot(grid_spec)
            ax.set_axis_off()
            ax.imshow(image)

    disable_colab_cell_scrollbar()
    plt.show()


print("✅ Object editing helpers defined")

Now, let's define a first editing step to restore the detected objects that can contain many real-life artifacts…

✨ Restoring visual objects

For this restoration step, we need to craft a prompt that is generic enough (to cover most use cases) but also specific enough (to take into account restoration needs).

An image editing prompt is based on natural language, typically using imperative or declarative instructions. With an imperative prompt, you describe the actions to perform on the input, while with a declarative prompt, you describe the expected output. Both are possible and will provide equivalent results. Your choice is really a matter of preference, as long as the prompt makes sense.

Our test suite is mostly composed of book photos, which can contain various photographic and paper artifacts. The Nano Banana models understand these subtleties and can edit images accordingly, which simplifies the prompt.

Here is a possible restoration function using an imperative prompt:

RESTORATION_PROMPT = """
- Isolate and straighten the visual on a pure white background, excluding any surrounding text.
- Clean up all physical artifacts and noise while preserving every original detail.
- Center the result and scale it to fit the canvas with minimal, symmetrical margins, ensuring no distortion or cropping.
"""

# Default config with low randomness for more deterministic restoration outputs
RESTORATION_CONFIG = GenerateContentConfig(
    temperature=0.0,
    top_p=0.0,
    seed=42,
    response_modalities=["IMAGE"],
)

restore_objects = object_editing_function(
    RESTORATION_PROMPT,
    WorkflowStep.CROPPED,
    WorkflowStep.RESTORED,
    default_config=RESTORATION_CONFIG,
)

print("✅ Restoration function defined")

🧪 Let's try to restore the illustration from the 1485 incunable:

restore_objects(Source.incunable)

• Source Image • Source Page • Vergaderinge der historien van Troy (1485) • Library of Congress, Rare Book and Special Collections Division •

💡 We now have a nice restoration of the hand-colored woodcut illustration. Note that our prompt is generic ("clean up all physical artifacts") and could be made more specific to remove more or fewer artifacts. In this example, there are remaining artifacts, such as the paper discoloration in the sword or the bleeding ink in the armor. We'll see if we can fix these in the colorization step.

🧪 What about the illustrations from the museum guidebook?

restore_objects(Source.museum_guidebook)

• Source Image • Source Page • Barnum's American Museum illustrated (1850) • Library of Congress, Rare Book and Special Collections Division •

💡 All good!

🧪 What about the slightly warped visuals?

restore_objects(Source.paintings)

• Source Image • Source Page • Open book showing paintings by Vincent van Gogh • Photo by Trung Manh cong on Unsplash •

💡 Remarks:

Notice how, on the last painting, the orange bookmark is properly removed and the hidden part inpainted to complete the painting.

We requested to "fit the canvas with minimal, symmetrical margins, without distortion or cropping". Depending on the aspect ratio and type of the visual, this degree of freedom can result in different white margins.

This example shows famous paintings by Vincent Van Gogh. Nano Banana does not fetch any reference images and only uses the provided input. If these were photos of private paintings, they would be restored in the same way.

In the Denver architecture book, the illustrations can be tilted, which our generic prompt does not fully take into account. When several geometric transformations are involved, it can be challenging to craft an imperative prompt that details all the operations to perform. Instead, a descriptive prompt can be more straightforward by directly describing the expected output.

🧪 Here's an example of a descriptive prompt focusing on the restoration of tilted visuals:

tilted_visual_prompt = """
An upright, high-fidelity rendition of the visual isolated against a pure white background, filling the canvas with minimal uniform margins. The output is clean, sharp, and free of physical artifacts.
"""

restore_objects(Source.denver_illustrated, tilted_visual_prompt)

• Source Image • Source Page • Denver illustrated (1893) • Library of Congress, Meeting of Frontiers •

💡 Remarks:

To get these results, the prompt focuses on requesting an "upright" visual "filling the canvas", which proves more straightforward to write than trying to account for all possible geometric corrections.

The native visual understanding automatically identifies the content type (photo, illustration, etc.) and the different artifacts (photographic, paper, printing, scanning…), allowing for precise restorations out of the box.

Notice how the consistency is preserved: the last visual is restored as an illustration, while the first visuals maintain their photographic style.

The results, with this rather generic prompt, are impressive. It is, of course, possible to be more specific and request particular lighting, styles, colors…

In this last test, the input visual has distortions not only from the page curvature but also from the photo perspective.

🧪 Here's an example of a descriptive prompt focusing on restoring warped illustrations:

warped_visual_prompt = """
An edge-to-edge digital extraction of the illustration from the provided book photo, excluding any peripheral text. All page curvature and perspective distortions are corrected, resulting in an image framed in a perfect rectangle, on a pure white canvas with minimal margins.
"""

restore_objects(Source.alice_drawing, warped_visual_prompt)

• Source Image • Source Page • Open book showing an illustration and text from Alice's Adventures in Wonderland • Photo by Brett Jordan on Unsplash •

💡 It is really impressive that such a restoration can be performed in a single step. Note that this prompt is not stable and can generate less optimal results (it would benefit from being more precise). If you have complex transformations, test descriptive prompts iteratively, and you might be pleasantly surprised. In the worst case, it's also possible to process the transformations in successive, easier steps.

Now, let's add a colorization step…

🎨 Colorization

Our restoration step respected the original styles of the input images. Recent image editing models excel at transforming image styles, starting with colors. This can generally be performed directly with a simple, precise instruction.

Here is a possible colorization function using an imperative prompt:

COLORIZATION_PROMPT = """
Colorize this image in a modern book illustration style, maintaining all original details without any additions.
"""

colorize = object_editing_function(
    COLORIZATION_PROMPT,
    WorkflowStep.RESTORED,
    WorkflowStep.COLORIZED,
)

print("✅ Colorization function defined")

🧪 Let's modernize our 1485 illustration:

colorize(Source.incunable)

• Source Image • Source Page • Vergaderinge der historien van Troy (1485) • Library of Congress, Rare Book and Special Collections Division •

💡 All details are preserved, as requested in the prompt. Notice how the colorization can naturally fix some remaining artifacts (e.g., the paper discoloration in the sword or the bleeding ink in the armor).

🧪 Let's colorize our museum guidebook illustrations:

colorize(Source.museum_guidebook)

• Source Image • Source Page • Barnum's American Museum illustrated (1850) • Library of Congress, Rare Book and Special Collections Division •

💡 Our prompt is very open as it only specifies "modern book illustration style". This can generate very creative colorizations, but they all seem to make perfect sense.

🧪 What about our Denver buildings?

colorize(Source.denver_illustrated)

• Source Image • Source Page • Denver illustrated (1893) • Library of Congress, Meeting of Frontiers •

💡 As requested, they all look like modern illustrations, including the first visuals (originating from noisy photos).

It's possible to go further by not only "colorizing" but also "transforming" the image into a significantly different one.

🧪 Let's make our "Alice's Adventures in Wonderland" drawing into a watercolor painting:

watercolor_prompt = """
Transform this visual into a warm, watercolor painting.
"""

colorize(Source.alice_drawing, watercolor_prompt)

• Source Image • Source Page • Open book showing an illustration and text from Alice's Adventures in Wonderland • Photo by Brett Jordan on Unsplash •

🧪 What about making it a traditional painting?

painting_prompt = """
Transform this visual into a traditional painting.
"""

colorize(Source.alice_drawing, painting_prompt)

• Source Image • Source Page • Open book showing an illustration and text from Alice's Adventures in Wonderland • Photo by Brett Jordan on Unsplash •

We can also change image compositions. Depending on the context, some compositions are more or less implied by default. For example, illustrations often have margins, while photos generally have edge-to-edge (full-bleed in the printing world) compositions. When possible, it's interesting to refer to a type of visual (which intrinsically brings a lot of semantics to the context) and adjust the instructions accordingly.

🧪 Let's see how we can detect engravings in this 1847 book, restore them, and transform them into modern digital graphics:

detect_objects(Source.engravings)

• Source Image • Source Page • Harper's illustrated catalogue (1847) • Library of Congress •

restore_objects(Source.engravings)

• Source Image • Source Page • Harper's illustrated catalogue (1847) • Library of Congress •

visual_to_digital_graphic_prompt = """
Transform this visual into a full-color, flat digital graphic, extending the content for a full-bleed effect.
"""

colorize(Source.engravings, visual_to_digital_graphic_prompt)

• Source Image • Source Page • Harper's illustrated catalogue (1847) • Library of Congress •

🧪 We can also transform the same engravings into photos with a very simple prompt:

visual_to_photo_prompt = """
Transform this visual into a high-end, modern camera photograph.
"""

colorize(Source.engravings, visual_to_photo_prompt)

• Source Image • Source Page • Harper's illustrated catalogue (1847) • Library of Congress •

💡 As photos are generally full-bleed, the prompt does not need to specify a composition.

It's really up to our imagination, as Nano Banana seems to grasp every aspect of the visual semantics.

Let's add a final step to see how far we can go, reimagining images as cinematic movie stills…

🎞️ Cinematization

We've used rather "closed" prompts so far, crafting specific instructions and constraints to control the outputs. It's possible to go even further with "open" prompts and generate images in full creative mode. Notably, it can be interesting to refer to photographic or cinematographic terminology as it encompasses many visual techniques.

Here is a possible generic cinematization function to reimagine images as movie stills:

CINEMATIZATION_PROMPT = """
Reimagine this image as a joyful, modern live-action cinematic movie still featuring professional lighting and composition.
"""

cinematize = object_editing_function(
    CINEMATIZATION_PROMPT,
    WorkflowStep.RESTORED,
    WorkflowStep.CINEMATIZED,
)

🧪 Let's cinematize the "Alice's Adventures in Wonderland" drawing:

cinematize(Source.alice_drawing)

• Source Image • Source Page • Open book showing an illustration and text from Alice's Adventures in Wonderland • Photo by Brett Jordan on Unsplash •

💡 This looks like a high-budget movie still. There are lots of degrees of freedom in the prompt, but you're likely to get foreground figures in sharp focus, a gradual background blur, "golden hour" lighting (a magical ingredient for many cinematographers), and detailed textures. Such compositions really evoke different atmospheres compared to the photos generated in the previous test.

🧪 Let's test the workflow on a page from the Wonderful Wizard of Oz containing three drawings:

detect_objects(Source.wizard_of_oz_drawings)

• Source Image • Source Page • The wonderful Wizard of Oz (1899) • Library of Congress, Rare Book and Special Collections Division •

restore_objects(Source.wizard_of_oz_drawings)

• Source Image • Source Page • The wonderful Wizard of Oz (1899) • Library of Congress, Rare Book and Special Collections Division •

cinematize(Source.wizard_of_oz_drawings)

• Source Image • Source Page • The wonderful Wizard of Oz (1899) • Library of Congress, Rare Book and Special Collections Division •

💡 The cast for a new movie is ready ;)

Cinematic images have various use cases:

These cinematized stills can be perfect "reference images" for video generation models like Veo. See Generate Veo videos from reference images.
As they are photorealistic representations, they can also be a source for generating 2D or 3D visuals, in any style, with realistic figures, perfect proportions, advanced lighting, enhanced compositions…
You can use them in many professional contexts or for high-end products: presentations, magazines, posters, storyboards, brainstorming sessions…

🏁 Conclusion

Gemini's native spatial understanding enables the detection of specific visual objects based on a single prompt in natural language.
We tested the detection of illustrations in book photos, which traditional machine learning (ML) models usually miss, as they are typically trained to detect people, animals, vehicles, food, and a finite set of physical object classes.
We tested the detection of straight, tilted, and even significantly warped illustrations, and they were always precisely identified.
The core implementation was straightforward, requiring minimal code using the Python SDK and customized prompts. By comparison, fine-tuning a traditional object detection model is time-consuming: it involves assembling an image dataset, labeling objects, and managing training jobs.
This solution is very flexible: we could switch from detecting illustrations to electronic components, by adapting the prompt, while keeping the code unchanged.
Using structured outputs (with a JSON schema or Pydantic classes, and the Python SDK) makes the code both easy to implement and ready to deploy to production.
Then, Nano Banana allows editing these visual objects in virtually any way imaginable.
We tested a workflow with restoration, colorization, and even cinematization steps, using imperative and descriptive prompts.
The possibilities seem really endless, and the principles in this exploration can be reused in different contexts.

➕ More!

Try it yourself: Use the companion notebook to reproduce the results in this article.
Get inspired: Check out the Nano Banana recipes notebook for more practical examples.
Follow me: Connect with me (@picardparis) on LinkedIn or Twitter-X for more cloud, applied AI, and Python explorations…

Thanks for reading. Let me know if you create something cool!

Counting Gemini text tokens locally

Laurent Picard — Thu, 04 Jul 2024 12:43:14 +0000

The Vertex AI SDK for Python now offers local tokenization. This feature allows you to calculate the number of tokens in your text input before sending requests to Gemini. Let’s see how this feature works…

🧩 What is a token?

Large language models (LLMs) use a fundamental unit: the token. In other words, they process input tokens and generate output tokens.

A text token can represent characters, words, or phrases. On average, one token represents approximately four characters in English text. When you send a query to Gemini, your text input is transformed into tokens. This step is called tokenization. Gemini generates output tokens that are then converted back into text using the reverse operation.

📦️ Setting up the Vertex AI SDK

To use Gemini tokenizers, you need both the latest google-cloud-aiplatform package with the tokenization extra:

pip install --upgrade google-cloud-aiplatform[tokenization]

Ensure you have version 1.57.0 or later installed:

pip show google-cloud-aiplatform

Name: google-cloud-aiplatform
Version: 1.57.0
…

🧮 Counting tokens locally

Create a tokenizer for the Gemini model you’re using and call the count_tokens() method on your text input:

from vertexai.preview import tokenization

model_name = "gemini-1.5-flash-001"
tokenizer = tokenization.get_tokenizer_for_model(model_name)

contents = "Hello World!"
result = tokenizer.count_tokens(contents)

print(f"{result.total_tokens = :,}")

result.total_tokens = 3

Now, let’s try with a larger document:

import http.client
import typing
import urllib.request

def download_text_from_url(url: str) -> str:
    with urllib.request.urlopen(url) as response:
        response = typing.cast(http.client.HTTPResponse, response)
        encoding = response.headers.get_content_charset() or "utf-8"
        return response.read().decode(encoding)

url = "https://storage.googleapis.com/dataflow-samples/shakespeare/hamlet.txt"
contents = download_text_from_url(url)
result = tokenizer.count_tokens(contents)

print(f"{result.total_tokens = :,}")

result.total_tokens = 53,824

🚀 Perfect! The number of tokens is computed in a fraction of second.

🖼️ Multimodal inputs

In the tested version (1.57.0), local token counting is only supported for text inputs.

For multimodal inputs (image, video, audio, documents), check out the documentation for details on how different medias account for different token counts:

In all cases, you can send a request using the Vertex AI API as usual:

import vertexai
from vertexai.generative_models import GenerativeModel

# project_id = "PROJECT_ID"
# location = "us-central1"
# vertexai.init(project=project_id, location=location)

model = GenerativeModel(model_name)

response = model.count_tokens(contents)

print(f"{response.total_tokens = :,}")
print(f"{response.total_billable_characters = :,}")

response.total_tokens = 53,824
response.total_billable_characters = 144,443

🔍 Under the hood

Text tokenizers use a fixed data file, also called the LLM vocabulary. This vocabulary determines how to encode a text string into a sequence of tokens and, reversely, how to decode a sequence of tokens into a text string.

Here is what happens under the hood:

When you request a tokenizer for your model for the first time, the LLM vocabulary is downloaded and locally cached. On subsequent calls, the cached data is used.
When you call tokenizer.count_tokens(contents), the text is tokenized (becomes a sequence of tokens) and the corresponding number of tokens is returned.

Key points to remember:

💡 For Gemini 1.0 and 1.5, the LLM vocabulary size is only about 4 MB.
💡 Different Gemini models can use the same tokenizer but this is not guaranteed. Make sure to get the tokenizer for your model.

⚡ Benefits of knowing token counts

Knowing how many tokens your text represents enables the following:

✅ Determine whether your text input will be processed by a model. For example, Gemini 1.0 Pro supports up to 32,760 input tokens, while Gemini 1.5 Pro currently accepts up to 2,097,152 input tokens.
✅ Consolidate or estimate costs on a per-token basis (Gemini pricing is per 1k characters).
✅ Estimate how much time will be needed to process your text input before you start receiving a response (processing times are roughly proportional to the numbers of processed tokens).

You now have a new local tool to manage your inputs before sending requests to Gemini!

🖖 Follow me on Twitter/X or LinkedIn for more cloud explorations

Making AI more Open and Accessible to Cloud Developers with Gemma on Vertex AI

Laurent Picard — Mon, 26 Feb 2024 13:39:01 +0000

Gemma just opened ;)

Gemma is a family of open, lightweight, and easy-to-use models developed by Google Deepmind. The Gemma models are built from the same research and technology used to create Gemini.

This means that we (ML developers & practitioners) now have additional versatile large language models (LLMs) in our toolbox!

If you’d rather read code, you can jump straight to this Python notebook:
→ Finetune Gemma using KerasNLP and deploy to Vertex AI

Availability

Gemma is available today in Google Cloud and the machine learning (ML) ecosystem. You’ll probably find an ML platform you’re already familiar with:

Gemma joins 130+ models in Vertex AI Model Garden
Gemma joins the Kaggle Models
Gemma joins the Hugging Face Models
Gemma joins Google AI for Developers

Here are the Gemma Terms of Use.

Gemma models

The Gemma family launches in two sizes, Gemma 2B and 7B , targeting two typical platform types:

Model	Parameters	Platforms
Gemma 2B	2.5 billion	Mobile devices and laptops
Gemma 7B	8.5 billion	Desktop computers and small servers

These models are text-to-text English LLMs:

Input: a text string, such as a question, a prompt, or a document.
Output: generated English text, such as an answer or a summary.

They were trained on a diverse and massive dataset of 8 trillion tokens, including web documents, source code, and mathematical text.

Each size is available in untuned and tuned versions:

Pretrained (untuned): Models were trained on core training data, not using any specific tasks or instructions.
Instruction-tuned: Models were additionally trained on human language interactions.

That gives us four variants. As an example, here are the corresponding model IDs when using Keras:

gemma_2b_en
gemma_instruct_2b_en
gemma_7b_en
gemma_instruct_7b_en

Use cases

Google now offers two families of LLMs: Gemini and Gemma. Gemma is a complement to Gemini, is based on technologies developed for Gemini, and addresses different use cases.

Examples of Gemini benefits:

Enterprise applications
Multilingual tasks
Optimal qualitative results and complex tasks
State-of-the-art multimodality (text, image, video)
Groundbreaking features (e.g. Gemini 1.5 Pro 1M token context window)

Examples of Gemma benefits:

Learning, research, or prototyping based on lightweight models
Focused tasks such as text generation, summarization, and Q&A
Framework or cross-device interoperability
Offline or real-time text-only processings

The Gemma model variants are useful in the following use cases:

Pretrained (untuned): Consider these models as a lightweight foundation and perform a custom finetuning to optimize them to your own needs.
Instruction-tuned: You can use these models for conversational applications, such as a chatbot, or to customize them even further.

Interoperability

As Gemma models are open, they are virtually interoperable with all ML platforms and frameworks.

For the launch, Gemma is supported by the following ML ecosystem players:

Google Cloud
Kaggle
Keras, which means that Gemma runs on JAX, PyTorch, and TensorFlow
Hugging Face
Nvidia

And, of course, Gemma can run on GPUs and TPUs.

Requirements

Gemma models are lightweight but, in the LLM world, this still means gigabytes.

In my tests, when running inferences in half precision (bfloat16), here are the minimum storage and GPU memory that were required:

Model	Total parameters	Assets size	Min. GPU RAM to run
Gemma 2B	2,506,172,416	4.67 GB	8.3 GB
Gemma 7B	8,537,680,896	15.90 GB	20.7 GB

To experiment with Gemma and Vertex AI, I used Colab Enterprise with a g2-standard-8 runtime, which comes with an NVIDIA L4 GPU and 24 GB of GPU RAM. This is a cost-effective configuration to save time and avoid running out-of-memory when prototyping in a notebook.

Finetuning Gemma

Depending on your preferred frameworks, you’ll find different ways to customize Gemma. Keras is one of them and provides everything you need.

KerasNLP lets you load a Gemma model in a single line:

import keras
import keras_nlp

gemma_lm = keras_nlp.models.GemmaCausalLM.from_preset("gemma_instruct_2b_en")

Then, you can directly generate text:

inputs = [
    "What's the most famous painting by Monet?",
    "Who engineered the Statue of Liberty?",
    'Who were "The Lumières"?',
]

for input in inputs:
    response = gemma_lm.generate(input, max_length=25)
    output = response[len(input) :]
    print(f"{input!r}\n{output!r}\n")

With the instruction-tuned version, Gemma gives you expected answers, as would a good LLM-based chatbot:

# With "gemma_instruct_2b_en"

"What's the most famous painting by Monet?"
'\n\nThe most famous painting by Monet is "Water Lilies," which'

'Who engineered the Statue of Liberty?'
'\n\nThe Statue of Liberty was built by French sculptor Frédéric Auguste Bartholdi between 1'

'Who were "The Lumières"?'
'\n\nThe Lumières were a group of French scientists and engineers who played a significant role'

Now, try the untuned version:

gemma_lm = keras_nlp.models.GemmaCausalLM.from_preset("gemma_2b_en")

You’ll get different results, which is expected as this version was not trained on any specific task and is designed to be finetuned to your own needs:

# With "gemma_2b_en"

"What's the most famous painting by Monet?"
"\n\nWhat's the most famous painting by Van Gogh?\n\nWhat"

'Who engineered the Statue of Liberty?'
'\n\nA. George Washington\nB. Napoleon Bonaparte\nC. Robert Fulton\nD'

'Who were "The Lumières"?'
' What did they invent?\n\nIn the following sentence, underline the correct modifier from the'

If you’d like, for example, to build a Q&A application, prompt engineering may fix some of these issues but, to be grounded on facts and return consistent results, untuned models need to be finetuned with training data.

The Gemma models have billions of trainable parameters. The next step consists of using the Low Rank Adaptation (LoRA) to greatly reduce the number of trainable parameters:

# Number of trainable parameters before enabling LoRA: 2.5B

# Enable LoRA for the model and set the LoRA rank to 4
gemma_lm.backbone.enable_lora(rank=4)

# Number of trainable parameters after enabling LoRA: 1.4M (1,800x less)

A training can now be performed with reasonable time and GPU memory requirements.

For prototyping on a small training dataset, you can launch a local finetuning:

training_data: list[str] = [...]

# Reduce the input sequence length to limit memory usage
gemma_lm.preprocessor.sequence_length = 128

# Use AdamW (a common optimizer for transformer models)
optimizer = keras.optimizers.AdamW(
    learning_rate=5e-5,
    weight_decay=0.01,
)

# Exclude layernorm and bias terms from decay
optimizer.exclude_from_weight_decay(var_names=["bias", "scale"])

gemma_lm.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=optimizer,
    weighted_metrics=[keras.metrics.SparseCategoricalAccuracy()],
)
gemma_lm.fit(training_data, epochs=1, batch_size=1)

After a couple of minutes of training (on 1,000 examples) and using a structured prompt, the finetuned model can now answer questions based on your training data:

# With "gemma_2b_en" before finetuning
'Who were "The Lumières"?'
' What did they invent?\n\nIn the following sentence, underline the correct modifier from the'

# With "gemma_2b_en" after finetuning
"""
Instruction:
Who were "The Lumières"?

Response:
"""
"The Lumières were the inventors of the first motion picture camera. They were"

The prototyping stage is over. Before you proceed to finetuning models in production with large datasets, check out new specific tools to build a responsible Generative AI application.

Responsible AI

With great power comes great responsibility, right?

The Responsible Generative AI Toolkit will help you build safer AI applications with Gemma. You’ll find expert guidance on safety strategies for the different aspects of your solution.

Also, do not miss the Learning Interpretability Tool (LIT) for visualizing and understanding the behavior of your Gemma models. Here’s a video tour showing the tool in action:

Deploying Gemma

We’re in MLOps territory here and you’ll find different LLM-optimized serving frameworks running on GPUs or TPUs.

Here are popular serving frameworks:

vLLM (GPU)
TGI: Text Generation Interface (GPU)
Saxml (TPU or GPU)
TensorRT-LLM (NVIDIA Triton GPU)

These frameworks come with prebuilt container images that you can easily deploy to Vertex AI.

Here is a simple notebook to get you started:

Finetune Gemma using KerasNLP and deploy to Vertex AI

For production finetuning, you can use Vertex AI custom training jobs. Here are detailed notebooks:

Gemma Finetuning (served by vLLM or HexLLM)
Gemma Finetuning (served by TGI)

Those notebooks focus on deployment and serving:

Gemma Deployment (served by vLLM or HexLLM)
Gemma Deployment to GKE using TGI on GPU

For more details and updates, check out the documentation:

All the best to Gemini and Gemma!

After years of consolidation in machine learning hardware and software, it’s exciting to see the pace at which the overall ML landscape now evolves and in particular with LLMs. LLMs are technologies still in their infancy, so we can expect to see more breakthroughs in the near future.

I very much look forward to seeing what the ML community is going to build with Gemma!

All the best to the Gemini and Gemma families!

Follow me on Twitter or LinkedIn for more cloud explorations…

Moderating text with the Natural Language API

Laurent Picard — Tue, 20 Jun 2023 15:46:47 +0000

2023–09–12: text moderation got Generally Available (GA) over the summer + added link to Sep. blog post

Credit: Photo by Da Nina on Unsplash

The Natural Language API lets you extract information from unstructured text using Google machine learning and provides a solution to the following problems:

Sentiment analysis
Entity analysis
Entity sentiment analysis
Syntax analysis
Content classification
Text moderation

🔍 Moderation categories

Text moderation lets you detect sensitive or harmful content. The first moderation category that comes to mind is "toxicity", but there can be many more topics of interest. A PaLM 2-based model powers the predictions and scores 16 categories:


Toxic	Insult	Public Safety	War & Conflict
Derogatory	Profanity	Health	Finance
Violent	Death, Harm & Tragedy	Religion & Belief	Politics
Sexual	Firearms & Weapons	Illicit Drugs	Legal

⚡ Moderating text

Like always, you can call the API through the REST/RPC interfaces or with idiomatic client libraries.

Here is an example using the Python client library (google-cloud-language) and the moderate_text method:

from google.cloud import language

def moderate_text(text: str) -> language.ModerateTextResponse:
    client = language.LanguageServiceClient()
    document = language.Document(
        content=text,
        type_=language.Document.Type.PLAIN_TEXT,
    )
    return client.moderate_text(document=document)

text = (
    "I have to read Ulysses by James Joyce.\n"
    "I'm a little over halfway through and I hate it.\n"
    "What a pile of garbage!"
)
response = moderate_text(text)

🚀 It's fast! The model latency is very low, allowing real-time analyses.

The response contains confidence scores for each moderation category. Let's sort them out:

import pandas as pd

def confidence(category: language.ClassificationCategory) -> float:
    return category.confidence

columns = ["category", "confidence"]
categories = sorted(
    response.moderation_categories,
    key=confidence,
    reverse=True,
)
data = ((category.name, category.confidence) for category in categories)
df = pd.DataFrame(columns=columns, data=data)

print(f"Text analyzed:\n{text}\n")
print(f"Moderation categories:\n{df}")

You may typically ignore scores below 50% and calibrate your solution by defining upper limits (or buckets) for the confidence scores. In this example, depending on your thresholds, you may flag the text as disrespectful (toxic) and insulting:

Text analyzed:
I have to read Ulysses by James Joyce.
I'm a little over halfway through and I hate it.
What a pile of garbage!

Moderation categories:
                 category  confidence
0                   Toxic    0.680873
1                  Insult    0.609475
2               Profanity    0.482516
3                 Violent    0.333333
4                Politics    0.237705
5   Death, Harm & Tragedy    0.189759
6                 Finance    0.176955
7       Religion & Belief    0.151079
8                   Legal    0.100946
9                  Health    0.096305
10          Illicit Drugs    0.083333
11     Firearms & Weapons    0.076923
12             Derogatory    0.073953
13         War & Conflict    0.052632
14          Public Safety    0.051813
15                 Sexual    0.028222

🖖 More

Run this Colab notebook: Using the Natural Language API
See the supported languages
Read more about text moderation
See the latest blog post on Improving Trust in AI and Online Communities
Follow me on Twitter or LinkedIn for more cloud explorations

Using Google Cloud from Colab

Laurent Picard — Fri, 16 Jun 2023 08:49:26 +0000

Google Colab is an amazing tool for Pythonistas. It can be used for a variety of tasks, from quickly experimenting with Python code to sharing extensive data processing notebooks with the world. Colab runs on Google Cloud, which gives you a boost when accessing cloud services because your code is running inside Google’s high performance network, and also offers a simple way to use Google Cloud services, letting you run powerful cloud workloads from your browser.

📦️ Install dependencies

Colab comes with hundreds of preinstalled packages, including pandas, numpy, matplotlib, Flask, Pillow, tensorflow, and pytorch. You can install additional required dependencies as needed.

For example, to analyze text with the power of machine learning using the Natural Language API, install the google-cloud-language client library:

!pip install google-cloud-language>=2.9.1

🔑 Authenticate

To authenticate with your Google Cloud account within the Colab notebook, use the authenticate_user method. A new parameter lets you specify your project ID:

from google.colab import auth

PROJECT_ID = ""  # @param {type:"string"}

auth.authenticate_user(project_id=PROJECT_ID)

After this step:

👍 You are authenticated for gcloud CLI commands.
👍 You are also authenticated when using Google Cloud Python client libraries. Note that you generally won’t need to create a service account.
👍 Your default project is set.
👍 Your notebook can also access your Google Drive, letting you ingest or generate your own files.

🔓 Enable APIs

Ensure required APIs are enabled. In our example, that’s the service language.googleapis.com:

!gcloud services enable language.googleapis.com

🤯 Use Google Cloud services

That’s it! You can now directly use the service by calling its Python client library:

from google.cloud import language

def analyze_text_sentiment(text: str) -> language.AnalyzeSentimentResponse:
    client = language.LanguageServiceClient()
    document = language.Document(
        content=text,
        type_=language.Document.Type.PLAIN_TEXT,
    )
    return client.analyze_sentiment(document=document)

# Input
text = "Python is a very readable language, ..."  # @param {type:"string"}

# Send a request to the API
response = analyze_text_sentiment(text)

# Use the results
print(response)

📊 Benefit from notebook advanced features

Colab is hosting a Jupyter notebook. I personally depict Colab as the “Google Drive of notebooks”. As notebooks have additional superpowers, this lets you nicely process and visualize your data, such as tables, images, or charts.

In our example, the function show_text_sentiment gathers results in a pandas DataFrame, which renders as a table:

✨ Check it out

In these examples, click the “Open in Colab” link:

Using Google Cloud from Colab (short)
Using the Natural Language API with Python (detailed)

💡 You can directly create a new notebook by opening the colab.new URL.

👋 Have fun!

Read the Google Colab blog to learn more about Colab.
Follow me (Twitter / LinkedIn) for more cloud explorations ;)

From pixels to information with Document AI

Laurent Picard — Wed, 15 Mar 2023 17:18:46 +0000

We’re seeing successively difficult problems getting solved thanks to machine learning (ML) models. For example, Natural Language AI and Vision AI extract insights from text and images, with human-like results. They solve problems central to the way we communicate:

✅ Natural language processing (NLP)
✅ Optical character recognition (OCR)
✅ Handwriting recognition (HWR)

What's next? Well, it's already here, with Document AI, and keeps growing:

✅ Document understanding (DU)

Document AI builds on these foundations to let you extract information from documents, in many forms:

Text
Structure
Semantic entities
Normalized values
Enrichments
Qualitative signals

In this article, you’ll see the following:

An overview of Document AI
Practical and visual examples
A document processing demo (source code included)

Processing documents

Processor types

There are as many document types as you can imagine so, to meet all needs, document processors are at the heart of Document AI. They can be of the following types:

General processors handle common tasks such as detecting document tables or parsing forms.
Specialized processors analyze specific documents such as invoices, receipts, pay slips, bank statements, identity documents, official forms, etc.
Custom processors meet other specific needs, letting you automatically train private ML models based on your own documents, and perform tasks such as custom document classification or custom entity detection.

Processor locations

When you create a processor, you specify its location. This helps control where the documents will be processed. Here are the current multi-region locations:

Location	API `location`	API endpoint
European Union	`eu`	`eu-documentai.googleapis.com`
United States	`us`	`us-documentai.googleapis.com`

In addition, some processors are available in single-region locations; this lets you address local regulation requirements. If you regularly process documents in real-time or large batches of documents, this can also help get responses with even lower latencies. As an example, here are the current locations for the "Document OCR" general processor:

`OCR_PROCESSOR` location	API `location`
United States	`us`
European Union	`eu`
India (Mumbai)	`asia-south1`
Singapore (Jurong West)	`asia-southeast1`
Australia (Sydney)	`australia-southeast1`
England (London)	`europe-west2`
Germany (Frankfurt)	`europe-west3`
Québec (Montréal)	`northamerica-northeast1`

Note: API endpoints are named according to the convention {location}-documentai.googleapis.com.

For more information, check out Regional and multi-regional support.

Processor versions

Processors can evolve over time, to offer more precise results, new features, or fixes. For example, here are the current versions available for the "Expense Parser" specialized processor:

`name`	`display_name`	`create_time`
pretrained-expense-v1.1-2021-04-09	Stable	2021-04-09
pretrained-expense-v1.2-2022-02-18	Stable	2022-02-18
pretrained-expense-v1.3-2022-07-15	Stable	2022-07-15
pretrained-expense-v1.4-2022-11-18	Release Candidate	2022-11-18

You may typically do the following:

Use Stable version 1.3 in production.
Use Release Candidate version 1.4 to benefit from new features or test a future version.

To stay updated, follow the Release notes.

Interfaces

Document AI is available to developers and practitioners through the usual interfaces:

REST API (universal JSON-based interface)
RPC API (low-latency gRPC interface, used by all Google services)
Client Libraries (gRPC wrappers, to develop in your preferred programming language)
Cloud Console (web admin user interface)

Requests

There are two ways you can process documents:

Synchronously with online requests, to analyze a single document and directly use the results
Asynchronously with batch requests, to launch a batch processing operation on multiple or larger documents

Interface	Synchronous method	Asynchronous method
REST	`process`	`batchProcess`
RPC	`ProcessDocument`	`BatchProcessDocuments`
Client library: Python	`process_document`	`batch_process_documents`
Client library: Node.js	`processDocument`	`batchProcessDocuments`
Client library: Java,…	…	…

Extracting information from pixels

Let's start by analyzing this simple screenshot with the "Document OCR" general processor, and check how pixels become structured data.

Input image

Document AI response

The response contains a Document instance.
The entire text of the input, detected in natural reading order, is serialized under Document.text.
The structured data is returned on a per page basis (a single page here).
The "Document OCR" processor returns a structural skeleton common to all processors.

{ // document
  "mime_type": "image/png",
  "text": "Willkommen!\nBienvenue !\nWelcome!\nQuestion: What is the answer to the ultimate question of life?\nAnswer: 42\n",
  "pages": [
    {
      "page_number": 1,
      "dimension": {/*…*/},
      "layout": {/*…*/},
      "detected_languages": [/*…*/],
      "blocks": [/*…*/],
      "paragraphs": [/*…*/],
      "lines": [/*…*/],
      "tokens": [/*…*/],
      "image": {/*…*/}
    }
  ]
}

The examples in this article show snake-case field names (like mime_type), using the convention used by gRPC and programming languages like Python. For camel-case environments (like REST/JSON), there is a direct mapping between the two conventions:

mime_type ↔ mimeType
page_number ↔ pageNumber
…

Text levels

For each page, four levels of text detection are returned:

blocks
paragraphs
lines
tokens

In these lists, each item exposes a layout including the item's position relative to the page in bounding_poly.normalized_vertices.

This lets us, for example, highlight the 22 detected tokens:

Here is the last token:

{ // document.pages[0].tokens[21]
  "layout": {
    "text_anchor": {
      // "42\n" ← document.text[104:107]
      "text_segments": [{ "start_index": "104", "end_index": "107" }]
    },
    "confidence": 0.99, // Confident at 99% that the answer is "42" ;)
    "bounding_poly": {
      // The 4 bounding box vertices, relative to the page
      // Page (top, left)..(bottom, right): (0.0, 0.0)..(1.0, 1.0)
      "normalized_vertices": [ { "x": 0.4028, "y": 0.5352 },
                               { "x": 0.4327, "y": 0.5352 },
                               { "x": 0.4327, "y": 0.6056 },
                               { "x": 0.4028, "y": 0.6056 } ]
    } //,…
  } //,…
}

Note: Float values are presented truncated for the sake of readability.

Language support

Documents are often written using one single language, but sometimes use multiple languages. You can retrieve the detected languages at different text levels.

In our previous example, two blocks are detected (pages[0].blocks[]). Let's highlight them:

The left block is a mix of German, French, and English, while the right block is English only. Here is how the three languages are reported at the page level:

{ // document.pages[0]
  "detected_languages": [
    { "language_code": "en", "confidence": 0.77 },
    { "language_code": "de", "confidence": 0.12 },
    { "language_code": "fr", "confidence": 0.11 }
  ]
}

Note: At this level, the language confidence ratios roughly correspond to the proportion of text detected in each language.

Now, let's highlight the five detected lines (pages[0].lines[]):

Each language is also reported at the line level:

{ // document.pages[0]
  "lines": [
    // "Willkommen!"
    { "detected_languages": [{ "language_code": "de", "confidence": 1.0 }] },
    // "Bienvenue !"
    { "detected_languages": [{ "language_code": "fr", "confidence": 1.0 }] },
    // "Welcome!"
    { "detected_languages": [{ "language_code": "en", "confidence": 1.0 }] },
    // "Question: What is the answer to the ultimate question of life?"
    { "detected_languages": [{ "language_code": "en", "confidence": 1.0 }] },
    // "Answer: 42"
    { "detected_languages": [{ "language_code": "en", "confidence": 1.0 }] }
  ]
}

If needed, you can get language info at the token level too. "Question" is the same word in French and in English, and is adequately returned as an English token in this context:

{ // document.pages[0].tokens[6]
  "layout": {
    "text_anchor": {
      // "Question" ← document.text[33:41]
      "text_segments": [{ "start_index": "33", "end_index": "41" }]
    },
    "confidence": 0.99
  },
  // In this context, the token "Question" is English, not French
  "detected_languages": [{ "language_code": "en" }]
}

In the screenshot, did you notice something peculiar in the left block?

Well, punctuation rules can be different between languages. French uses a typographical space for double punctuation marks ("double" as "written in two parts", such as in "!", "?", "«",…). Punctuation is an important part of languages that can get "lost in translation". Here, the space is preserved in the transcription of "Bienvenue !". Nice touch Document AI or, should I say, touché!

{ // document.pages[0].lines[1]
  "layout": {
    "text_anchor": {
      "text_segments": [
        // "Bienvenue !\n" ← document.text[12:24]
        // The French typographical space before "!" is preserved \o/
        { "start_index": "12", "end_index": "24" }
      ]
    },
    "confidence": 0.99
  }
}

For more information, see Language support.

Handwriting detection

Now — a much harder problem — let's check how handwriting is handled.

This example is a mix of printed and handwritten text, where I wrote both a question and an answer. Here are the detected tokens:

I am pleasantly surprised to see my own handwriting transcribed:

{ // document
  "text": "Willkommen!\nBienvenue !\nWelcome!\nQuestion: What is the answer to the ultimate question of life?\nAnswer:\n42\n"
}

I asked my family (used to writing French) to do the same. Each unique handwriting sample also gets correctly detected:

… and transcribed:

{ // document
  "text": [
    "Willkommen!\nBienvenue !\nWelcome!\n",
    "Willkommen!\nBienvenue !\nWelcome!\n",
    "Willkommen!\nBienvenue !\nWelcome!\n",
    "Willkommen!\nBienvenue !\nWelcome!\n",
    "Question: What is the answer to the ultimate question of life?\nAnswer: 42\n",
    "Question: What is the answer to the ultimate question of life?\nAnswer: 42\n",
    "Question: What is the answer to the ultimate question of life?\nAnswer: 42\n",
    "Question: What is the answer to the ultimate question of life?\nAnswer: 42\n"
  ]
}

This can look magical but that's one of the goals of ML models: return results as close as possible to human responses.

Confidence scores

We make mistakes, and so can ML models. To better appreciate the structured data you get, results include confidence scores:

Confidence scores do not represent accuracy.
They represent how confident the model is with the extracted results.
They let you — and your users — ponder the model’s extractions.

Let's overlay confidence scores on top of the previous example:

After grouping them in buckets, confidence scores appear to be generally high, with a few outliers:

The lowest confidence score here is 57%. It corresponds to a handwritten word (token) that is both short (less context given for confidence) and not particularly legible indeed:

{ // document.pages[0].tokens[71].layout
  "text_anchor": {
    // "42\n" ← document.text[351:354]
    "text_segments": [ { "start_index": "351", "end_index": "354" } ]
  },
  // 57% ← Confidence that this token is "42"
  "confidence": 0.57
}

For best results, keep in mind these general rules of thumb:

ML results are best guesses, which can be correct or incorrect.
Results with lower confidence scores are more likely to be incorrect guesses.
If we can't smoothly read a part of a document, or need to think twice about it, it's probably also harder for the ML model.

Although all text is correctly transcribed in the presented examples, this won't always be the case depending on the input document. To build safer solutions — especially with critical business applications — you may consider the following:

Be clear with your users that results are auto-generated.
Communicate the scope and limitations of your solution.
Improve the user experience with interpretable information or filterable results.
Design your processes to trigger human intervention on lower confidence scores.
Monitor your solution to detect changes over time (drift of stable metrics, decline in user satisfaction, etc.).

To learn more about AI principles and best practices, check out Responsible AI practices.

Rotation, skew, distortion

How many times did you scan a document upside down by mistake? Well, this shouldn't be a concern anymore. Text detection is very robust to rotation, skew, and other distortions.

In this example, the webcam input is not only upside down but also skewed, blurry, and with text in unusual orientations:

Before further analysis, Document AI considers the best reading orientation, at the page level, and preprocesses (deskews) each page if needed. This gives you results that can be used and visualized in a more natural way. Once processed by Document AI, the preceding example gets easier to read, without straining your neck:

In the results, each page has an image field by default. This represents the image — deskewed if needed — used by Document AI to extract information. All the page results and coordinates are relative to this image. When a page has been deskewed, a transforms element is present and contains the list of transformation matrices applied to the image:

{ // document.page[0]
  // The image for the page. All coordinates are relative to this image.
  "image": { "content": "…", "mime_type": "image/png", "width": 1150 /*…*/ },
  // 1 transformation matrix was applied to deskew the page
  "transforms": [ { "rows": 2, "cols": 3 /*…*/ } ]
}

Notes:

Page images can be in different formats. For instance, if your input is a JPEG image, the response will include either the same JPEG or a deskewed PNG (as in the earlier example).
Deskewed images have larger dimensions (deskewing adds blank outer areas).
If you don't need visual results in your solution, you can specify a field_mask in your request to receive lighter responses, with only your fields of interest.

Orientation

Documents don't always have all of their text in a single orientation, in which case deskewing alone is not enough. In this example, the sentence is broken out in four different orientations. Each part gets properly recognized and processed in its natural orientation:

Orientations are reported in the layout field:

[ // document.pages[0].blocks
  { "layout": { "orientation": "PAGE_RIGHT" } }, // → 90° clockwise
  { "layout": { "orientation": "PAGE_UP" } },    // ↑ Natural orientation
  { "layout": { "orientation": "PAGE_LEFT" } },  // ← 90° counterclockwise
  { "layout": { "orientation": "PAGE_DOWN" } }   // ↓ 180° from upright
]

Note: Orientations are returned at each OCR level (blocks, paragraphs, lines, and tokens).

Noise

When documents come from our analog world, you can expect … the unexpected. As ML models are trained from real-world samples — containing real-life noise — a very interesting outcome is that Document AI is also significantly robust to noise.

In this example with crumpled paper, the text starts to be difficult to read but still gets correctly transcribed by the OCR model:

Documents can also be dirty or stained. With the same sample, this keeps working after adding some layers of noise:

In both cases, the exact same text is correctly detected:

{ // document
  "text": "Coffee\nCoffee is a brewed beverage made from particular beans. Arthur Dent made himself some coffee on the\nThursday of Earth's destruction; this behaviour was most unusual, as he usually drank tea.\n"
}

You've seen most core features. They are supported by the "Document OCR" general processor as well as the other processors, which leverage these features to focus on more specific document types and provide additional information. Let's check the next-level processor: the "Form Parser" processor.

Form fields

The "Form Parser" processor lets you detect form fields. A form field is the combination of a field name and a field value, also called a key-value pair.

In this example, printed and handwritten text is detected as seen before:

In addition, the form parser returns a list of form_fields:

{ // document
  "text": "My name:\nDEEP THOUGHT\nYour question: What is the answer to the ultimate question?\nMy answer:\n42\n",
  "pages": [
    {
      "page_number": 1,
      "dimension": {/*…*/},
      "layout": {/*…*/},
      "detected_languages": [/*…*/],
      "blocks": [/*…*/],
      "paragraphs": [/*…*/],
      "lines": [/*…*/],
      "tokens": [/*…*/],
      "form_fields": [/* NEW */],
      "image": {/*…*/}
    }
  ]
}

Here is how the detected key-value pairs are returned:

{ // document.pages[0]
  "form_fields": [
    { // "My name:\n"
      "field_name":  { "text_anchor": {/*…*/}, "confidence": 0.96 /*,…*/ },
      // "DEEP THOUGHT\n"
      "field_value": { "text_anchor": {/*…*/}, "confidence": 0.96 /*,…*/ }
    },
    { // "Your question: "
      "field_name":  { "text_anchor": {/*…*/}, "confidence": 0.98 /*,…*/ },
      // "What is the answer to the ultimate question?\n"
      "field_value": { "text_anchor": {/*…*/}, "confidence": 0.98 /*,…*/ }
    },
    { // "My answer:\n"
      "field_name":  { "text_anchor": {/*…*/}, "confidence": 0.76 /*,…*/ },
      // "42\n"
      "field_value": { "text_anchor": {/*…*/}, "confidence": 0.76 /*,…*/ }
    }
  ]
}

And here are their detected bounding boxes:

Note: Form fields can follow flexible layouts. In this example, keys and values are in a left-right order. You'll see a right-left example next. Those are just simple arbitrary examples. It also works with vertical or free layouts where keys and values are logically (visually) related.

Checkboxes

The form parser also detects checkboxes. A checkbox is actually a particular form field value.

This example is a French exam with affirmations that should be checked when exact. To test this, I used checkboxes of different kinds, printed or handmade. All form fields are detected, with the affirmations as field names and the corresponding checkboxes as field values:

When a checkbox is detected, the form field contains an additional value_type field, which value is either unfilled_checkbox or filled_checkbox:

{ // document.pages[0]
  "form_fields": [
    { // [ ] Les cyclines se lient aux CDK…
      "field_name": {/*…*/},
      "field_value": {/*…*/},
      "value_type": "unfilled_checkbox"
    },
    { // [x] Les mutations peuvent être provoquées…
      "field_name": {/*…*/},
      "field_value": {/*…*/},
      "value_type": "filled_checkbox"
    } //,…
  ]
}

Being able to analyze forms can lead to huge time savings, by consolidating — or even autoprocessing — content for you.
The preceding checkbox detection example was actually an evolution of a prior experiment to autocorrect my wife's pile of exam copies.
The proof of concept got better using checkboxes, but was already conclusive enough with True/False handwritten answers. Here is how it can autocorrect and autograde:

Tables

The form parser can also detect another important structural element: tables.

In this example, words are presented in a tabular layout without any borders. The form parser finds a table very close to the (hidden) layout. Here are the detected cells:

In this other example, some cells are filled with text while others are blank. There are enough signals for the form parser to detect a tabular structure:

When tables are detected, the form parser returns a list of tables with their rows and cells. Here is how the table is returned:

{ // document.pages[0]
  "tables": [{
      "layout": {/*…*/},
      "header_rows": [{
        // | ALPHA |       |     |     | EPSILON |     | ETA |       |
        "cells": [/*…*/]
      }],
      "body_rows": [{
        // |       | KAPPA |     | MU  |         | XI  |     | PI    |
        "cells": [/*…*/]
      },
      {
        // | RHO   |       | TAU |     | PHI     |     |     | OMEGA |
        "cells": [/*…*/]
      }]
  }]
}

And here is the first cell:

{ // document.pages[0].tables[0].header_rows[0].cells[0]
  "layout": {
    // ALPHA\n
    "text_anchor": {/*…*/},
    "confidence": 0.9979,
    "bounding_poly": {/*…*/},
    "orientation": "PAGE_UP"
  },
  "row_span": 1,
  "col_span": 1
}

Specialized processors

Specialized processors focus on domain-specific documents and extract entities. They cover many different document types that can currently be classified in the following families:

Procurement — receipts, invoices, utility bills, purchase orders,…
Lending — bank statements, pay slips, official forms,…
Identity — national IDs, driver licenses, passports,…
Contract — legal agreements

For example, procurement processors typically detect the total_amount and currency entities:

{ // document
  "entities": [
    {
      "text_anchor": {/*…*/},
      "type_": "total_amount",
      "mention_text": "15",
      "confidence": 0.96,
      "page_anchor": {/*…*/},
      "id": "0"
    },
    {
      "text_anchor": {/*…*/},
      "type_": "currency",
      "mention_text": "USD",
      "confidence": 0.95,
      "page_anchor": {/*…*/},
      "id": "1"
    }
  ]
}

For more information, check out Fields detected.

Expenses

The "Expense Parser" lets you process receipts of various types. Let's analyze this actual (French) receipt:

A few remarks:

All expected entities are extracted.
Ideally (to be picky), I'd like to get the tax rate too. If there's customer demand for it, this may be a supported entity in a future version.
Due to the receipt's very thin paper, the text on the back side is visible by transparency (another type of noise).
The supplier name is wrong (it's actually mirrored text from the back side) but with a very low confidence (4%). You'd typically handle it with special care (it's shown differently here) or ignore it.
The actual supplier info is hidden (the top part of the receipt is folded) on purpose. You'll see another example with supplier data a bit later.
Receipts are often printed on single (sometimes on multiple) pages. The expense parser supports analyzing expense documents of up to 10 pages.

Procurement documents often list parts of the data in tabular layouts. Here, they're returned as many line_item/* entities. When the entities are detected as part of a hierarchical structure, the results are nested in the properties field of a parent entity, providing an additional level of information. Here's an excerpt:

{ // document.entities[7]
  "text_anchor": {/*…*/},
  "type_": "line_item",
  "mention_text": "*BANANE BIO 1.99",
  "confidence": 1.0,
  "page_anchor": {/*…*/},
  "id": "9",
  "properties": [
    {
      "text_anchor": {/*…*/},
      "type_": "line_item/description",
      "mention_text": "*BANANE BIO",
      "confidence": 0.54613465,
      "page_anchor": {/*…*/},
      "id": "10"
    },
    {
      "text_anchor": {/*…*/},
      "type_": "line_item/amount",
      "mention_text": "1.99",
      "confidence": 0.36692643,
      "page_anchor": {/*…*/},
      "id": "11"
    }
  ]
}

For more information, see the Expense Parser details.

Entity normalization

Getting results is generally not enough. Results often need to be handled in a post-processing stage, which can be both time consuming and a source of errors. To address this, specialized processors also return normalized values when possible. This lets you directly use standard values consolidated from the context of the whole document.

Let's check it with this other receipt:

First, the receipt currency is returned with its standard code under normalized_value:

{ // document.entities[0]
  "type_": "currency",
  "mention_text": "€",
  "normalized_value": {
    "text": "EUR"
  } //,…
}

Then, the receipt is dated 11/12/2022. But is it Nov. 12 or Dec. 11? Document AI uses the context of the document (a French receipt) and provides a normalized value that removes all ambiguity:

{ // document.entities[1]
  "type_": "receipt_date",
  "mention_text": "11/12/2022",
  "normalized_value": {
    "text": "2022-12-11",
    "date_value": {
      "year": 2022,
      "month": 12,
      "day": 11
    }
  } //,…
}

Likewise, the receipt contains a purchase time, written in a non-standard way. The result also includes a canonical value that avoids any interpretation:

{ // document.entities[5]
  "type_": "purchase_time",
  "mention_text": "12h42\n",
  "normalized_value": {
    "datetime_value": {
      "hours": 12,
      "minutes": 42
    }
  } //,…
}

Normalized values simplify the post-processing stage:

They provide standard values that are straightforward to use (e.g. enable direct storage in a data warehouse).
They prevent bugs (esp. the recurring developer mistakes we make when converting data).
They remove ambiguity and avoid incorrect interpretations by using the context of the whole document.

For more information, check out the NormalizedValue structure.

Entity enrichment

Did you notice there was more information in the receipt?

"Maison Jeanne d'Arc, Place de Gaulle" mentions a Joan of Arc's House and a place name. Nonetheless, there is no address, zip code, city, or even country.
This receipt comes most likely from a museum in France, but Joan of Arc lived in a few places (starting with her birthplace in the Domrémy village).
So, which location does this correspond to?
A manual search should give hints to investigate, but can we keep this automated?

Extracting the information behind the data requires extra knowledge or human investigation.
An automated solution would generally ignore this partial data, but Document AI handles this in a unique way.
To understand the world's information, Google has been consistently analyzing the web for over 20 years.
The result is a gigantic up-to-date knowledge base called the Knowledge Graph.
Document AI leverages this knowledge graph to normalize and enrich entities.

First, the supplier is correctly detected and normalized with its usual name:

{ // document.entities[6]
  "type_": "supplier_name",
  "normalized_value": {
    "text": "Maison de Jeanne d'Arc"
  } //,…
}

Then, the supplier city is also returned:

{ // document.entities[8]
  "type_": "supplier_city",
  "normalized_value": {
    "text": "Orléans"
  } //,…
}

Note: Joan of Arc spent some time in Orléans in 1429, as she led the liberation of the besieged city at the age of 17 (but that's another story).

And finally, the complete and canonical supplier address is also part of the results, closing our case here:

{ // document.entities[7]
  "type_": "supplier_address",
  "normalized_value": {
    "text": "3 Pl. du Général de Gaulle\n45000 Orléans\nFrance"
  } //,…
}

Enriched entities bring significant value:

They are canonical results, which avoids information conflicts and discrepancies.
They can add information, which prevents ignoring useful data.
They can complete or fix partially correct data.
In a nutshell, they provide information that is reliable, consistent, and comparable.

Here is a recap of the expected — as well as the non-obvious — entities detected in the receipt:

Note: The slightest characteristic element can be sufficient to detect an entity.
I've been regularly surprised to get entities I wasn't expecting, to eventually realize I had missed clues in the document.
For example, I recently wondered how the expense processor was able to identify a specific store.
The receipt only specified a zip code and the retail chain has several stores in my neighborhood.
Well, a public phone number (hidden in the footer) was enough to uniquely identify the store in question and provide its full address.

To learn more about the Knowledge Graph and possible enrichments, check out Enrichment and normalization.

Invoices

Invoices are the most elaborate type of procurement documents, often spreading over multiple pages.

Here's an example showing entities extracted by the "Invoice Parser":

A few remarks:

The relevant information is extracted (even the supplier tax ID is detected on the 90°-rotated left side).
As seen earlier with the expense parser, the invoice parser extracts many line_item/* entities but also supplier_* and receiver_* info.
This is a typical invoice (issued by the historical French energy provider) with many numbers on the first pages.
The original source is a PDF (digital) that was anonymized, printed, stained, and scanned as a PDF (raster).

For more information, see the Invoice Parser details.

Barcodes

Barcode detection is enabled for some processors. In this example, the invoice parser detects the barcode (a manifest number):

Note: I didn't have an invoice with barcodes at hand, so I used a (slightly anonymized) packing list.

Barcodes are returned, at the page level, like this:

{ // document.pages[0]
  "detected_barcodes": [
    {
      "layout": {/*…*/},
      "barcode": {
        "format_": "CODE_39",
        "value_format": "TEXT",
        "raw_value": "E-58702865"
      }
    }
  ]
}

For more information, check out the DetectedBarcode structure.

Identity documents

Identity processors let you extract identity entities. In this US passport specimen (credit: Bureau of Consular Affairs), the expected fields can be automatically extracted:

For more details about identity processors, you can read my previous article Automate identity document processing.

Document signals

Some processors return document signals — information relative to the document itself.

For example, the "Document OCR" processor returns quality scores for the document, estimating the defects that might impact the accuracy of the results.

The preceding crumpled paper example gets a high quality score of 95%, with glare as a potential defect:

The same example, at a 4x lower resolution, gets a lower quality score of 53%, with blurriness detected as the main potential issue:

Some additional remarks:

A low score lets you flag documents that may need a manual review (or a new better capture).
Surprisingly, the exact same text is correctly extracted in both cases (I checked twice) but that might not always be the case.
The quality scores are returned both as entities (with a quality_score parent entity) and in the image_quality_scores field at the page level (see ImageQualityScores).
This was tested with release candidate pretrained-ocr-v1.1-2022-09-12.

Here's how the image quality scores are returned at the page level:

{ // document.pages[0]
  "image_quality_scores": {
    "quality_score": 0.53,
    "detected_defects": [
      { "confidence": 0.98, "type_": "quality/defect_blurry" },
      { "confidence": 0.77, "type_": "quality/defect_glare" },
      { "confidence": 0.16, "type_": "quality/defect_text_too_small" },
      { "confidence": 0.05, "type_": "quality/defect_noisy" },
      { "confidence": 0.03, "type_": "quality/defect_faint" },
      { "confidence": 0.01, "type_": "quality/defect_dark" }
    ]
  }
}

Likewise, the "Identity Proofing" processor gives you signals about the validity of ID documents.

First, this can help detect whether documents look like IDs. This preceding example is a random document analyzed as NOT_AN_ID:

Here are the corresponding entities:

{ // document
  "entities": [
    {
      "type_": "fraud_signals_is_identity_document",
      "mention_text": "NOT_AN_ID" //,…
    },
    {
      "type_": "fraud_signals_suspicious_words",
      "mention_text": "PASS" //,…
    },
    {
      "type_": "fraud_signals_image_manipulation",
      "mention_text": "PASS" //,…
    },
    {
      "type_": "fraud_signals_online_duplicate",
      "mention_text": "PASS" //,…
    }
  ]
}

The preceding passport example does get detected as an ID but also triggers useful fraud signals:

The results include evidences that #1 it's a specimen, #2 which can be found online:

{ // document
  "entities": [
    {
      "type_": "fraud_signals_is_identity_document",
      "mention_text": "PASS" //,…
    },
    {
      "type_": "fraud_signals_suspicious_words",
      "mention_text": "SUSPICIOUS_WORDS_FOUND" //,…
    },
    {
      "type_": "evidence_suspicious_word",
      "mention_text": "SPECIMEN" //,…
    },
    {
      "type_": "fraud_signals_image_manipulation",
      "mention_text": "POSSIBLE_IMAGE_MANIPULATION" //,…
    },
    {
      "type_": "evidence_thumbnail_url",
      "mention_text": "https://….gstatic.com/images?…" //,…
    },
    {
      "type_": "evidence_hostname",
      "mention_text": "pbs.twimg.com" //,…
    },
    {
      "type_": "fraud_signals_online_duplicate",
      "mention_text": "POSSIBLE_ONLINE_DUPLICATE" //,…
    }
  ]
}

As new use cases appear, it's likely that some processors will extract new document signals. Follow the Release notes to stay updated.

Pre-trained processors

Document AI is already huge and keeps evolving. Here is a screencast showing the current processor gallery and how to create a processor from the Cloud Console:

Custom processors with Workbench

If you have your own business-specific documents, you may wish to extract custom entities not covered by the existing processors. Document AI Workbench can help you solve this by creating your own custom processor, trained with your own documents, in two ways:

With uptraining, you can extend an existing processor.
You can also create a new processor from scratch.

For more information, watch this great introduction video made by my teammate Holt:

Performance

To capture your documents, you'll probably use either:

A scanner
A camera

With scanners, you'll need to choose a resolution in dots per inch (dpi). To optimize the performance (especially the accuracy and consistency of the results), you can keep in mind the following:

300 dpi is often a nice spot
200 dpi is generally a hard minimum

Here are the resolutions needed to capture a sheet of paper:

Note: Dimensions are presented in the horizontal orientation (image sensors generally have a landscape native resolution).

With cameras, the captured dots per inch depend on the camera resolution but also on the way you zoom on the document. Here's an example with an A4 sheet of paper:

This translates into different dpi ranges. Here are indicative values:

To give you an idea, here is how a PNG image size evolves when capturing the same document at different resolutions. The total number of pixels is a surface. Though the PNG compression slightly limits the growth, the size increases almost quadratically:

A few general observations:

Document captures are usually in the 100-600 dpi range, depending on the use case.
100-200 dpi may be enough for some specific end user solutions, for example to process small receipts captured from mobile phone cameras or laptop webcams.
Scans of 300-600 dpi may provide better results but will generate larger files and make your processing pipeline slower.
Scans above 600 dpi are less likely. They are useful for very small documents (such as slides or negative films) or documents that need to be upscaled (e.g. to print a poster).
If your use case involves documents with small text, barcodes, or very noisy (e.g. blurry) inputs, you'll typically need to compensate with a higher capture resolution.
Capture devices have noise intrinsic to the technology they use. They generally have options that will produce different results. Also, image sensors are more noisy in low-light environments.
Every conversion step in your architecture is likely to introduce side effects and a potential loss of information.
Lossy image formats (such as JPEG) can nicely reduce image sizes. On the other hand, too high compression levels (a setting) can generate artifacts with a negative impact on the detection performance (causing a strong degradation of image quality and consequently of result accuracy).
Lossless formats (such as PNG) imply larger file sizes but may bring the benefit of using lower resolutions. They may also be more future-proof (e.g. for archival) by preserving the possibility of later conversions to other formats or to smaller resolutions without losing information.
Storage costs are rather low, so it may also be interesting to consider higher acceptable scanning resolutions, to optimize both the performance and the overall cost of your solution.
In a nutshell, choosing capture devices, resolutions, and formats depends on your use case and is a tradeoff between accuracy, speed, and cost.

To finetune your solution and get more accurate, faster, or consistent results, you may consider the following:

Check the performance of your solution with worst-case documents at different resolutions.
Have a clear understanding of the different formats used in your processing pipeline.
Test your solution with actual documents (as close as possible to what you'll have in production). Also, as the ML models are trained on real documents, you may get better results when testing real documents.
Keep a lossless format upstream when possible; documents can be converted to a lossy format downstream.
Evaluate whether capturing at a higher resolution and then downscaling produces better results.
When using lossy formats, finetune the compression level. There is usually a threshold at which compression artifacts become negligible while maintaining a good compression ratio.
When using cameras, control your lighting environment and avoid low-light conditions.
Make your users aware that blurry captures may decrease the performance.

For example:

The examples in this article can all be analyzed using cameras with resolutions ranging from 100 to 220 dpi. However, results won't always be consistent with low resolutions.
For practical reasons, I mostly use laptop, mobile phone, or dedicated cameras. There are excellent ones (called document cameras) that can sit on your desk and provide solid captures with a fast auto-focus.
For an enterprise-grade solution, I'd probably stick to 300 dpi scans. Multifunction printers generally have scanners that provide excellent PNG, PDF, or TIFF captures.
The barcode (contained in the preceding packing list sample) starts to be detected at 150 dpi with a scanner, and at 200 dpi with a webcam. After being photocopied (i.e. scanned + printed), it's still detected at 150 dpi with a scanner but needs to be zoomed in with the webcam.
While capturing documents from a browser and a webcam, I checked the HTML canvas default format: it's PNG (great!) but in RGBA (with a transparency channel, unnecessary for document images). This made the data larger without any benefit. The optimization was a parameter away (canvas.getContext('2d') → canvas.getContext('2d', { alpha: false })) and made the captures sent to the backend 9 to 18% smaller.

Sample demo

I put myself in your shoes to see what it takes to build a document processing prototype and made the following choices:

A frontend using vanilla JavaScript, responsible for managing user interactions and input documents
A backend using the Python client library, responsible for all document processings and renderings
Synchronous calls to Document AI, to process documents in real-time
No cloud storage, to avoid storing personally identifiable information (analyses are stored client-side)

Here is the chosen software stack based on open-source Python projects:

… and one possible architecture to deploy it to production using Cloud Run:

Here is the core function I use to process a document live in Python:

from typing import BinaryIO

from google.cloud.documentai_v1 import (
    Document,
    DocumentProcessorServiceClient,
    ProcessRequest,
    RawDocument,
)


def process_document(
    file: BinaryIO,
    mime_type: str,
    project: str,
    location: str,
    processor_id: str,
) -> Document:
    """Analyze the input file with Document AI."""
    client_options = {"api_endpoint": f"{location}-documentai.googleapis.com"}
    client = DocumentProcessorServiceClient(client_options=client_options)
    raw_document = RawDocument(content=file.read(), mime_type=mime_type)
    name = client.processor_path(project, location, processor_id)

    request = ProcessRequest(
        raw_document=raw_document,
        name=name,
        skip_human_review=True,
    )
    response = client.process_document(request)

    return response.document

To make sure that "what you see is what you get", the sample images and animations in this article have been generated by the demo:

Note: To avoid making repeated and unnecessary calls to Document AI while developing the app, sample document analyses are cached (JSON serialization). This lets you check the returned structured documents and also explains why responses are immediate in this (real-time, not sped up) screencast.

It takes seconds to analyze a document. Here is an example with the user uploading a PDF scan:

You can also take camera captures from the web app:

Check out the source code and feel free to reuse it. You'll also find instructions to deploy it as a serverless app.

More?

Test documents? → Try Document AI in your browser.
Video series? → Watch The future of documents.
New to Google Cloud? → Check out the $300 free trial offer.
Learn and grow? → Become a Google Cloud Innovator.
Feedback or questions? → Reach out on Twitter (@PicardParis) or LinkedIn (in/PicardParis).

Automate identity document processing with Document AI

Laurent Picard — Thu, 02 Jun 2022 13:07:59 +0000

How many times have you filled out forms requesting personal information? It's probably too many times to count. When online and signed in, you can save a lot of time thanks to your browser’s autofill feature. In other cases, you often have to provide the same data manually, again and again. The first Document AI identity processors are now generally available and can help you solve this problem.

In this post, you’ll see how to…

Process identity documents with Document AI
Create your own identity form autofiller

Use cases

Here are a few situations that you've probably encountered:

Financial accounts: Companies need to validate the identity of individuals. When creating a customer account, you need to present a government-issued ID for manual validation.
Transportation networks: To handle subscriptions, operators often manage fleets of custom identity-like cards. These cards are used for in-person validation, and they require an ID photo.
Identity gates: When crossing a border (or even when flying domestically), you need to pass an identity check. The main gates have streamlined processes and are generally well equipped to scale with the traffic. On the contrary, smaller gates along borders can have manual processes – sometimes on the way in and the way out – which can lead to long lines and delays.
Hotels: When traveling abroad and checking in, you often need to show your passport for a scan. Sometimes, you also need to fill out a longer paper form and write down the same data.
Customer benefits: For benefit certificates or loyalty cards, you generally have to provide personal info, which can include a portrait photo.

In these examples, the requested info – including the portrait photo – is already on your identity document. Moreover, an official authority has already validated it. Checking or retrieving the data directly from this source of truth would not only make processes faster and more effective, but also remove a lot of friction for end users.

Identity processors

Processor types

Each Document AI identity processor is a machine learning model trained to extract information from a standard ID document such as:

Driver license
National ID
Passport

Note: an ID can have information on both sides, so identity processors support up to two pages per document.

Availability

Generally available as of June 2022, you can use two US identity processors in production:

Processor	API `type`
US Driver License Parser	`US_DRIVER_LICENSE_PROCESSOR`
US Passport Parser	`US_PASSPORT_PROCESSOR`

Currently available in Preview:

The Identity Doc Fraud Detector, to check whether an ID document has been tampered with
Three French identity processors

Processor	API `type`
Identity Doc Fraud Detector	`ID_FRAUD_DETECTION_PROCESSOR`
France Driver License Parser	`FR_DRIVER_LICENSE_PROCESSOR`
France National ID Parser	`FR_NATIONAL_ID_PROCESSOR`
France Passport Parser	`FR_PASSPORT_PROCESSOR`

Notes:

More identity processors are in the pipe.
To request access to processors in Preview, please fill out the Access Request Form.

Processor creation

You can create a processor:

Manually from Cloud Console (web admin UI)
Programmatically with the API

Processors are location-based. This helps guarantee where processing will occur for each processor.

Here are the current multi-region locations:

Location	API `location`
United States	`us`
European Union	`eu`

Once you've created a processor, you reference it with its ID (PROCESSOR_ID hereafter).

Note: To manage processors programmatically, see the codelab Managing Document AI processors with Python.

Document processing

You can process documents in two ways:

Synchronously with an online request, to analyze a single document and directly use the results
Asynchronously with a batch request, to launch a batch processing operation on multiple or larger documents

Online requests

Example of a REST online request:

The method is named process.
The input document here is a PNG image (base64 encoded).
This request is processed in the European Union.
The response is returned synchronously.

POST https://eu-documentai.googleapis.com
     /v1/projects/PROJECT_ID/locations/eu/processors/PROCESSOR_ID
     :process

{
  "rawDocument": {
    "content": "iVBORw0KGg…",
    "mimeType": "image/png"
  },
  "skipHumanReview": true
}

Batch requests

Example of a REST batch request:

The method is named batchProcess.
The batchProcess method launches the batch processing of multiple documents.
This request is processed in the United States.
The response is returned asynchronously; output files will be stored under my-storage-bucket/output/.

POST https://us-documentai.googleapis.com
     /v1/projects/PROJECT_ID/locations/us/processors/PROCESSOR_ID
     :batchProcess

{
  "inputDocuments": {
    "gcsDocuments": {
      "documents": [
        {
          "gcsUri": "gs://my-storage-bucket/input/id-doc-1.pdf",
          "mimeType": "application/pdf"
        },
        {
          "gcsUri": "gs://my-storage-bucket/input/id-doc-2.tiff",
          "mimeType": "image/tiff"
        },
        {
          "gcsUri": "gs://my-storage-bucket/input/id-doc-3.png",
          "mimeType": "image/png"
        },
        {
          "gcsUri": "gs://my-storage-bucket/input/id-doc-4.gif",
          "mimeType": "image/gif"
        }
      ]
    }
  },
  "documentOutputConfig": {
    "gcsOutputConfig": {
      "gcsUri": "gs://my-storage-bucket/output/"
    }
  },
  "skipHumanReview": true
}

Interfaces

Document AI is available through the usual Google Cloud interfaces:

The RPC API (low-latency gRPC)
The REST API (JSON requests and responses)
Client libraries (gRPC wrappers, currently available for Python, Node.js, and Java)
Cloud Console (web admin UI)

Note: With the client libraries, you can develop in your preferred programming language. You'll see an example later in this post.

Identity fields

A typical REST response looks like the following:

The text and pages fields include the OCR data detected by the underlying ML models. This part is common to all Document AI processors.
The entities list contains the fields specifically detected by the identity processor.

{
  "text": "…",
  "pages": […],
  "entities": [
    {
      "textAnchor": {…},
      "type": "Family Name",
      "mentionText": "PICHAI",
      "confidence": 0.999945,
      "pageAnchor": {…},
      "id": "4"
    },
    {
      "textAnchor": {…},
      "type": "Given Names",
      "mentionText": "Sundar",
      "confidence": 0.9999612,
      "pageAnchor": {…},
      "id": "5"
    }
    ,…
  ]
}

Here are the detectable identity fields:

Entity type	Comment
`Portrait`	Bounding box (portrait photo)
`Family Name`	String
`Given Names`	String
`Document Id`	String
`Expiration Date`	Date (+ normalization)
`Date Of Birth`	Date (+ normalization)
`Issue Date`	Date (+ normalization)
`Address`	String
`MRZ Code`	String (Machine Readable Zone)

Please note that Address and MRZ Code are optional fields. For example, a US passport contains an MRZ but no address.

Fraud detection

Available in preview, the Identity Doc Fraud Detector helps detect tampering attempts. Typically, when an identity document does not "pass" the fraud detector, your automated process can block the attempt or trigger a human validation.

Here is an example of signals returned:

Entity type	Example
`fraud-signals/is-identity-document`	"NOT_AN_ID"
`fraud-signals/suspicious-words`	"PASS"
`fraud-signals/image-manipulation`	"POSSIBLE_IMAGE_MANIPULATION"

Sample demo

You can process a document live with just a few lines of code.

Here is a Python example:

import google.cloud.documentai_v1 as docai


def process_document(
    file: typing.BinaryIO,
    mime_type: str,
    project_id: str,
    location: str,
    processor_id: str,
) -> docai.Document:
    api_endpoint = {"api_endpoint": f"{location}-documentai.googleapis.com"}
    client = docai.DocumentProcessorServiceClient(client_options=api_endpoint)

    raw_document = docai.RawDocument(content=file.read(), mime_type=mime_type)
    name = client.processor_path(project_id, location, processor_id)

    request = docai.ProcessRequest(
        raw_document=raw_document,
        name=name,
        skip_human_review=True,
    )
    response = client.process_document(request)

    return docai.Document(response.document)

This function uses the Python client library:

The input is a file (any format supported by the processor).
client is an API wrapper (configured for processing to take place in the desired location).
process_document calls the API process method, which returns results in seconds.
The output is a structured Document.

You can collect the detected fields by parsing the document entities:

def id_data_from_document(document: docai.Document) -> dict:
    id_data = defaultdict(dict)

    for entity in document.entities:
        key = entity.type_
        page_index = 0
        value = entity.mention_text
        confidence, normalized = None, None

        if entity.page_anchor:
            page_index = entity.page_anchor.page_refs[0].page
        if not value:  # Send the detected portrait image instead
            image = crop_entity(document, entity)
            value = data_url_from_image(image)
        if entity.confidence != 0.0:
            confidence = int(entity.confidence * 100 + 0.5)
        if entity.normalized_value:
            normalized = entity.normalized_value.text

        id_data[key][page_index] = dict(
            value=value,
            confidence=confidence,
            normalized=normalized,
        )

    return id_data

Note: This function builds a mapping ready to be sent to a frontend. A similar function can be used for other specialized processors.

Finalize your app:

Define your user experience and architecture
Implement your backend and its API
Implement your frontend with a mix of HTML + CSS + JS
Add a couple of features: file uploads, document samples, or webcam captures
That's it; you've built an identity form autofiller

Here is a sample web app in action:

Here is the processing of a French national ID, dropping images from the client:

Note: For documents with multiple pages, you can use a PDF or TIFF container. In this example, the two uploaded PNG images are merged by the backend and processed as a TIFF file.

And this is the processing of a US driver license, captured with a laptop 720p webcam:

Notes:

Did you notice that the webcam capture is skewed and the detected portrait image straight? That's because Document AI automatically deskews the input at the page level. Documents can even be upside down.
Some fields (such as the dates) are returned with their normalized values. This can make storing and processing these values a lot easier – and less error-prone – for developers.

The source code for this demo is available in our Document AI sample repository.

Stay tuned; the family of Document AI processors keeps growing and growing.

🎨 Image processing as a service 🐍

Laurent Picard — Thu, 07 Apr 2022 07:56:59 +0000

👋 Hello

Have you ever written a script to transform an image? Did you share the script with others or did you run it on multiple computers? How many times did you need to update the script or the setup instructions? Did you end up making it a service or an online app? If your script is useful, you’ll likely want to make it available to others. Deploying processing services is a recurring need – one that comes with its own set of challenges. Serverless technologies let you solve these challenges easily and efficiently.

In this post, you’ll see how to…

Create an image processing service that generates coloring pages
Make it available online using minimal resources

…and do it all in less than 200 lines of Python and JavaScript!

🛠️ Tools

To build and deploy a coloring page generator, you’ll need a few tools:

A library to process images
A web application framework
A web server
A serverless solution to make the demo available 24/7

🧱 Architecture

Here is one possible architecture for a coloring page generator using Cloud Run:

And here is the workflow:

1 - The user opens the web app: the browser requests the main page.
2 - Cloud Run serves the app HTML code.
3 - The browser requests the additional needed resources.
4 - Cloud Run serves the CSS, JavaScript, and other resources.
A - The user selects an image and the frontend sends the image to the /api/coloring-page endpoint.
B - The backend processes the input image and returns an output image, which the user can then visualize, download, or print via the browser.

🐍 Software stack

Of course, there are many different software stacks that you could use to implement such an architecture.

Here is a good one based on Python:

It includes:

Gunicorn: A production-grade WSGI HTTP server
Flask: A popular web app framework
scikit-image: An extensive image processing library

Define these app dependencies in a file named requirements.txt:

# https://pypi.org/project/gunicorn
gunicorn==20.1.0

# https://pypi.org/project/flask
Flask==2.1.1

# https://pypi.org/project/scikit-image
# scikit-image dependencies include NumPy and Pillow
scikit-image==0.19.2

🎨 Image processing

How do you remove colors from an image? One way is by detecting the object edges and removing everything but the edges in the result image. This can be done with a Sobel filter, a convolution filter that detects the regions in which the image intensity changes the most.

Create a Python file named main.py, define an image processing function, and within it use the Sobel filter and other functions from scikit-image:

import numpy as np
import skimage
from PIL import Image
from PIL.Image import Image as PilImage


def generate_coloring_page(input: PilImage) -> PilImage:
    # Convert to grayscale if needed
    if input.mode != "L":
        input = input.convert("L")
    np_image = np.asarray(input)

    # Detect the edges
    np_image = skimage.filters.sobel(np_image)
    # Convert to 8 bpp
    np_image = skimage.util.img_as_ubyte(np_image)
    # Invert to get dark edges on a light background
    np_image = 255 - np_image
    # Improve the contrast
    np_image = skimage.exposure.rescale_intensity(np_image)

    return Image.fromarray(np_image)

Note: The NumPy and Pillow libraries are automatically installed as dependencies of scikit-image.

As an example, here is how the Cloud Run logo is processed at each step:

✨ Web app

Backend

To expose both endpoints (GET / and POST /api/coloring-page), add Flask routes in main.py:

import io

import flask
from PIL import Image

app = flask.Flask(__name__, static_url_path="")


@app.get("/")
def index():
    return app.send_static_file("index.html")


@app.post("/api/coloring-page")
def coloring_page():
    file = flask.request.files.get("input-image")
    if file is None:
        return "Missing input-image parameter", 400

    input_image = Image.open(file.stream)
    output_image = generate_coloring_page(input_image)

    image_io = io.BytesIO()
    output_format = "png"
    output_image.save(image_io, format=output_format)
    image_io.seek(0)

    return flask.send_file(image_io, mimetype=f"image/{output_format}")

Frontend

On the browser side, write a JavaScript function that calls the /api/coloring-page endpoint and receives the processed image:

async function fetchColoringPage(inputFile) {
    const formData = new FormData()
    formData.append('input-image', inputFile)

    const url = '/api/coloring-page'
    const init = { method: 'POST', body: formData }
    try {
        const response = await fetch(url, init)
        return response.ok ? response.blob() : null
    } catch (error) {
        console.error(error)
        return null
    }
}

The base of your app is there. Now you just need to add a mix of HTML + CSS + JS to complete the desired user experience.

Local development

To develop and test the app on your computer, once your environment is set up, make sure you have the needed dependencies:

pip install --upgrade -r requirements.txt

Add the following block to main.py. It will only execute when you run your app manually:

import os

# ...

if __name__ == "__main__":
    os.environ["FLASK_ENV"] = "development"
    app.run(host="localhost", port=8080, debug=True)

Run your app:

python main.py

Flask starts a local web server:

 * Serving Flask app 'main' (lazy loading)
 * Environment: development
 * Debug mode: on
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 718-408-327
 * Running on http://localhost:8080/ (Press CTRL+C to quit)

Note: In this mode, you’re using a development web server (one that is not suited for production). You’ll next set up the deployment to serve your app with Gunicorn, a production-grade server.

You're all set. Open localhost:8080 in your browser, test, refine, and iterate.

🚀 Deployment

Once your app is ready for prime time, you can define how it will be served with this single line in a file named Procfile:

web: gunicorn --bind :$PORT --workers 1 --threads 8 --timeout 0 main:app

At this stage, here are the files found in a typical project:

.
├── main.py
├── Procfile
├── requirements.txt
└── static
    ├── favicon.ico
    ├── index.html
    ├── scripts.js
    └── styles.css

That's it, you can now deploy your app from the source folder:

SERVICE="coloring-page-generator"
SOURCE="."

gcloud run deploy $SERVICE --source $SOURCE --allow-unauthenticated

⚙️ Under the hood

The command line output details all the different steps:

This command is equivalent to running `gcloud builds submit --pack image=[IMAGE] SOURCE` and `gcloud run deploy SERVICE --image [IMAGE]`

Building using Buildpacks and deploying container to Cloud Run service [SERVICE] in project [PROJECT_ID] region [REGION]
OK Building and deploying... Done.
  OK Creating Container Repository...
  OK Uploading sources...
  OK Building Container... Logs are available at […].
  OK Creating Revision...
  OK Routing traffic...   
  OK Setting IAM Policy...
Done.
Service [SERVICE] revision [SERVICE-REVISION] has been deployed and is serving 100 percent of traffic.
Service URL: https://SERVICE-PROJECTHASH-REGIONID.a.run.app

Cloud Build is indirectly called to containerize your app. One of its core components is Google Cloud Buildpacks, which automatically builds a production-ready container image from your source code. Here are the main steps:

Cloud Build fetches the source code.
Buildpacks autodetects the app language (Python, in this case) and uses the corresponding secure base image.
Buildpacks installs the app dependencies (defined in requirements.txt for Python).
Buildpacks configures the service entrypoint (defined in Procfile for Python).
Cloud Build pushes the container image to Artifact Registry.
Cloud Run creates a new revision of the service based on this container image.
Cloud Run routes production traffic to it.

Notes:

Buildpacks currently supports the following runtimes: Go, Java, .NET, Node.js, and Python.
The base image is actively maintained by Google, scanned for security vulnerabilities, and patched against known issues. This means that, when you deploy an update, your service is based on an image that is as secure as possible.
If you need to build your own container image, for example with a custom runtime, you can add your own Dockerfile and Buildpacks will use it instead.

💫 Updates

More testing from real-life users shows some issues.

First, the app does not handle pictures taken with digital cameras in non-native orientations. You can fix this using the EXIF orientation data:

-from PIL import Image
+from PIL import Image, ImageOps
...
def generate_coloring_page(input: PilImage) -> PilImage:
    # Convert to grayscale if needed
    if input.mode != "L":
        input = input.convert("L")
+   # Transpose if taken in non-native orientation (rotated digital camera)
+   NATIVE_ORIENTATION = 1
+   if input.getexif().get(0x0112, NATIVE_ORIENTATION) != NATIVE_ORIENTATION:
+       input = ImageOps.exif_transpose(input)
    np_image = np.asarray(input)
    ...

In addition, the app is too sensitive to details in the input image. Textures in paintings, or noise in pictures, can generate many edges in the processed image. You can improve the processing algorithm by adding a denoising step upfront:

...
def generate_coloring_page(input: PilImage) -> PilImage:
    ...
+   # Remove some noise to keep the most visible edges
+   np_image = skimage.restoration.denoise_tv_chambolle(np_image, weight=0.05)
    # Detect the edges
    np_image = skimage.filters.sobel(np_image)
    ...

This additional step makes the coloring page cleaner and reduces the quantity of ink used if you print it:

Redeploy, and the app is automatically updated:

gcloud run deploy $SERVICE --source $SOURCE

🎉 It's alive

The app is visible as a service in Cloud Run:

The service dashboard gives you an overview of app usage:

That's it; your image processing app is in production!

🤯 It's serverless

There are many benefits to using Cloud Run in this architecture:

Your app is available 24/7.
The environment is fully managed: you can focus on your code and not worry about the infrastructure.
Your app is automatically available through HTTPS.
You can map your app to a custom domain.
Cloud Run scales the number of instances automatically and the billing includes only the resources used when your code runs.
If your app is not used, Cloud Run scales down to zero.
If your app gets more traffic (imagine it makes the news), Cloud Run scales up to the number of instances needed.
You can control performance and cost by fine-tuning many settings: CPU, memory, concurrency, minimum instances, maximum instances, and more.
Every month, the free tier offers the first 50 vCPU-hours, 100 GiB-hours, and 2 million requests for no cost.

💾 Source code

The project includes just seven files and less than 200 lines of Python + JavaScript code.

You can reuse this demo as a base to build your own image processing app:

Check out the source code on GitHub.
For step-by-step instructions on deploying the app yourself in a few minutes, see “Deploying from scratch”.

🖖 More

Try the demo and generate your own coloring pages.
Learn more about Cloud Run.
For more cloud content, follow me on Twitter (@PicardParis) or LinkedIn (in/PicardParis), and feel free to get in touch with any feedback or questions.

🕵️ Detecting faces as a service 🐍

Laurent Picard — Tue, 29 Sep 2020 16:00:36 +0000

👋 Hello!

In this article, you'll see the following:

how to detect faces in pictures,
how to automatically anonymize, crop,… a picture with faces,
how to make this a serverless online demo,
in less than 300 lines of Python code.

Here is a famous face that has been automatically anonymized and cropped. Do you guess who this is?

Note: We're talking about face detection, not face recognition. Though technically possible, face recognition can have harmful applications. Responsible companies have established AI principles and avoid exposing such potentially harmful technologies (e.g. Google AI Principles).

🛠️ Tools

A few tools will do:

a machine learning model to analyze images,
a library to process images,
a web application framework,
a serverless solution to keep the demo available 24/7 and at minimal cost.

🧱 Architecture

Here is an architecture using 2 Google Cloud services (App Engine + Vision API):

The workflow is the following:

Open the demo: App Engine serves the home page.
Take a selfie: the frontend sends it to the /analyze-image endpoint.
The backend sends a request to the Vision API: the image is analyzed and the results (annotations) are returned.
The backend returns the annotations, in addition to the number of detected faces (to display the info directly in the web page).
The frontend sends image, annotations, and processing options to the /process-image endpoint.
The backend processes the image with the given options and returns the result image.
Change the options: steps 5 and 6 are repeated.
Get the image with new options.

This is one of many possible architectures. The advantages of this one are the following:

The web browser caches both the selfie and the annotations: no storage is involved and no private images are stored anywhere in the cloud.
The Vision API is only called once per image.

🐍 Python libraries

Google Cloud Vision

The client library that wraps calls to the Vision API.
https://pypi.org/project/google-cloud-vision

Pillow

A very popular imaging library, both extensive and easy to use.
https://pypi.org/project/Pillow

Flask

One of the most popular web app frameworks.
https://pypi.org/project/Flask

Dependencies

Define the dependencies in the requirements.txt file:

google-cloud-vision==2.4.4

Pillow==8.3.2

Flask==2.0.2

Notes:

As a best practice, also specify the dependency versions. This freezes your production environment in a known state and prevents newer versions from potentially breaking future deployments.

App Engine will automatically deploy these dependencies.

🧠 Image analysis

Vision API

The Vision API gives access to state-of-the-art machine learning models for image analysis. One of the multiple features is face detection. Here is a way to detect faces in an image:

from google.cloud import vision_v1 as vision

Annotations = vision.AnnotateImageResponse
MAX_DETECTED_FACES = 50

def detect_faces(image_bytes: bytes) -> Annotations:
    client = vision.ImageAnnotatorClient()
    api_image = vision.Image(content=image_bytes)
    return client.face_detection(api_image, max_results=MAX_DETECTED_FACES)

Backend endpoint

Exposing an API endpoint with Flask consists in wrapping a function with a route. Here is a possible POST endpoint:

import base64

import flask

app = flask.Flask(__name__)

@app.post("/analyze-image")
def analyze_image():
    image_file = flask.request.files.get("image")
    annotations = detect_faces(image_file.read())

    return flask.jsonify(
        faces_detected=len(annotations.face_annotations),
        annotations=encode_annotations(annotations),
    )

def encode_annotations(annotations: Annotations) -> str:
    binary_data = Annotations.serialize(annotations)
    base64_data = base64.urlsafe_b64encode(binary_data)
    base64_annotations = base64_data.decode("ascii")
    return base64_annotations

Frontend request

Here is a javascript function to call the API from the frontend:

async function analyzeImage(imageBlob) {
  const formData = new FormData();
  formData.append("image", imageBlob);
  const params = { method: "POST", body: formData };
  const response = await fetch("/analyze-image", params);

  return response.json();
}

🎨 Image processing

Face bounding box and landmarks

The Vision API provides the bounding box of the detected faces and the position of 30+ face landmarks (mouth, nose, eyes,…). Here is a way to visualize them with Pillow (PIL):

from PIL import Image, ImageDraw

PilImage = Image.Image
ANNOTATION_COLOR = "#00FF00"
ANNOTATION_LANDMARK_DIM_PERMIL = 8
ANNOTATION_LANDMARK_DIM_MIN = 4

def draw_face_landmarks(image: PilImage, annotations: Annotations):
    r_half = min(image.size) * ANNOTATION_LANDMARK_DIM_PERMIL // 1000
    r_half = max(r_half, ANNOTATION_LANDMARK_DIM_MIN) // 2
    border = max(r_half // 2, 1)

    draw = ImageDraw.Draw(image)
    for face in annotations.face_annotations:
        v = face.bounding_poly.vertices
        r = (v[0].x, v[0].y, v[2].x + 1, v[2].y + 1)
        draw.rectangle(r, outline=ANNOTATION_COLOR, width=border)

        for landmark in face.landmarks:
            x = int(landmark.position.x + 0.5)
            y = int(landmark.position.y + 0.5)
            r = (x - r_half, y - r_half, x + r_half + 1, y + r_half + 1)
            draw.rectangle(r, outline=ANNOTATION_COLOR, width=border)

Face anonymization

Here is way to anonymize the faces thanks to the bounding boxes:

ANONYMIZATION_PIXELS=13

def anonymize_faces(image: PilImage, annotations: Annotations):
    for face in annotations.face_annotations:
        v = face.bounding_poly.vertices
        face = image.crop((v[0].x, v[0].y, v[2].x + 1, v[2].y + 1))

        face1_w, face1_h = face.size
        pixel_dim = max(face1_w, face1_h) // ANONYMIZATION_PIXELS
        face2_w, face2_h = face1_w // pixel_dim, face1_h // pixel_dim
        face = face.resize((face2_w, face2_h), Image.NEAREST)
        face = face.resize((face1_w, face1_h), Image.NEAREST)

        image.paste(face, (v[0].x, v[0].y))

Face cropping

Similarly, to focus on the detected faces, you can crop everything around the faces:

CROP_MARGIN_PERCENT = 10

def crop_faces(image: PilImage, annotations: Annotations) -> PilImage:
    mask = Image.new("L", image.size, 0x00)
    draw = ImageDraw.Draw(mask)
    for face in annotations.face_annotations:
        draw.ellipse(face_crop_box(face), fill=0xFF)
    image.putalpha(mask)
    return image

def face_crop_box(face_annotation) -> tuple[int, int, int, int]:
    v = face_annotation.bounding_poly.vertices
    x1, y1, x2, y2 = v[0].x, v[0].y, v[2].x + 1, v[2].y + 1
    w, h = x2 - x1, y2 - y1
    hx, hy = x1 + w / 2, y1 + h / 2
    m = max(w, h) * (100 + CROP_MARGIN_PERCENT) / 100 / 2
    return int(hx - m + 0.5), int(hy - m + 0.5), int(hx + m + 1.5), int(hy + m + 1.5)

🍒 Cherry on Py 🐍

Now, the icing on the cake (or the "cherry on the pie" as we say in French):

Having independent rendering functions lets you combine multiple options at once.
Knowing the bounding box for all faces allows cropping the image to the minimal bounding box.
Using the location of the nose and the mouth, you can add a moustache to everyone.
If your functions have parameters to render a single frame, you can generate animations with a few lines of code.
Once your Flask app works locally, you can deploy and keep it available 24/7 at minimal cost.

Here is what's detected on famous photorealistic paintings:

American Gothic

Girl with a Pearl Earring

Shakespeare

Here are some animated versions:

American Gothic

Girl with a Pearl Earring

Shakespeare

And, of course, this works even better on real pictures:

Personal pictures (aged from 2 to 44)

Yes, I've had a moustache for over 42 years, and my sister too ;)

And, finally, here is our famous anonymous from the beginning:

Mona Lisa

🚀 Source code and deployment

The Python source for the backend takes less than 300 lines of code.
You can deploy this demo in 4 minutes.
See "Deploying from scratch".

🎉 Online demo

Try the demo by yourself:

➡️ https://face-detection.lolo.dev ⬅️

Here is a preview:

🖖 See you

Feedback or questions? I'd love to read from you! Follow me on Twitter for more…

⏳ Updates

2021-10-08: Updated with latest library versions + Python 3.7 → 3.9

🎞️ Video object tracking as a service 🐍

Laurent Picard — Thu, 25 Jun 2020 17:58:35 +0000

👋 Hello!

In this article, following up on the previous one ("Video summary as a service"), you'll see the following:

how to track objects present in a video,
with an automated processing pipeline,
in less than 300 lines of Python code.

Here is an example of an auto-generated object summary for the video :

🛠️ Tools

A few tools will do:

Storage space for videos and results
A serverless solution to run the code
A machine learning model to analyze videos
A library to extract frames from videos
A library to render the objects

🧱 Architecture

Here is a possible architecture using 3 Google Cloud services (Cloud Storage, Cloud Functions, and the Video Intelligence API):

The processing pipeline follows these steps:

You upload a video
The upload event automatically triggers the tracking function
The function sends a request to the Video Intelligence API
The Video Intelligence API analyzes the video and uploads the results (annotations)
The upload event triggers the rendering function
The function downloads both annotation and video files
The function renders and uploads the objects
You know which objects are present in your video!

🐍 Python libraries

Video Intelligence API

To analyze videos
https://pypi.org/project/google-cloud-videointelligence

Cloud Storage

To manage downloads and uploads
https://pypi.org/project/google-cloud-storage

OpenCV

To extract video frames
OpenCV offers a headless version (without GUI features, ideal for a service)
https://pypi.org/project/opencv-python-headless

Pillow

To render and annotate object images
Pillow is a very popular imaging library, both extensive and easy to use
https://pypi.org/project/Pillow

🧠 Video analysis

Video Intelligence API

The Video Intelligence API is a pre-trained machine learning model that can analyze videos. One of its multiple features is detecting and tracking objects. For the 1st Cloud Function, here is a possible core function calling annotate_video() with the OBJECT_TRACKING feature:

from google.cloud import storage, videointelligence

def launch_object_tracking(video_uri: str, annot_bucket: str):
    """ Detect and track video objects (asynchronously)

    Results will be stored in <annot_uri> with this naming convention:
    - video_uri: gs://video_bucket/path/to/video.ext
    - annot_uri: gs://annot_bucket/video_bucket/path/to/video.ext.json
    """
    print(f"Launching object tracking for <{video_uri}>...")
    features = [videointelligence.Feature.OBJECT_TRACKING]
    video_blob = storage.Blob.from_string(video_uri)
    video_bucket = video_blob.bucket.name
    path_to_video = video_blob.name
    annot_uri = f"gs://{annot_bucket}/{video_bucket}/{path_to_video}.json"
    request = dict(features=features, input_uri=video_uri, output_uri=annot_uri)

    video_client = videointelligence.VideoIntelligenceServiceClient()
    video_client.annotate_video(request)

Cloud Function entry point

import os

ANNOTATION_BUCKET = os.getenv("ANNOTATION_BUCKET", "")
assert ANNOTATION_BUCKET, "Undefined ANNOTATION_BUCKET environment variable"

def gcf_track_objects(data, context):
    """Cloud Function triggered by a new Cloud Storage object"""
    video_bucket = data["bucket"]
    path_to_video = data["name"]
    video_uri = f"gs://{video_bucket}/{path_to_video}"
    launch_object_tracking(video_uri, ANNOTATION_BUCKET)

Notes:

This function will be called when a video is uploaded to the bucket defined as a trigger.

Using an environment variable makes the code more portable and lets you deploy the exact same code with different trigger and output buckets.

🎨 Object rendering

Code structure

It's interesting to split the code into 2 main classes:

StorageHelper for managing local files and cloud storage objects
VideoProcessor for all graphical processings

Here is a possible core function for the 2nd Cloud Function:

class VideoProcessor:
    @staticmethod
    def render_objects(annot_uri: str, output_bucket: str):
        """Render objects from video annotations"""
        with StorageHelper(annot_uri, output_bucket) as storage:
            with VideoProcessor(storage) as video_proc:
                print(f"Objects to render: {video_proc.object_count}")
                video_proc.render_object_summary()

Cloud Function entry point

def gcf_render_objects(data, context):
    """Cloud Function triggered by a new Cloud Storage object"""
    annotation_bucket = data["bucket"]
    path_to_annotation = data["name"]
    annot_uri = f"gs://{annotation_bucket}/{path_to_annotation}"
    VideoProcessor.render_objects(annot_uri, OBJECT_BUCKET)

Note: This function will be called when an annotation file is uploaded to the bucket defined as a trigger.

Frame rendering

OpenCV and Pillow easily let you extract video frames and compose over them:

class VideoProcessor:
    def get_frame_with_overlay(
        self, obj: vi.ObjectTrackingAnnotation, frame: vi.ObjectTrackingFrame
    ) -> PilImage:
        def get_video_frame() -> PilImage:
            pos_ms = frame.time_offset.total_seconds() * 1000
            self.video.set(cv.CAP_PROP_POS_MSEC, pos_ms)
            _, cv_frame = self.video.read()
            image = Image.fromarray(cv.cvtColor(cv_frame, cv.COLOR_BGR2RGB))
            image.thumbnail(self.cell_size)  # Makes it smaller if needed
            return image

        def add_caption(image: PilImage) -> PilImage:
            ...

        def add_bounding_box(image: PilImage) -> PilImage:
            ...
            draw.rectangle(r, outline=BBOX_COLOR, width=BBOX_WIDTH_PX)
            return image

        return add_bounding_box(add_caption(get_video_frame()))

Note: It would probably be possible to only use OpenCV but I found it more productive developing with Pillow (code is more readable and intuitive).

🔎 Results

Here are the main objects found in the video :

Notes:

The machine learning model has correctly identified different wildlife species: those are "true positives". It has also incorrectly identified our planet as "packaged goods": this is a "false positive". Machine learning models keep learning by being trained with new samples so, with time, their precision keeps increasing (resulting in less false positives).

The current code filters out objects detected with a confidence below 70% or with less than 10 frames. Lower the thresholds to get more results.

🍒 Cherry on Py 🐍

Now, the icing on the cake (or the "cherry on the pie" as we say in French), you can enrich the architecture to add new possibilities:

Trigger the processing for videos from any bucket (including external public buckets)
Generate individual object animations (in parallel to object summaries)

Architecture (v2)

A - Video object tracking can also be triggered manually with an HTTP GET request
B - The same rendering code is deployed in 2 sibling functions, differentiated with an environment variable
C - Object summaries and animations are generated in parallel

Cloud Function HTTP entry point

def gcf_track_objects_http(request):
    """Cloud Function triggered by an HTTP GET request"""
    if request.method != "GET":
        return ("Please use a GET request", 403)
    if not request.args or "video_uri" not in request.args:
        return ('Please specify a "video_uri" parameter', 400)
    video_uri = request.args["video_uri"]
    launch_object_tracking(video_uri, ANNOTATION_BUCKET)
    return f"Launched object tracking for <{video_uri}>"

Note: This is the same code as gcf_track_objects() with the video URI parameter specified by the caller through a GET request.

🎉 Results

Here are some auto-generated trackings for the video :

The left elephant (a big object ;) is detected:
The right elephant is perfectly isolated too:
The veterinarian is correctly identified:
The animal he's feeding too:

Moving objects or static objects in moving shots are tracked too, as in :

A building in a moving shot:
Neighbor buildings are tracked too:
Persons in a moving shot:
A surfer crossing the shot:

Here are some others for the video :

A butterfly (easy?):
An insect, in larval stage, climbing a moving twig:
An ape in a tree far away (hard?):
A monkey jumping from the top of a tree (harder?):
Now, a trap! If we can be fooled, current machine learning state of the art can too:

🚀 Source code and deployment

The Python source code is less than 300 lines.
You can deploy this architecture in less than 8 minutes.
See "Deploying from scratch".

🖖 See you

Do you want more, do you have questions? I'd love to read your feedback. You can also follow me on Twitter.

⏳ Updates

2021-10-08: Updated with latest library versions + Python 3.7 → 3.9

🎞️ Video summary as a service 🐍

Laurent Picard — Fri, 29 May 2020 17:28:11 +0000

👋 Hello!

Dear developers,

Do you like the adage "a picture is worth a thousand words"? I do!
Let's check if it also works for "a picture is worth a thousand frames".
In this tutorial, you'll see the following:
- How to understand the content of a video in a blink,
- in less than 300 lines of Python (3.9) code.

Here is a visual summary example, generated from a 2'42" video made of 35 sequences (also known as video shots):

Note: The summary is a grid where each cell is a frame representing a video shot.

🔭 Objectives

This tutorial has 2 objectives, 1 practical and 1 technical:

Automatically generate visual summaries of videos
Build a processing pipeline with these properties:
- managed (always ready and easy to set up)
- scalable (able to ingest several videos in parallel)
- not costing anything when not used

🛠️ Tools

A few tools are enough:

Storage space for videos and results
A serverless solution to run the code
A machine learning model to analyze videos
A library to extract frames from videos
A library to generate the visual summaries

🧱 Architecture

Here is a possible architecture using 3 Google Cloud services (Cloud Storage, Cloud Functions, and Video Intelligence API):

The processing pipeline follows these steps:

You upload a video to the 1st bucket (a bucket is a storage space in the cloud)
The upload event automatically triggers the 1st function
The function sends a request to the Video Intelligence API to detect the shots
The Video Intelligence API analyzes the video and uploads the results (annotations) to the 2nd bucket
The upload event triggers the 2nd function
The function downloads both annotation and video files
The function renders and uploads the summary to the 3rd bucket
The video summary is ready!

🐍 Python libraries

Open source client libraries let you interface with Google Cloud services in idiomatic Python. You'll use the following:

Cloud Storage
- To manage downloads and uploads
- https://pypi.org/project/google-cloud-storage
Video Intelligence API
- To analyze videos
- https://pypi.org/project/google-cloud-videointelligence

Here is a choice of 2 additional Python libraries for the graphical needs:

OpenCV
- To extract video frames
- There's even a headless version (without GUI features), which is ideal for a service
- https://pypi.org/project/opencv-python-headless
Pillow
- To generate the visual summaries
- Pillow is a very popular imaging library, both extensive and easy to use
- https://pypi.org/project/Pillow

⚙️ Project setup

Assuming you have a Google Cloud account, you can set up the architecture from Cloud Shell with the gcloud and gsutil commands. This lets you script everything from scratch in a reproducible way.

Environment variables

# Project
PROJECT_NAME="Visual Summary"
PROJECT_ID="visual-summary-REPLACE_WITH_UNIQUE_SUFFIX"
# Cloud Storage region (https://cloud.google.com/storage/docs/locations)
GCS_REGION="europe-west1"
# Cloud Functions region (https://cloud.google.com/functions/docs/locations)
GCF_REGION="europe-west1"
# Source
GIT_REPO="cherry-on-py"
PROJECT_SRC=~/$PROJECT_ID/$GIT_REPO/gcf_video_summary

# Cloud Storage buckets (environment variables)
export VIDEO_BUCKET="b1-videos_${PROJECT_ID}"
export ANNOTATION_BUCKET="b2-annotations_${PROJECT_ID}"
export SUMMARY_BUCKET="b3-summaries_${PROJECT_ID}"

Note: You can use your GitHub username as a unique suffix.

New project

gcloud projects create $PROJECT_ID \
  --name="$PROJECT_NAME" \
  --set-as-default

Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/PROJECT_ID].
Waiting for [operations/cp...] to finish...done.
Enabling service [cloudapis.googleapis.com] on project [PROJECT_ID]...
Operation "operations/acf..." finished successfully.
Updated property [core/project] to [PROJECT_ID].

Billing account

# Link project with billing account (single account)
BILLING_ACCOUNT=$(gcloud beta billing accounts list \
    --format 'value(name)')
# Link project with billing account (specific one among multiple accounts)
BILLING_ACCOUNT=$(gcloud beta billing accounts list \
    --format 'value(name)' \
    --filter "displayName='My Billing Account'")

gcloud beta billing projects link $PROJECT_ID --billing-account $BILLING_ACCOUNT

billingAccountName: billingAccounts/XXXXXX-YYYYYY-ZZZZZZ
billingEnabled: true
name: projects/PROJECT_ID/billingInfo
projectId: PROJECT_ID

Buckets

# Create buckets with uniform bucket-level access
gsutil mb -b on -c regional -l $GCS_REGION gs://$VIDEO_BUCKET
gsutil mb -b on -c regional -l $GCS_REGION gs://$ANNOTATION_BUCKET
gsutil mb -b on -c regional -l $GCS_REGION gs://$SUMMARY_BUCKET

Creating gs://VIDEO_BUCKET/...
Creating gs://ANNOTATION_BUCKET/...
Creating gs://SUMMARY_BUCKET/...

You can check how it looks like in the Cloud Console:

Service account

Create a service account. This is for development purposes only (not needed for production). This provides you with credentials to run your code locally.

mkdir ~/$PROJECT_ID
cd ~/$PROJECT_ID

SERVICE_ACCOUNT_NAME="dev-service-account"
SERVICE_ACCOUNT="${SERVICE_ACCOUNT_NAME}@${PROJECT_ID}.iam.gserviceaccount.com"
gcloud iam service-accounts create $SERVICE_ACCOUNT_NAME
gcloud iam service-accounts keys create ~/$PROJECT_ID/key.json --iam-account $SERVICE_ACCOUNT

Created service account [SERVICE_ACCOUNT_NAME].
created key [...] of type [json] as [~/PROJECT_ID/key.json] for [SERVICE_ACCOUNT]

Set the GOOGLE_APPLICATION_CREDENTIALS environment variable and check that it points to the service account key. When you run the application code in the current shell session, client libraries will use these credentials for authentication. If you open a new shell session, set the variable again.

export GOOGLE_APPLICATION_CREDENTIALS=~/$PROJECT_ID/key.json
cat $GOOGLE_APPLICATION_CREDENTIALS

{
  "type": "service_account",
  "project_id": "PROJECT_ID",
  "private_key_id": "...",
  "private_key": "-----BEGIN PRIVATE KEY-----\n...",
  "client_email": "SERVICE_ACCOUNT",
  ...
}

Authorize the service account to access the buckets:

IAM_BINDING="serviceAccount:${SERVICE_ACCOUNT}:roles/storage.objectAdmin"
gsutil iam ch $IAM_BINDING gs://$VIDEO_BUCKET
gsutil iam ch $IAM_BINDING gs://$ANNOTATION_BUCKET
gsutil iam ch $IAM_BINDING gs://$SUMMARY_BUCKET

APIs

A few APIs are enabled by default:

gcloud services list

NAME                              TITLE
bigquery.googleapis.com           BigQuery API
bigquerystorage.googleapis.com    BigQuery Storage API
cloudapis.googleapis.com          Google Cloud APIs
clouddebugger.googleapis.com      Cloud Debugger API
cloudtrace.googleapis.com         Cloud Trace API
datastore.googleapis.com          Cloud Datastore API
logging.googleapis.com            Cloud Logging API
monitoring.googleapis.com         Cloud Monitoring API
servicemanagement.googleapis.com  Service Management API
serviceusage.googleapis.com       Service Usage API
sql-component.googleapis.com      Cloud SQL
storage-api.googleapis.com        Google Cloud Storage JSON API
storage-component.googleapis.com  Cloud Storage

Enable the Video Intelligence, Cloud Functions, and Cloud Build APIs:

gcloud services enable \
  videointelligence.googleapis.com \
  cloudfunctions.googleapis.com \
  cloudbuild.googleapis.com

Operation "operations/acf..." finished successfully.

Note: Cloud Build generates container images for Cloud Functions upon deployment.

Source code

Retrieve the source code:

cd ~/$PROJECT_ID
git clone https://github.com/PicardParis/$GIT_REPO.git

Cloning into 'GIT_REPO'...
...

🧠 Video analysis

Video shot detection

The Video Intelligence API is a pre-trained machine learning model that can analyze videos. One of the multiple features is video shot detection. For the 1st Cloud Function, here is a possible core function calling annotate_video() with the SHOT_CHANGE_DETECTION feature:

from google.cloud import storage, videointelligence

def launch_shot_detection(video_uri: str, annot_bucket: str):
    """Detect video shots (asynchronous operation)

    Results will be stored in <annot_uri> with this naming convention:
    - video_uri: gs://video_bucket/path/to/video.ext
    - annot_uri: gs://annot_bucket/video_bucket/path/to/video.ext.json
    """
    print(f"Launching shot detection for <{video_uri}>...")
    features = [videointelligence.Feature.SHOT_CHANGE_DETECTION]
    video_blob = storage.Blob.from_string(video_uri)
    video_bucket = video_blob.bucket.name
    path_to_video = video_blob.name
    annot_uri = f"gs://{annot_bucket}/{video_bucket}/{path_to_video}.json"
    request = dict(features=features, input_uri=video_uri, output_uri=annot_uri)

    video_client = videointelligence.VideoIntelligenceServiceClient()
    video_client.annotate_video(request)

Local development and tests

Before deploying the function, you need to develop and test it. Create a Python 3 virtual environment and activate it:

cd ~/$PROJECT_ID
python3 -m venv venv
source venv/bin/activate

Install the dependencies:

pip install -r $PROJECT_SRC/gcf1_detect_shots/requirements.txt

Check the dependencies:

pip list

Package                        Version
------------------------------ ----------
...
google-cloud-storage           1.42.3
google-cloud-videointelligence 2.3.3
...

You can use the main scope to test the function in script mode:

import os

ANNOTATION_BUCKET = os.getenv("ANNOTATION_BUCKET", "")
assert ANNOTATION_BUCKET, "Undefined ANNOTATION_BUCKET environment variable"

if __name__ == "__main__":
    # Local tests only (service account needed)
    import argparse

    parser = argparse.ArgumentParser()
    parser.add_argument(
        "video_uri", type=str, help="gs://video_bucket/path/to/video.ext"
    )
    args = parser.parse_args()
    launch_shot_detection(args.video_uri, ANNOTATION_BUCKET)

Note: You have already exported the ANNOTATION_BUCKET environment variable earlier in the shell session; you will also use it later at deployment stage. This makes the code generic and lets you reuse it independently of the output bucket.

Test the function:

VIDEO_PATH="cloud-samples-data/video/gbikes_dinosaur.mp4"
VIDEO_URI="gs://$VIDEO_PATH"
python $PROJECT_SRC/gcf1_detect_shots/main.py $VIDEO_URI

Launching shot detection for <gs://cloud-samples-data/video/gbikes_dinosaur.mp4>...

Note: The test video <gbikes_dinosaur.mp4> is located in an external bucket. This works because the video is publicly accessible.

Wait a moment and check that the annotations have been generated:

gsutil ls -r gs://$ANNOTATION_BUCKET

964  YYYY-MM-DDThh:mm:ssZ  gs://ANNOTATION_BUCKET/VIDEO_PATH.json
TOTAL: 1 objects, 964 bytes (964 B)

Check the last 200 bytes of the annotation file:

gsutil cat -r -200 gs://$ANNOTATION_BUCKET/$VIDEO_PATH.json

}
    }, {
      "start_time_offset": {
        "seconds": 28,
        "nanos": 166666000
      },
      "end_time_offset": {
        "seconds": 42,
        "nanos": 766666000
      }
    } ]
  } ]
}

Note: Those are the start and end positions of the last video shot. Everything seems fine.

Clean up when you're finished:

gsutil rm gs://$ANNOTATION_BUCKET/$VIDEO_PATH.json

deactivate

rm -rf venv

Function entry point

def gcf_detect_shots(data, context):
    """Cloud Function triggered by a new Cloud Storage object"""
    video_bucket = data["bucket"]
    path_to_video = data["name"]
    video_uri = f"gs://{video_bucket}/{path_to_video}"
    launch_shot_detection(video_uri, ANNOTATION_BUCKET)

Note: This function will be called whenever a video is uploaded to the bucket defined as a trigger.

Function deployment

Deploy the 1st function:

GCF_NAME="gcf1_detect_shots"
GCF_SOURCE="$PROJECT_SRC/gcf1_detect_shots"
GCF_ENTRY_POINT="gcf_detect_shots"
GCF_TRIGGER_BUCKET="$VIDEO_BUCKET"
GCF_ENV_VARS="ANNOTATION_BUCKET=$ANNOTATION_BUCKET"
GCF_MEMORY="128MB"

gcloud functions deploy $GCF_NAME \
  --runtime python39 \
  --source $GCF_SOURCE \
  --entry-point $GCF_ENTRY_POINT \
  --update-env-vars $GCF_ENV_VARS \
  --trigger-bucket $GCF_TRIGGER_BUCKET \
  --region $GCF_REGION \
  --memory $GCF_MEMORY \
  --quiet

Note: The default memory allocated for a Cloud Function is 256 MB (possible values are 128MB, 256MB, 512MB, 1024MB, and 2048MB). As the function has no memory or CPU needs (it sends a simple API request), the minimum memory setting is enough.

Deploying function (may take a while - up to 2 minutes)...done.
availableMemoryMb: 128
entryPoint: gcf_detect_shots
environmentVariables:
  ANNOTATION_BUCKET: b2-annotations...
eventTrigger:
  eventType: google.storage.object.finalize
...
status: ACTIVE
timeout: 60s
updateTime: 'YYYY-MM-DDThh:mm:ss.mmmZ'
versionId: '1'

Note: The ANNOTATION_BUCKET environment variable is defined with the --update-env-vars flag. Using an environment variable lets you deploy the exact same code with different trigger and output buckets.

Here is how it looks like in the Cloud Console:

Production tests

Make sure to test the function in production. Copy a video into the video bucket:

VIDEO_NAME="gbikes_dinosaur.mp4"
SRC_URI="gs://cloud-samples-data/video/$VIDEO_NAME"
DST_URI="gs://$VIDEO_BUCKET/$VIDEO_NAME"

gsutil cp $SRC_URI $DST_URI

Copying gs://cloud-samples-data/video/gbikes_dinosaur.mp4 [Content-Type=video/mp4]...
- [1 files][ 62.0 MiB/ 62.0 MiB]
Operation completed over 1 objects/62.0 MiB.

Query the logs to check that the function has been triggered:

gcloud functions logs read --region $GCF_REGION

LEVEL  NAME               EXECUTION_ID  TIME_UTC  LOG
D      gcf1_detect_shots  ...           ...       Function execution started
I      gcf1_detect_shots  ...           ...       Launching shot detection for <gs://VIDEO_BUCKET/VIDEO_NAME>...
D      gcf1_detect_shots  ...           ...       Function execution took 874 ms, finished with status: 'ok'

Wait a moment and check the annotation bucket:

gsutil ls -r gs://$ANNOTATION_BUCKET

You should see the annotation file:

gs://ANNOTATION_BUCKET/VIDEO_BUCKET/:
gs://ANNOTATION_BUCKET/VIDEO_BUCKET/VIDEO_NAME.json

The 1st function is operational!

🎞️ Visual Summary

Code structure

It's interesting to split the code into 2 main classes:

StorageHelper for local file and cloud storage object management
VideoProcessor for graphical processings

Here is a possible core function:

class VideoProcessor:
    @staticmethod
    def generate_summary(annot_uri: str, output_bucket: str):
        """ Generate a video summary from video shot annotations """
        try:
            with StorageHelper(annot_uri, output_bucket) as storage:
                with VideoProcessor(storage) as video_proc:
                    print("Generating summary...")
                    image = video_proc.render_summary()
                    video_proc.upload_summary_as_jpeg(image)
        except Exception:
            logging.exception("Could not generate summary from <%s>", annot_uri)

Note: If exceptions are raised, it's handy to log them with logging.exception() to get a stack trace in production logs.

Class `StorageHelper`

The class manages the following:

The retrieval and parsing of video shot annotations
The download of source videos
The upload of generated visual summaries
File names

class StorageHelper:
    """Local+Cloud storage helper

    - Uses a temp dir for local processing (e.g. video frame extraction)
    - Paths are relative to this temp dir (named after the output bucket)

    Naming convention:
    - video_uri:                 gs://video_bucket/path/to/video.ext
    - annot_uri:    gs://annot_bucket/video_bucket/path/to/video.ext.json
    - video_path:                     video_bucket/path/to/video.ext
    - summary_path:                   video_bucket/path/to/video.ext.SUFFIX
    - summary_uri: gs://output_bucket/video_bucket/path/to/video.ext.SUFFIX
    """

    client = storage.Client()
    video_shots: list[VideoShot]
    video_path: Path
    video_local_path: Path
    upload_bucket: storage.Bucket

    def __init__(self, annot_uri: str, output_bucket: str):
        if not annot_uri.endswith(ANNOT_EXT):
            raise RuntimeError(f"annot_uri must end with <{ANNOT_EXT}>")
        self.video_shots = self.get_video_shots(annot_uri)
        self.video_path = self.video_path_from_uri(annot_uri)
        temp_root = Path(tempfile.gettempdir(), output_bucket)
        temp_root.mkdir(parents=True, exist_ok=True)
        self.video_local_path = temp_root.joinpath(self.video_path)
        self.upload_bucket = self.client.bucket(output_bucket)

The source video is handled in the with statement context manager:

    def __enter__(self):
        self.download_video()
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        self.video_local_path.unlink()

Note: Once downloaded, the video uses memory space in the /tmp RAM disk (the only writable space for the serverless function). It's best to delete temporary files when they're not needed anymore, to avoid potential out-of-memory errors on future invocations of the function.

The video annotations can be retrieved with the methods storage.Blob.download_as_text() and json.loads():

    def get_video_shots(self, annot_uri: str) -> list[VideoShot]:
        json_blob = storage.Blob.from_string(annot_uri, self.client)
        api_response: dict = json.loads(json_blob.download_as_text())
        single_video_results: dict = api_response["annotation_results"][0]
        annotations: list = single_video_results["shot_annotations"]
        return [VideoShot.from_dict(annotation) for annotation in annotations]

The parsing is handled with this VideoShot helper class:

class VideoShot(NamedTuple):
    """Video shot start/end positions in nanoseconds"""

    pos1_ns: int
    pos2_ns: int
    NANOS_PER_SECOND = 10 ** 9

    @classmethod
    def from_dict(cls, annotation: dict) -> "VideoShot":
        def time_offset_in_ns(time_offset) -> int:
            seconds: int = time_offset.get("seconds", 0)
            nanos: int = time_offset.get("nanos", 0)
            return seconds * cls.NANOS_PER_SECOND + nanos

        pos1_ns = time_offset_in_ns(annotation["start_time_offset"])
        pos2_ns = time_offset_in_ns(annotation["end_time_offset"])
        return cls(pos1_ns, pos2_ns)

The naming convention was chosen to keep consistent object paths between the different buckets. This also lets you deduce the video path from the annotation URI:

    def video_path_from_uri(self, annot_uri: str) -> Path:
        annot_blob = storage.Blob.from_string(annot_uri)
        return Path(annot_blob.name[: -len(ANNOT_EXT)])

The video is directly downloaded with storage.Blob.download_to_filename():

    def download_video(self):
        video_uri = f"gs://{self.video_path.as_posix()}"
        blob = storage.Blob.from_string(video_uri, self.client)
        print(f"Downloading -> {self.video_local_path}")
        self.video_local_path.parent.mkdir(parents=True, exist_ok=True)
        blob.download_to_filename(self.video_local_path)

On the opposite, results can be uploaded with storage.Blob.upload_from_string():

    def upload_summary(self, image_bytes: bytes, image_type: str):
        path = self.summary_path(image_type)
        blob = self.upload_bucket.blob(path.as_posix())
        content_type = f"image/{image_type}"
        print(f"Uploading -> {blob.name}")
        blob.upload_from_string(image_bytes, content_type)

Note: Pillow supports working with memory images, which avoids having to manage local files.

And finally, here is a possible naming convention for the summary files:

    def summary_path(self, image_type: str) -> Path:
        video_name = self.video_path.name
        shot_count = len(self.video_shots)
        suffix = f"summary{shot_count:03d}.{image_type}"
        summary_name = f"{video_name}.{suffix}"
        return Path(self.video_path.parent, summary_name)

Class `VideoProcessor`

The class manages the following:

Video frame extraction
Visual summary generation

import cv2 as cv
from PIL import Image

from storage_helper import StorageHelper

PilImage = Image.Image
ImageSize = NamedTuple("ImageSize", [("w", int), ("h", int)])


class VideoProcessor:
    storage: StorageHelper
    video: cv.VideoCapture
    cell_size: ImageSize
    grid_size: ImageSize

    def __init__(self, storage: StorageHelper):
        self.storage = storage

Opening and closing the video is handled in the with statement context manager:

    def __enter__(self):
        video_path = self.storage.video_local_path
        self.video = cv.VideoCapture(str(video_path))
        if not self.video.isOpened():
            raise RuntimeError(f"Could not open video <{video_path}>")
        self.compute_grid_dimensions()
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        self.video.release()

The video summary is a grid of cells which can be rendered in a single loop with two generators:

    def render_summary(self, shot_ratio: float = 0.5) -> PilImage:
        grid_img = Image.new("RGB", self.grid_size, RGB_BACKGROUND)

        img_and_pos_iter = zip(self.gen_cell_img(shot_ratio), self.gen_cell_pos())
        for cell_img, cell_pos in img_and_pos_iter:
            cell_img.thumbnail(self.cell_size)  # Makes it smaller if needed
            grid_img.paste(cell_img, cell_pos)

        return grid_img

Note: shot_ratio is set to 0.5 by default to extract video shot middle frames.

The first generator yields cell images:

    def gen_cell_img(self, shot_ratio: float) -> Iterator[PilImage]:
        assert 0.0 <= shot_ratio <= 1.0
        MS_IN_NS = 10 ** 6
        for video_shot in self.storage.video_shots:
            pos1_ns, pos2_ns = video_shot
            pos_ms = (pos1_ns + shot_ratio * (pos2_ns - pos1_ns)) / MS_IN_NS
            yield self.frame_at_position(pos_ms)

The second generator yields cell positions:

    def gen_cell_pos(self) -> Iterator[tuple[int, int]]:
        cell_x, cell_y = 0, 0
        while True:
            yield cell_x, cell_y
            cell_x += self.cell_size.w
            if self.grid_size.w <= cell_x:  # Move to next row?
                cell_x, cell_y = 0, cell_y + self.cell_size.h

OpenCV easily allows extracting video frames at a given position:

    def frame_at_position(self, pos_ms: float) -> PilImage:
        self.video.set(cv.CAP_PROP_POS_MSEC, pos_ms)
        _, cv_frame = self.video.read()
        return Image.fromarray(cv.cvtColor(cv_frame, cv.COLOR_BGR2RGB))

Choosing the summary grid composition is arbitrary. Here is an example to compose a summary preserving the video proportions:

    def compute_grid_dimensions(self):
        shot_count = len(self.storage.video_shots)
        if shot_count < 1:
            raise RuntimeError(f"Expected 1+ video shots (got {shot_count})")
        # Try to preserve the video aspect ratio
        # Consider cells as pixels and try to fit them in a square
        cols = rows = int(shot_count ** 0.5 + 0.5)
        if cols * rows < shot_count:
            cols += 1
        cell_w = int(self.video.get(cv.CAP_PROP_FRAME_WIDTH))
        cell_h = int(self.video.get(cv.CAP_PROP_FRAME_HEIGHT))
        if SUMMARY_MAX_SIZE.w < cell_w * cols:
            scale = SUMMARY_MAX_SIZE.w / (cell_w * cols)
            cell_w = int(scale * cell_w)
            cell_h = int(scale * cell_h)
        self.cell_size = ImageSize(cell_w, cell_h)
        self.grid_size = ImageSize(cell_w * cols, cell_h * rows)

Finally, Pillow gives full control on image serializations:

    def upload_summary_as_jpeg(self, image: PilImage):
        mem_file = BytesIO()
        image_type = "jpeg"
        jpeg_save_parameters = dict(optimize=True, progressive=True)
        image.save(mem_file, format=image_type, **jpeg_save_parameters)

        image_bytes = mem_file.getvalue()
        self.storage.upload_summary(image_bytes, image_type)

Note: Working with in-memory images avoids managing local files and uses less memory.

Local development and tests

You can use the main scope to test the function in script mode:

import os

from video_processor import VideoProcessor

SUMMARY_BUCKET = os.getenv("SUMMARY_BUCKET", "")
assert SUMMARY_BUCKET, "Undefined SUMMARY_BUCKET environment variable"

if __name__ == "__main__":
    # Local tests only (service account needed)
    import argparse

    parser = argparse.ArgumentParser()
    parser.add_argument(
        "annot_uri", type=str, help="gs://annotation_bucket/path/to/video.ext.json"
    )
    args = parser.parse_args()
    VideoProcessor.generate_summary(args.annot_uri, SUMMARY_BUCKET)

Test the function:

cd ~/$PROJECT_ID
python3 -m venv venv
source venv/bin/activate

pip install -r $PROJECT_SRC/gcf2_generate_summary/requirements.txt

VIDEO_NAME="gbikes_dinosaur.mp4"
ANNOTATION_URI="gs://$ANNOTATION_BUCKET/$VIDEO_BUCKET/$VIDEO_NAME.json"

python $PROJECT_SRC/gcf2_generate_summary/main.py $ANNOTATION_URI

Downloading -> /tmp/SUMMARY_BUCKET/VIDEO_BUCKET/VIDEO_NAME
Generating summary...
Uploading -> VIDEO_BUCKET/VIDEO_NAME.summary004.jpeg

Note: The uploaded video summary shows 4 shots.

Clean up:

deactivate
rm -rf venv

Function entry point

def gcf_generate_summary(data, context):
    """Cloud Function triggered by a new Cloud Storage object"""
    annotation_bucket = data["bucket"]
    path_to_annotation = data["name"]
    annot_uri = f"gs://{annotation_bucket}/{path_to_annotation}"
    VideoProcessor.generate_summary(annot_uri, SUMMARY_BUCKET)

Note: This function will be called whenever an annotation file is uploaded to the bucket defined as a trigger.

Function deployment

GCF_NAME="gcf2_generate_summary"
GCF_SOURCE="$PROJECT_SRC/gcf2_generate_summary"
GCF_ENTRY_POINT="gcf_generate_summary"
GCF_TRIGGER_BUCKET="$ANNOTATION_BUCKET"
GCF_ENV_VARS="SUMMARY_BUCKET=$SUMMARY_BUCKET"
GCF_TIMEOUT="540s"
GCF_MEMORY="512MB"

gcloud functions deploy $GCF_NAME \
  --runtime python39 \
  --source $GCF_SOURCE \
  --entry-point $GCF_ENTRY_POINT \
  --update-env-vars $GCF_ENV_VARS \
  --trigger-bucket $GCF_TRIGGER_BUCKET \
  --region $GCF_REGION \
  --timeout $GCF_TIMEOUT \
  --memory $GCF_MEMORY \
  --quiet

Notes:

The default timeout for a Cloud Function is 60 seconds. As you're deploying a background function with potentially long processings, set it to the maximum value (540 seconds = 9 minutes).
You also need to bump up the memory a little for the video and image processings. Depending on the size of your videos and the maximum resolution of your output summaries, or if you need to generate the summary faster (memory size and vCPU speed are correlated), you might use a higher value (1024MB or 2048MB).

Deploying function (may take a while - up to 2 minutes)...done.
availableMemoryMb: 512
entryPoint: gcf_generate_summary
environmentVariables:
  SUMMARY_BUCKET: b3-summaries...
...
status: ACTIVE
timeout: 540s
updateTime: 'YYYY-MM-DDThh:mm:ss.mmmZ'
versionId: '1'

Here is how it looks like in the Cloud Console:

Production tests

Make sure to test the function in production. You can upload an annotation file in the 2nd bucket:

VIDEO_NAME="gbikes_dinosaur.mp4"
ANNOTATION_FILE="$VIDEO_NAME.json"
ANNOTATION_URI="gs://$ANNOTATION_BUCKET/$VIDEO_BUCKET/$ANNOTATION_FILE"
gsutil cp $ANNOTATION_URI .
gsutil cp $ANNOTATION_FILE $ANNOTATION_URI
rm $ANNOTATION_FILE

Note: This reuses the previous local test annotation file and overwrites it. Overwriting a file in a bucket also triggers attached functions.

Wait a few seconds and query the logs to check that the function has been triggered:

gcloud functions logs read --region $GCF_REGION

LEVEL  NAME                   EXECUTION_ID  TIME_UTC  LOG
...
D      gcf2_generate_summary  ...           ...       Function execution started
I      gcf2_generate_summary  ...           ...       Downloading -> /tmp/SUMMARY_BUCKET/VIDEO_BUCKET/VIDEO_NAME
I      gcf2_generate_summary  ...           ...       Generating summary...
I      gcf2_generate_summary  ...           ...       Uploading -> VIDEO_BUCKET/VIDEO_NAME.summary004.jpeg
D      gcf2_generate_summary  ...           ...       Function execution took 11591 ms, finished with status: 'ok'

The 2nd function is operational and the pipeline is in place! You can now do end-to-end tests by copying new videos in the 1st bucket.

Results

Download the generated summary on your computer:

cd ~/$PROJECT_ID
gsutil cp -r gs://$SUMMARY_BUCKET/**.jpeg .
cloudshell download *.jpeg

Here is the visual summary for gbikes_dinosaur.mp4 (4 detected shots):

You can also directly preview the file from the Cloud Console:

🍒 Cherry on the Py 🐍

Now, the icing on the cake (or the "cherry on the pie" as we say in French)...

Based on the same architecture and code, you can add a few features:
- Trigger the processing for videos from other buckets
- Generate summaries in multiple formats (such as JPEG, PNG, WEBP)
- Generate animated summaries (also in multiple formats, such as GIF, PNG, WEBP)
Enrich the architecture to duplicate 2 items:
- The video shot detection function, to get it to run as an HTTP endpoint
- The summary generation function to handle animated images
Adapt the code to support the new features:
- An animated parameter to generate still or animated summaries
- Save and upload the results in multiple formats

Architecture (v2)

A. Video shot detection can also be triggered manually with an HTTP GET request
B. Still and animated summaries are generated in 2 functions in parallel
C. Summaries are uploaded in multiple image formats

HTTP entry point

def gcf_detect_shots_http(request):
    """Cloud Function triggered by an HTTP GET request"""
    if request.method != "GET":
        return ("Please use a GET request", 403)
    if not request.args or "video_uri" not in request.args:
        return ('Please specify a "video_uri" parameter', 400)
    video_uri = request.args["video_uri"]
    launch_shot_detection(video_uri, ANNOTATION_BUCKET)
    return f"Launched shot detection for <{video_uri}>"

Note: This is the same code as gcf_detect_shots with the video URI parameter provided from a GET request.

Function deployment

GCF_NAME="gcf1_detect_shots_http"
GCF_SOURCE="$PROJECT_SRC/gcf1_detect_shots"
GCF_ENTRY_POINT="gcf_detect_shots_http"
GCF_TRIGGER_BUCKET="$VIDEO_BUCKET"
GCF_ENV_VARS="ANNOTATION_BUCKET=$ANNOTATION_BUCKET"
GCF_MEMORY="128MB"

gcloud functions deploy $GCF_NAME \
  --runtime python39 \
  --source $GCF_SOURCE \
  --entry-point $GCF_ENTRY_POINT \
  --update-env-vars $GCF_ENV_VARS \
  --trigger-http \
  --region $GCF_REGION \
  --memory $GCF_MEMORY \
  --quiet

Here is how it looks like in the Cloud Console:

Animation support

Add an animated option in the core function:

class VideoProcessor:
    @staticmethod
-   def generate_summary(annot_uri: str, output_bucket: str):
+   def generate_summary(annot_uri: str, output_bucket: str, animated=False):
        """ Generate a video summary from video shot annotations """
        try:
            with StorageHelper(annot_uri, output_bucket) as storage:
                with VideoProcessor(storage) as video_proc:
                    print("Generating summary...")
-                   image = video_proc.render_summary()
-                   video_proc.upload_summary_as_jpeg(image)
+                   if animated:
+                       video_proc.generate_summary_animations()
+                   else:
+                       video_proc.generate_summary_stills()
        except Exception:
            logging.exception("Could not generate summary from <%s>", annot_uri)

Define the formats you're interested in generating:

ImageFormat = NamedTuple("ImageFormat", [("type", str), ("save_parameters", dict)])

IMAGE_JPEG = ImageFormat("jpeg", dict(optimize=True, progressive=True))
IMAGE_GIF = ImageFormat("gif", dict(optimize=True))
IMAGE_PNG = ImageFormat("png", dict(optimize=True))
IMAGE_WEBP = ImageFormat("webp", dict(lossless=False, quality=80, method=1))
SUMMARY_STILL_FORMATS = (IMAGE_JPEG, IMAGE_PNG, IMAGE_WEBP)
SUMMARY_ANIMATED_FORMATS = (IMAGE_GIF, IMAGE_PNG, IMAGE_WEBP)

Add support to generate still and animated summaries in different formats:

    def generate_summary_stills(self):
        image = self.render_summary()
        for image_format in SUMMARY_STILL_FORMATS:
            self.upload_summary([image], image_format)

    def generate_summary_animations(self):
        frame_count = ANIMATION_FRAMES
        images = []
        for frame_index in range(frame_count):
            shot_ratio = (frame_index + 1) / (frame_count + 1)
            print(f"shot_ratio: {shot_ratio:.0%}")
            image = self.render_summary(shot_ratio)
            images.append(image)
        for image_format in SUMMARY_ANIMATED_FORMATS:
            self.upload_summary(images, image_format)

The serialization can still take place in a single function:

    def upload_summary(self, images: list[PilImage], image_format: ImageFormat):
        if not images:
            raise RuntimeError("Empty image list")
        mem_file = BytesIO()
        image_type = image_format.type
        save_parameters = image_format.save_parameters.copy()
        if animated := 1 < len(images):
            save_parameters |= dict(
                save_all=True,
                append_images=images[1:],
                duration=ANIMATION_FRAME_DURATION_MS,
                loop=0,  # Infinite loop
            )
        images[0].save(mem_file, format=image_type, **save_parameters)

        image_bytes = mem_file.getvalue()
        self.storage.upload_summary(image_bytes, image_type, animated)

Note: Pillow is both versatile and consistent, allowing for significant and clean code factorization.

Add an animated optional parameter to the StorageHelper class:

class StorageHelper:
-    def upload_summary(self, image_bytes: bytes, image_type: str):
-       path = self.summary_path(image_type)
+    def upload_summary(self, image_bytes: bytes, image_type: str, animated=False):
+       path = self.summary_path(image_type, animated)
        blob = self.upload_bucket.blob(path.as_posix())
        content_type = f"image/{image_type}"
        print(f"Uploading -> {blob.name}")
        blob.upload_from_string(image_bytes, content_type)

-   def summary_path(self, image_type: str) -> Path:
+   def summary_path(self, image_type: str, animated=False) -> Path:
        video_name = self.video_path.name
        shot_count = self.shot_count()
-       suffix = f"summary{shot_count:03d}.{image_type}"
+       still_or_anim = "anim" if animated else "still"
+       suffix = f"summary{shot_count:03d}_{still_or_anim}.{image_type}"
        summary_name = f'{video_name}.{suffix}'
        return Path(self.video_path.parent, summary_name)

And finally, add an ANIMATED optional environment variable in the entry point:

...
+ANIMATED = os.getenv("ANIMATED", "0") == "1"

def gcf_generate_summary(data, context):
    ...
-   VideoProcessor.generate_summary(annot_uri, SUMMARY_BUCKET)
+   VideoProcessor.generate_summary(annot_uri, SUMMARY_BUCKET, ANIMATED)

if __name__ == '__main__':
    ...
-   VideoProcessor.generate_summary(args.annot_uri, SUMMARY_BUCKET)
+   VideoProcessor.generate_summary(args.annot_uri, SUMMARY_BUCKET, ANIMATED)

Function deployment

Duplicate the 2nd function with the additional ANIMATED environment variable:

GCF_NAME="gcf2_generate_summary_animated"
GCF_SOURCE="$PROJECT_SRC/gcf2_generate_summary"
GCF_ENTRY_POINT="gcf_generate_summary"
GCF_TRIGGER_BUCKET="$ANNOTATION_BUCKET"
GCF_ENV_VARS1="SUMMARY_BUCKET=$SUMMARY_BUCKET"
GCF_ENV_VARS2="ANIMATED=1"
GCF_TIMEOUT="540s"
GCF_MEMORY="2048MB"

gcloud functions deploy $GCF_NAME \
  --runtime python39 \
  --source $GCF_SOURCE \
  --entry-point $GCF_ENTRY_POINT \
  --update-env-vars $GCF_ENV_VARS1 \
  --update-env-vars $GCF_ENV_VARS2 \
  --trigger-bucket $GCF_TRIGGER_BUCKET \
  --region $GCF_REGION \
  --timeout $GCF_TIMEOUT \
  --memory $GCF_MEMORY \
  --quiet

Here is how it looks like in the Cloud Console:

🎉 Final tests

The HTTP endpoint lets you trigger the pipeline with a GET request:

GCF_NAME="gcf1_detect_shots_http"
VIDEO_URI="gs://cloud-samples-data/video/visionapi.mp4"
GCF_URL="https://$GCF_REGION-$PROJECT_ID.cloudfunctions.net/$GCF_NAME?video_uri=$VIDEO_URI"

curl $GCF_URL -H "Authorization: bearer $(gcloud auth print-identity-token)"

Launched shot detection for <VIDEO_URI>

Note: The test video <visionapi.mp4> is located in an external bucket but is publicly accessible.

In addition, copy one or several videos into the video bucket. You can drag and drop videos:

The videos are then processed in parallel. Here are a few logs:

LEVEL NAME                           EXECUTION_ID ... LOG
...
D     gcf2_generate_summary_animated f6n6tslsfwdu ... Function execution took 49293 ms, finished with status: 'ok'
I     gcf2_generate_summary          yd1vqabafn17 ... Uploading -> b1-videos.../JaneGoodall.mp4.summary035_still.png
I     gcf2_generate_summary_animated qv9b03814jjk ... shot_ratio: 43%
I     gcf2_generate_summary          yd1vqabafn17 ... Uploading -> b1-videos.../JaneGoodall.mp4.summary035_still.webp
D     gcf2_generate_summary          yd1vqabafn17 ... Function execution took 54616 ms, finished with status: 'ok'
I     gcf2_generate_summary_animated g4d2wrzxz2st ... shot_ratio: 71%
...
D     gcf2_generate_summary          amwmov1wk0gn ... Function execution took 65256 ms, finished with status: 'ok'
I     gcf2_generate_summary_animated 7pp882fz0x84 ... shot_ratio: 57%
I     gcf2_generate_summary_animated i3u830hsjz4r ... Uploading -> b1-videos.../JaneGoodall.mp4.summary035_anim.png
I     gcf2_generate_summary_animated i3u830hsjz4r ... Uploading -> b1-videos.../JaneGoodall.mp4.summary035_anim.webp
D     gcf2_generate_summary_animated i3u830hsjz4r ... Function execution took 70862 ms, finished with status: 'ok'
...

In the 3rd bucket, you'll find all still and animated summaries:

You've already seen the still summary for <JaneGoodall.mp4> as an introduction to this tutorial. In the animated version, and in only 6 frames, you get an even better idea of what the whole video is about:

If you don't want to keep your project, you can delete it:

gcloud projects delete $PROJECT_ID

➕ One more thing

How big is the code base?

first_line_after_licence=16
find $PROJECT_SRC -name '*.py' -exec tail -n +$first_line_after_licence {} \; | grep -v "^$" | wc -l

Number of Python lines:

Video analysis and processing, with different options, run in less than 300 lines of readable Python.
Less lines, less bugs!
🔥🐍 Mission accomplished! 🐍🔥

🖖 See you

I hope you appreciated this tutorial and would love to read your feedback. You can also follow me on Twitter.

⏳ Updates

2021-10-08: Updated with latest library versions + Python 3.7 → 3.9