DEV Community: Laurent Picard

How to Count Gemini Tokens Locally

Laurent Picard — Thu, 02 Jul 2026 13:50:42 +0000

✨ Overview

This article explores how Gemini tokenizes data and demonstrates how to count or estimate tokens locally. You’ll learn how to use the local tokenizer to estimate text token counts offline, understand the tokenization math for multimodal inputs (images, audio, video, PDFs), and see how to retrieve precise token usage metadata from API responses for accurate tracking and billing.

ℹ️ The complete source code is available in this notebook (including all setup details and future updates) under the Apache 2.0 license. You can also directly open the notebook in Colab. This article reproduces all the results generated by a click on “Run all”.

⚙️ Setup

🐍 Google Gen AI Python SDK

To call the Gemini API, we'll use the Google Gen AI Python SDK. The Gemini API provides a count_tokens method, and the SDK offers an experimental implementation of a LocalTokenizer class.

Make sure you have a recent version of the google-genai package with its local-tokenizer extra:

%pip install --quiet "google-genai[local-tokenizer]>=2.9.0"

🛠️ Google Cloud Project

To get started using the Gemini API on Agent Platform, you must have an existing Google Cloud project and enable the Agent Platform API.

Learn more about setting up a project and a development environment.

import os

# fmt: off
PROJECT_ID = ""  # @param {type: "string", placeholder: "[your-project-id]", isTemplate: true}
LOCATION = "global" # @param {type: "string", placeholder: "[your-region]", isTemplate: true}
# fmt: on

if not PROJECT_ID:
    PROJECT_ID = os.environ.get("GOOGLE_CLOUD_PROJECT")
    assert PROJECT_ID, "❌ Please set the PROJECT_ID variable"
if not LOCATION:
    LOCATION = os.environ.get("GOOGLE_CLOUD_REGION", "global")

🤖 Gen AI SDK Client

To interact with the Gemini API, we initialize a genai.Client. Since we're using the enterprise-ready Agent Platform backend (formerly Vertex AI), we pass enterprise=True along with our Google Cloud project and location:

from google import genai


def print_configuration(client: genai.Client) -> None:
    service = "Agent Platform" if client.vertexai else "Google AI"
    print(f"ℹ️ Using the {service} API", end="")
    if client._api_client.project:
        print(f' with project "{client._api_client.project[:7]}…"', end="")
        print(f' in location "{client._api_client.location}"')
    elif client._api_client.api_key:
        api_key = client._api_client.api_key
        print(f' with API key "{api_key[:5]}…{api_key[-5:]}"', end="")
        print(f" (in case of error, make sure it was created for {service})")


client = genai.Client(enterprise=True, project=PROJECT_ID, location=LOCATION)

print_configuration(client)

ℹ️ Using the Agent Platform API with project "lpdemo-…" in location "global"

🧠 Gemini Model

We'll use gemini-3.1-flash-lite as our default model for token counting and content generation. This lightweight, fast model is ideal for high-throughput tasks.

MODEL_ID = "gemini-3.1-flash-lite"

🧩 The Basics: Tokens and Tokenizers

Tokens

Large language models (LLMs) don't process our inputs directly, nor do they generate the final text or media we see. Instead, they operate on fundamental units called tokens, ingesting them as inputs and generating them as outputs.

Here's what happens when we send an LLM request:

Our inputs are transformed into tokens. In other words, they are tokenized.
The model generates output tokens, which represent the most likely next tokens based on the overall context.
These output tokens are transformed back into the final content we can use.

You can think of a token as a piece of information, and this tokenization process acts as an information compression codec:

Encoding: Input → Input tokens
Decoding: Output tokens → Output

Tokenization is necessary to compress information to the right level of semantic granularity, allowing the model's attention mechanism to focus and develop an understanding of the provided data.

Tokenizers

Gemini is natively multimodal and accepts text, images, audio, video, and PDFs. These media types can be processed by a set of three tokenizers:

Input	Text Tokenizer	Image Tokenizer	Audio Tokenizer	Comment
Text	✅			The original tokenizer type, when LLMs were only chatbots.
Image		✅		An image is can be worth a thousand ~~words~~ tokens!
Audio	✅		✅	Text tokens are used for timestamps (`MM:SS` or `H:MM:SS`).
Video	✅	✅	[✅]	By default, one frame is sampled per second, along with its corresponding timestamp. Audio is optional for videos.
PDF	✅	✅		PDFs are processed by vision tokenizers. Text tokens are used for OCR and pagination data.

As you can see, up to three tokenizers can be involved, depending on the modality.

💡 Keep in mind that not all underlying tokens are necessarily billed. See the usage_metadata section below for examples of tokens actually billed per modality.

Vocabulary

The complete set of unique tokens that an LLM can ingest or generate makes up its vocabulary. Once an LLM is trained, its vocabulary is fixed and is used for inference.

A vocabulary is essentially a lookup table mapping text sequences to token IDs (which correspond to vector representations in a semantic space). This means tokenizers are simply algorithms that use this vocabulary to encode and decode tokens (i.e., to convert data to and from token IDs).

For example, the Gemini text tokenizers process common words like this:

Text	Tokens	Tokenization	Token IDs
`hello`	1	A single token for most common sequences	`23391`
`passion`	1	`passion`	`208039`
`passionate`	2	`pass • ionate`	`4373 • 84242`
`passionné`	2	`passion • né` (passionate in French)	`208039 • 8504`
`passionately`	2	`passion • ately`	`208039 • 2295`
`passionalmente`	2	`pass • ionalmente` (passionately in Italian)	`4373 • 134916`

💡 As you can see, words with the same root aren't necessarily split the same way. Text tokenizers have no concept of syllables, prefixes, or suffixes. They don't think like linguists or grammarians; they think like statisticians and look for statistically optimal combinations.

🌐 Baseline: API Token Counting

The Gemini API lets you count tokens for any multimodal input by sending a count_tokens request. While you need to be authenticated to use it, this method is free of charge, so you can audit your prompts before committing to a paid request. Likewise, the compute_tokens method lets you retrieve the list of corresponding tokens and token IDs.

Let's reproduce the previous table:

from collections.abc import Iterator

import IPython.display

from google.genai.types import (
    ComputeTokensResponse,
    ComputeTokensResult,
    CountTokensResponse,
    CountTokensResult,
)

RowData = tuple[str, str, str, str]


def display_token_info_from_api(model: str, texts: list[str]) -> None:
    def yield_data() -> Iterator[RowData]:
        for text in texts:
            count_result = client.models.count_tokens(model=model, contents=text)
            compute_result = client.models.compute_tokens(model=model, contents=text)
            yield get_text_token_info(text, count_result, compute_result)

    display_token_info(yield_data())


def display_token_info(yield_data: Iterator[RowData]) -> None:
    def yield_row() -> Iterator[RowData]:
        yield "Text", "Tokens", "Tokenization", "Token IDs"
        yield "-", ":-:", "-", "-"
        yield from yield_data

    markdown = "\n".join("| " + " | ".join(row) + " |" for row in yield_row())
    IPython.display.display(IPython.display.Markdown(markdown))


def get_text_token_info(
    text: str,
    count_tokens_res: CountTokensResponse | CountTokensResult,
    compute_tokens_res: ComputeTokensResponse | ComputeTokensResult,
) -> RowData:
    def inline_code(s: str) -> str:
        return f"`{s}`"

    total_tokens = count_tokens_res.total_tokens
    tokens_info = compute_tokens_res.tokens_info
    assert tokens_info is not None and len(tokens_info) == 1
    info = tokens_info[0]
    assert info.tokens is not None and info.token_ids is not None
    tokenization = " • ".join(t.decode("utf-8", errors="replace") for t in info.tokens)
    token_ids = " • ".join(str(token_id) for token_id in info.token_ids)

    return (
        inline_code(text),
        str(total_tokens),
        inline_code(tokenization),
        inline_code(token_ids),
    )


TEXTS = [
    "hello",
    "passion",
    "passionate",
    "passionné",
    "passionately",
    "passionalmente",
]
display_token_info_from_api(MODEL_ID, TEXTS)

Text	Tokens	Tokenization	Token IDs
`hello`	1	`hello`	`23391`
`passion`	1	`passion`	`208039`
`passionate`	2	`pass • ionate`	`4373 • 84242`
`passionné`	2	`passion • né`	`208039 • 8504`
`passionately`	2	`passion • ately`	`208039 • 2295`
`passionalmente`	2	`pass • ionalmente`	`4373 • 134916`

🚀 Why Count Tokens Locally?

Here are a few use cases where counting (or just estimating) tokens locally is useful:

Offline & Speed: You can count tokens completely offline. Plus, even when you're online, doing it locally means you don't have to wait for a network round-trip to the Gemini API just to check your prompt size.
Quotas: While the count_tokens method is free, counting locally saves bandwidth and prevents you from hitting API rate limits, especially during high-volume token counting.
Latency: You can estimate how much time is needed to process your text input before you start receiving a response (for a given model, the time-to-first-token latency is roughly proportional to the number of input tokens).
Cost Control: You can estimate and budget your API costs before committing to a paid request.
Routing: Knowing which token-count bucket your input falls into lets you route requests to different models based on speed, cost, or context size.
Privacy: You can audit the token count of sensitive data without sending it over your network.

🔤 Using the Local Text Tokenizer

Create a local tokenizer for the specific Gemini model you're using:

from google.genai.local_tokenizer import LocalTokenizer

tokenizer = LocalTokenizer(model_name=MODEL_ID)

💡 Remarks

Creating a tokenizer takes a few seconds, during which the configuration and vocabulary are loaded into memory.

On the first call, the tokenizer data is downloaded and stored in a local cache. This step requires an internet connection and about 30MB of storage.

If you want to build a fully offline solution, you can check out the SDK source code and persist the tokenizer assets (e.g., by configuring a persistent cache directory or building a container image).

Checking the internal tokenizer name confirms that the Gemma open-weight models share the same text tokenizer as the Gemini 3 family:

print(f'Text tokenizer name for "{MODEL_ID}": "{tokenizer._tokenizer_name}"')

Text tokenizer name for "gemini-3.1-flash-lite": "gemma4"

Call the count_tokens() method on a small text input:

contents = "Hello World!"
result = tokenizer.count_tokens(contents)

print(f"{result.total_tokens=}")

result.total_tokens=3

Now, let's reproduce the previous API tokenization tests with our local tokenizer:

def display_token_info_from_local_tokenizer(
    tokenizer: LocalTokenizer, texts: list[str]
) -> None:
    def yield_data() -> Iterator[RowData]:
        for text in texts:
            count_result = tokenizer.count_tokens(contents=text)
            compute_result = tokenizer.compute_tokens(contents=text)
            yield get_text_token_info(text, count_result, compute_result)

    display_token_info(yield_data())


display_token_info_from_local_tokenizer(tokenizer, TEXTS)

Text	Tokens	Tokenization	Token IDs
`hello`	1	`hello`	`23391`
`passion`	1	`passion`	`208039`
`passionate`	2	`pass • ionate`	`4373 • 84242`
`passionné`	2	`passion • né`	`208039 • 8504`
`passionately`	2	`passion • ately`	`208039 • 2295`
`passionalmente`	2	`pass • ionalmente`	`4373 • 134916`

💡 As expected, we get exactly the same results, but with 100% local execution this time.

Finally, let's download a longer text, like Hamlet:

import requests


def get_text_from_url(content_url: str, force_encoding: str = "") -> str:
    response = requests.get(content_url, timeout=10)
    response.raise_for_status()
    if force_encoding:  # Use for HTTP headers with unknown/incorrect charset
        response.encoding = force_encoding
    return response.text


TEXT_URL = "https://storage.googleapis.com/dataflow-samples/shakespeare/hamlet.txt"
contents = get_text_from_url(TEXT_URL)

print(contents[:256] + "[…]")

    HAMLET


    DRAMATIS PERSONAE


CLAUDIUS    king of Denmark. (KING CLAUDIUS:)

HAMLET  son to the late, and nephew to the present king.

POLONIUS    lord chamberlain. (LORD POLONIUS:)

HORATIO friend to Hamlet.

LAERTES son to Polonius.

LUCIANUS    nephew to the kin[…]

How many tokens do we need to encode Hamlet?

result = tokenizer.count_tokens(contents)

print(f"{result.total_tokens=:,}")

result.total_tokens=54,660

💡 Hamlet gets broken down locally into 50k+ tokens in a fraction of a second. If you tokenize War and Peace, you'll get 850k+ tokens.

🕵️‍♂️ Accounting for "Hidden" Tokens

When you send a request to Gemini, the total input token count isn't always just the sum of your input data.

To keep things simple, we tested text token counts with default parameters. The count_tokens and compute_tokens methods both have a config parameter. Depending on your request configuration, your inputs and outputs may include additional tokens.

Keep an eye out for these hidden additions:

System Instructions: Any system prompt you set will add to the total token count.
Thinking: If thinking is enabled, an internal chain of thought can generate additional thinking tokens.
Tools and Functions: If you provide a list of tools (like Python execution or custom functions), their declarations, calls, and responses are part of your prompt payload.
Response Schema: Enforcing structured outputs (like JSON) requires the model to process the schema definition you provide, which consumes input tokens.
Chat History: In multi-turn conversations, the entire chat history is sent back to the model with every new message, meaning your input token count grows with each turn.

🧮 Multimodal Token Math

Multimodal inputs (images, audio, video, and documents) aren't tokenized like text. They usually have specific calculation rules based on the model (and its underlying tokenizers), the media type, and the request configuration.

For multimodal inputs, refer to the documentation for details on how token counts are calculated for different media types:

There are generally multiple tokenization options, even for a single modality. You can use the count_tokens method and the calculation rules to estimate the token count of your own payloads. To get a clearer picture, let's look at actual requests and see how token counts are broken down by modality…

🎯 Tracking Actual Token Usage

While estimating token counts is super useful, you should always rely on the usage_metadata returned in the API response when you need to track your actual usage down to the exact token. It's the single source of truth for billing.

Here's the gist of how usage_metadata lets you get the token counts by modality:

class GenerateContentResponse:
    # …
    usage_metadata: Optional[GenerateContentResponseUsageMetadata]
    # …


class GenerateContentResponseUsageMetadata:
    # …
    prompt_token_count: Optional[int]
    prompt_tokens_details: Optional[list[ModalityTokenCount]]
    # …


class ModalityTokenCount:
    modality: Optional[MediaModality]
    token_count: Optional[int]


class MediaModality(StrEnum):
    MODALITY_UNSPECIFIED = "MODALITY_UNSPECIFIED"
    TEXT = "TEXT"
    IMAGE = "IMAGE"
    VIDEO = "VIDEO"
    AUDIO = "AUDIO"
    DOCUMENT = "DOCUMENT"

🐍 Let's define a few helpers:

from google.genai.types import (
    FileData,
    GenerateContentResponse,
    MediaModality,
    Part,
    PartMediaResolution,
    PartMediaResolutionLevel,
    VideoMetadata,
)

TokensPerModality = dict[MediaModality, int]


def display_tokens_per_modality(response: GenerateContentResponse) -> None:
    usage_metadata = response.usage_metadata
    if not usage_metadata:
        print("⚠️ No usage metadata found in the response.")
        return
    prompt_tokens_details = usage_metadata.prompt_tokens_details or []
    tokens_per_modality = get_empty_tokens_per_modality()

    for tokens_details in prompt_tokens_details:
        modality = tokens_details.modality
        if modality and modality in tokens_per_modality:
            tokens_per_modality[modality] += tokens_details.token_count or 0

    prompt_token_count = usage_metadata.prompt_token_count or 0
    display_token_table(tokens_per_modality, prompt_token_count)


def get_empty_tokens_per_modality() -> TokensPerModality:
    return {
        modality: 0
        for modality in MediaModality
        if modality != MediaModality.MODALITY_UNSPECIFIED
    }


def display_token_table(
    tokens_per_modality: TokensPerModality,
    total_tokens: int,
) -> None:
    def yield_row() -> Iterator[list[str]]:
        yield [mod.value for mod in tokens_per_modality.keys()] + ["Total"]
        yield [":-:" for _ in range(len(tokens_per_modality) + 1)]
        yield [f"{t:,d}" for t in tokens_per_modality.values()] + [f"{total_tokens:,d}"]

    markdown = "\n".join("| " + " | ".join(row) + " |" for row in yield_row())
    IPython.display.display(IPython.display.Markdown(markdown))

Let's check a few examples…

🖼️ Image Tokenization

Image token counts depend on the image itself and the configured media resolution:

class PartMediaResolutionLevel(StrEnum):
    MEDIA_RESOLUTION_UNSPECIFIED = "MEDIA_RESOLUTION_UNSPECIFIED"
    MEDIA_RESOLUTION_LOW = "MEDIA_RESOLUTION_LOW"
    MEDIA_RESOLUTION_MEDIUM = "MEDIA_RESOLUTION_MEDIUM"
    MEDIA_RESOLUTION_HIGH = "MEDIA_RESOLUTION_HIGH"
    MEDIA_RESOLUTION_ULTRA_HIGH = "MEDIA_RESOLUTION_ULTRA_HIGH"

For a given media resolution level, the Gemini 3 tokenizers will use these maximum token budgets per image:

`media_resolution`	Tokens
`MEDIA_RESOLUTION_LOW`	280
`MEDIA_RESOLUTION_MEDIUM`	560
`MEDIA_RESOLUTION_HIGH` (default)	1,120
`MEDIA_RESOLUTION_ULTRA_HIGH`	2,240

🐍 Check how this cat image is tokenized by default:

def display_tokens_for_image(
    image_uri: str,
    media_resolution_level: PartMediaResolutionLevel | None = None,
) -> None:
    print(f"🧪 {media_resolution_level=}")
    contents = Part.from_uri(
        file_uri=image_uri,
        mime_type="image/*",
        media_resolution=(
            PartMediaResolution(level=media_resolution_level)
            if media_resolution_level
            else None
        ),
    )
    response = client.models.generate_content(model=MODEL_ID, contents=contents)
    display_tokens_per_modality(response)


IMAGE_URI = "https://storage.googleapis.com/cloud-samples-data/generative-ai/image/chair-cat.png"
display_tokens_for_image(IMAGE_URI)

🧪 media_resolution_level=None

TEXT	IMAGE	VIDEO	AUDIO	DOCUMENT	Total
0	1,080	0	0	0	1,080

💡 This image is tokenized into only 1,080 tokens (instead of the maximum 1,120), saving us 40 tokens! It's a nice touch that helps keep costs down rather than defaulting to the upper limit.

🐍 For less detailed images, you can reduce token counts by a factor of 2 or 4 using the medium or low levels:

display_tokens_for_image(IMAGE_URI, PartMediaResolutionLevel.MEDIA_RESOLUTION_LOW)

🧪 media_resolution_level=<PartMediaResolutionLevel.MEDIA_RESOLUTION_LOW: 'MEDIA_RESOLUTION_LOW'>

TEXT	IMAGE	VIDEO	AUDIO	DOCUMENT	Total
0	264	0	0	0	264

💡 At the other end of the media resolution range, the ultra-high level is great for detailed images (like a photo of a circuit board with many components), ensuring maximum visual understanding. An image at this level uses between 2,000 and 2,240 tokens.

🔊 Audio Tokenization

Audio tokenization currently uses 25 tokens per second to represent the audio stream semantically.

🐍 Here is the tokenization for a 3.049-second audio file:

def display_tokens_for_audio(audio_uri: str) -> None:
    contents = Part.from_uri(file_uri=audio_uri, mime_type="audio/*")
    response = client.models.generate_content(model=MODEL_ID, contents=contents)
    display_tokens_per_modality(response)


AUDIO_URI = "https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/hello_gemini_are_you_there.wav"
display_tokens_for_audio(AUDIO_URI)

TEXT	IMAGE	VIDEO	AUDIO	DOCUMENT	Total
0	0	0	77	0	77

💡 ceil(3.049 s × 25 tok/s) = ceil(76.225 tok) = 77 tok

🐍 A longer, 30.772-second audio file requires 10 times as many tokens, as expected:

AUDIO_URI = "https://storage.googleapis.com/cloud-samples-data/generative-ai/audio/sailor_audio.mp3"
display_tokens_for_audio(AUDIO_URI)

TEXT	IMAGE	VIDEO	AUDIO	DOCUMENT	Total
0	0	0	770	0	770

💡 ceil(30.772 s × 25 tok/s) = ceil(769.3 tok) = 770 tok

🎬 Video Tokenization

For videos:

The audio tokenizer is the same as for standalone audio (25 tokens per second).
Video frames are sampled (1 FPS by default) and tokenized based on the media resolution.

For a given media resolution level, the Gemini 3 tokenizers will use these maximum token budgets per sampled frame:

`media_resolution`	Max. tokens
`MEDIA_RESOLUTION_LOW`/`MEDIA_RESOLUTION_MEDIUM` (default)	70
`MEDIA_RESOLUTION_HIGH`	280

🐍 Here's the tokenization for a 59-second video:

def display_tokens_for_video(
    video_uri: str,
    fps: float | None = None,
    media_resolution_level: PartMediaResolutionLevel | None = None,
) -> None:
    print(f"🧪 {fps=}, {media_resolution_level=}")
    contents = Part(
        file_data=FileData(file_uri=video_uri, mime_type="video/*"),
        video_metadata=VideoMetadata(fps=fps) if fps is not None else None,
        media_resolution=(
            PartMediaResolution(level=media_resolution_level)
            if media_resolution_level
            else None
        ),
    )
    response = client.models.generate_content(model=MODEL_ID, contents=contents)
    display_tokens_per_modality(response)


VIDEO_URI = "https://www.youtube.com/watch?v=0pJn3g8dfwk"
display_tokens_for_video(VIDEO_URI)

🧪 fps=None, media_resolution_level=None

TEXT	IMAGE	VIDEO	AUDIO	DOCUMENT	Total
0	0	3,894	1,475	0	5,369

💡 Details

Video: ceil(59 s × 1 frame/s × 66 tok/frame) = ceil(3894 tok) = 3894 tok

Audio: ceil(59 s × 25 tok/s) = ceil(1475 tok) = 1475 tok

🐍 Doubling the sampling rate requires twice as many video tokens:

display_tokens_for_video(VIDEO_URI, fps=2)

🧪 fps=2, media_resolution_level=None

TEXT	IMAGE	VIDEO	AUDIO	DOCUMENT	Total
0	0	7,788	1,475	0	9,263

💡 Details

Video: ceil(59 s × 2 frame/s × 66 tok/frame) = ceil(7788 tok) = 7788 tok

Audio: ceil(59 s × 25 tok/s) = ceil(1475 tok) = 1475 tok

🐍 If you switch from low/medium to high media resolution, sampled frames are tokenized in greater detail, requiring four times as many video tokens:

VIDEO_URI = "https://www.youtube.com/watch?v=0pJn3g8dfwk"
display_tokens_for_video(
    VIDEO_URI,
    media_resolution_level=PartMediaResolutionLevel.MEDIA_RESOLUTION_HIGH,
)

🧪 fps=None, media_resolution_level=<PartMediaResolutionLevel.MEDIA_RESOLUTION_HIGH: 'MEDIA_RESOLUTION_HIGH'>

TEXT	IMAGE	VIDEO	AUDIO	DOCUMENT	Total
0	0	15,576	1,475	0	17,051

💡 Details

Video: ceil(59 s × 1 frame/s × 264 tok/frame) = ceil(15576 tok) = 15576 tok

Audio: ceil(59 s × 25 tok/s) = ceil(1475 tok) = 1475 tok

📄 Document Tokenization

For a given media resolution level, the Gemini 3 tokenizers will use these maximum token budgets per PDF page:

`media_resolution`	Tokens
`MEDIA_RESOLUTION_LOW`	280
`MEDIA_RESOLUTION_MEDIUM` (default)	560
`MEDIA_RESOLUTION_HIGH`	1,120

🐍 Here's the tokenization for a one-page PDF at different media resolutions:

def display_tokens_for_document(
    document_uri: str,
    media_resolution_level: PartMediaResolutionLevel | None = None,
) -> None:
    print(f"🧪 {media_resolution_level=}")
    contents = Part.from_uri(
        file_uri=document_uri,
        mime_type="application/pdf",
        media_resolution=(
            PartMediaResolution(level=media_resolution_level)
            if media_resolution_level
            else None
        ),
    )
    response = client.models.generate_content(model=MODEL_ID, contents=contents)
    display_tokens_per_modality(response)


DOCUMENT_URI = (
    "https://storage.googleapis.com/cloud-samples-data/generative-ai/pdf/invoice.pdf"
)
media_resolution_levels = [
    PartMediaResolutionLevel.MEDIA_RESOLUTION_LOW,
    PartMediaResolutionLevel.MEDIA_RESOLUTION_MEDIUM,
    PartMediaResolutionLevel.MEDIA_RESOLUTION_HIGH,
]
for media_resolution_level in media_resolution_levels:
    display_tokens_for_document(DOCUMENT_URI, media_resolution_level)

🧪 media_resolution_level=<PartMediaResolutionLevel.MEDIA_RESOLUTION_LOW: 'MEDIA_RESOLUTION_LOW'>

TEXT	IMAGE	VIDEO	AUDIO	DOCUMENT	Total
0	266	0	0	0	266

🧪 media_resolution_level=<PartMediaResolutionLevel.MEDIA_RESOLUTION_MEDIUM: 'MEDIA_RESOLUTION_MEDIUM'>

TEXT	IMAGE	VIDEO	AUDIO	DOCUMENT	Total
0	532	0	0	0	532

🧪 media_resolution_level=<PartMediaResolutionLevel.MEDIA_RESOLUTION_HIGH: 'MEDIA_RESOLUTION_HIGH'>

TEXT	IMAGE	VIDEO	AUDIO	DOCUMENT	Total
0	1,092	0	0	0	1,092

💡 Remarks

Low: 266 tok/pg

Medium: 532 tok/pg

High: 1092 tok/pg

🐍 Here's another test for a 15-page PDF:

DOCUMENT_URI = "https://storage.googleapis.com/cloud-samples-data/generative-ai/pdf/1706.03762v7.pdf"
for media_resolution_level in media_resolution_levels:
    display_tokens_for_document(DOCUMENT_URI, media_resolution_level)

🧪 media_resolution_level=<PartMediaResolutionLevel.MEDIA_RESOLUTION_LOW: 'MEDIA_RESOLUTION_LOW'>

TEXT	IMAGE	VIDEO	AUDIO	DOCUMENT	Total
0	3,990	0	0	0	3,990

🧪 media_resolution_level=<PartMediaResolutionLevel.MEDIA_RESOLUTION_MEDIUM: 'MEDIA_RESOLUTION_MEDIUM'>

TEXT	IMAGE	VIDEO	AUDIO	DOCUMENT	Total
0	7,800	0	0	0	7,800

🧪 media_resolution_level=<PartMediaResolutionLevel.MEDIA_RESOLUTION_HIGH: 'MEDIA_RESOLUTION_HIGH'>

TEXT	IMAGE	VIDEO	AUDIO	DOCUMENT	Total
0	16,530	0	0	0	16,530

💡 Remarks

Low: 3990 tok / 15 pg = 266 tok/pg

Medium: 7800 tok / 15 pg = 520 tok/pg

High: 16530 tok / 15 pg = 1102 tok/pg

🎉 Conclusion

You've now mastered token counting both locally and via the Gemini API!

With the LocalTokenizer, you can estimate text token counts completely offline, saving bandwidth and avoiding rate limits. You've also seen how Gemini's multimodal tokenizers handle images, audio, video, and PDFs, and how to extract precise token usage from usage_metadata for accurate tracking and billing.

➕ More!

Try it yourself: Use the companion notebook (or run the notebook on Colab) to reproduce all results in this article.
Get inspired: Explore typical use cases in the Agent Platform Prompt Gallery.
Stay updated: Follow the Agent Platform Release Notes.
Follow me: Connect with me (@PicardParis) on LinkedIn or Twitter / X for more cloud, applied AI, and Python explorations…

Building Knowledge Graphs with Gemini

Laurent Picard — Fri, 12 Jun 2026 17:37:01 +0000

✨ Overview

In this exploration, we'll see how to turn raw, unstructured documents into structured knowledge graphs using Gemini. We'll start by prototyping to develop our intuition. Then, we'll optimize our prompts and outputs, and finally scale up to process entire books or dense legal contracts. By the end, we'll even visualize extracted book narratives and contractual network graphs!

A few notes before we start:

I'm a software engineer, developer advocate at Google Cloud, and hope you'll learn a few things. Thoughts and opinions are entirely my own.
The complete source code is available in this notebook (including setup details and future updates) under the Apache 2.0 license. You can also directly open the notebook in Colab. This article reproduces all the results generated by a click on “Run all”.
You can experiment and build for free with Gemini in the following platforms:
- Google AI Studio (get an API key to call Gemini programmatically)
- Agent Studio (start building on Google Cloud with $300 in free credits)

🔥 Challenge

Documents are everywhere. We use them for business, daily operations, legal matters, technical docs, education, and even just for fun. However, documents are not databases. They're generally unstructured, and fully understanding them requires multiple reading passes.

So, can we extract structured knowledge from documents using only the following?

1 document
1 prompt
1 request

Let's try with Gemini…

🏁 Setup

🐍 Python packages

We'll use the following packages:

google-genai for calling Gemini with the Google Gen AI Python SDK
networkx for graph management

We'll also need:

tenacity for request management (a dependency of google-genai)
matplotlib and pillow for data visualization (dependencies of networkx)

%pip install --quiet "google-genai>=2.6.0" "networkx[default]"

🤝 Gemini API

To use the Gemini API, we have two main options:

Via Agent Platform (formerly Vertex AI) with a Google Cloud project
Via Google AI Studio with a Gemini API key

The Google Gen AI SDK provides a unified interface to these APIs and we can use environment variables for the configuration. 🔽

🛠️ Option 1 - Gemini API via Agent Platform

Requirements:

A Google Cloud project
The Agent Platform API must be enabled for this project: ▶️ Enable the Agent Platform API

Gen AI SDK environment variables:

GOOGLE_GENAI_USE_ENTERPRISE="True"
GOOGLE_CLOUD_PROJECT="<PROJECT_ID>"
GOOGLE_CLOUD_LOCATION="<LOCATION>"

💡 For preview models, the location must be set to global. For generally available models, we can choose the closest location among the Google model endpoint locations.

ℹ️ Learn more about setting up a project and a development environment.

🛠️ Option 2 - Gemini API via Google AI Studio

Requirement:

A Gemini API key

Gen AI SDK environment variables:

GOOGLE_GENAI_USE_ENTERPRISE="False"
GOOGLE_API_KEY="<API_KEY>"

ℹ️ Learn more about getting a Gemini API key from Google AI Studio.

💡 You can store your environment configuration outside of the source code:

Environment	Method
IDE	`.env` file (or equivalent)
Colab	Colab Secrets (🗝️ icon in left panel, see code below)
Colab Enterprise	Google Cloud project and location are automatically defined
Workbench	Google Cloud project and location are automatically defined

Define the following environment detection functions. You can also define your configuration manually if needed. 🔽

import os
import sys
from collections.abc import Callable

from google import genai

# Manual setup (leave unchanged if setup is environment-defined)

# @markdown **Which API: Agent Platform (formerly Vertex AI) or Google AI Studio?**
GOOGLE_GENAI_USE_ENTERPRISE = True  # @param {type: "boolean"}

# @markdown **Option A - Google Cloud project [+location]**
GOOGLE_CLOUD_PROJECT = ""  # @param {type: "string"}
GOOGLE_CLOUD_LOCATION = "global"  # @param {type: "string"}

# @markdown **Option B - Google AI Studio API key**
GOOGLE_API_KEY = ""  # @param {type: "string"}


def check_environment() -> bool:
    check_colab_user_authentication()
    return check_manual_setup() or check_enterprise() or check_colab() or check_local()


def check_manual_setup() -> bool:
    return check_define_env_vars(
        GOOGLE_GENAI_USE_ENTERPRISE,
        GOOGLE_CLOUD_PROJECT.strip(),  # Might have been pasted with a newline
        GOOGLE_CLOUD_LOCATION,
        GOOGLE_API_KEY,
    )


def check_enterprise() -> bool:
    # Workbench and Colab Enterprise
    match os.getenv("VERTEX_PRODUCT", ""):
        case "WORKBENCH_INSTANCE":
            pass
        case "COLAB_ENTERPRISE":
            if not running_in_colab_env():
                return False
        case _:
            return False

    return check_define_env_vars(
        True,
        os.getenv("GOOGLE_CLOUD_PROJECT", ""),
        os.getenv("GOOGLE_CLOUD_REGION", ""),
        "",
    )


def check_colab() -> bool:
    if not running_in_colab_env():
        return False

    # Colab Enterprise was checked before, so this is Colab only
    from google.colab import auth as colab_auth  # type: ignore

    colab_auth.authenticate_user()

    # Use Colab Secrets (🗝️ icon in left panel) to store the environment variables
    # Secrets are private, visible only to you and the notebooks that you select
    # - Agent Platform: Store your settings as secrets
    # - Google AI: Directly import your Gemini API key from the UI
    enterprise, project, location, api_key = get_vars(get_colab_secret)

    return check_define_env_vars(enterprise, project, location, api_key)


def check_local() -> bool:
    enterprise, project, location, api_key = get_vars(os.getenv)

    return check_define_env_vars(enterprise, project, location, api_key)


def running_in_colab_env() -> bool:
    # Colab or Colab Enterprise
    return "google.colab" in sys.modules


def check_colab_user_authentication() -> None:
    if running_in_colab_env():
        from google.colab import auth as colab_auth  # type: ignore

        colab_auth.authenticate_user()


def get_colab_secret(secret_name: str, default: str) -> str:
    from google.colab import errors, userdata  # type: ignore

    try:
        return userdata.get(secret_name)
    except errors.SecretNotFoundError:
        return default


def disable_colab_cell_scrollbar() -> None:
    if running_in_colab_env():
        from google.colab import output  # type: ignore

        output.no_vertical_scroll()


def get_vars(getenv: Callable[[str, str], str]) -> tuple[bool, str, str, str]:
    # Limit getenv calls to the minimum (may trigger UI confirmation for secret access)
    enterprise_str = getenv("GOOGLE_GENAI_USE_ENTERPRISE", "")
    if not enterprise_str:
        enterprise_str = getenv("GOOGLE_GENAI_USE_VERTEXAI", "")
    if enterprise_str:
        enterprise = enterprise_str.lower() in ["true", "1"]
    else:
        enterprise = bool(getenv("GOOGLE_CLOUD_PROJECT", ""))

    project = getenv("GOOGLE_CLOUD_PROJECT", "") if enterprise else ""
    location = getenv("GOOGLE_CLOUD_LOCATION", "") if project else ""
    api_key = getenv("GOOGLE_API_KEY", "") if not project else ""

    return enterprise, project, location, api_key


def check_define_env_vars(
    enterprise: bool,
    project: str,
    location: str,
    api_key: str,
) -> bool:
    match (enterprise, bool(project), bool(location), bool(api_key)):
        case (True, True, _, _):
            # Agent Platform - Google Cloud project [+location]
            location = location or "global"
            define_env_vars(enterprise, project, location, "")
        case (True, False, _, True):
            # Agent Platform - API key
            define_env_vars(enterprise, "", "", api_key)
        case (False, _, _, True):
            # Google AI Studio - API key
            define_env_vars(enterprise, "", "", api_key)
        case _:
            return False

    return True


def define_env_vars(
    enterprise: bool,
    project: str,
    location: str,
    api_key: str,
) -> None:
    os.environ["GOOGLE_GENAI_USE_ENTERPRISE"] = str(enterprise)
    os.environ["GOOGLE_GENAI_USE_VERTEXAI"] = str(enterprise)
    os.environ["GOOGLE_CLOUD_PROJECT"] = project
    os.environ["GOOGLE_CLOUD_LOCATION"] = location
    os.environ["GOOGLE_API_KEY"] = api_key


def check_configuration(client: genai.Client) -> None:
    service = "Agent Platform" if client.vertexai else "Google AI Studio"
    print(f"✅ Using the {service} API", end="")

    if client._api_client.project:
        print(f' with project "{client._api_client.project[:7]}…"', end="")
        print(f' in location "{client._api_client.location}"')
    elif client._api_client.api_key:
        api_key = client._api_client.api_key
        print(f' with API key "{api_key[:5]}…{api_key[-5:]}"', end="")
        print(f" (in case of error, make sure it was created for {service})")


print("✅ Environment functions defined")

✅ Environment functions defined

🤖 Gen AI SDK

To send Gemini requests, we'll use a google.genai client:

from google import genai

check_environment()

client = genai.Client()

check_configuration(client)

✅ Using the Agent Platform API with project "lpdemo-…" in location "global"

🔣 Input data

We need a suite of test data to develop our solution.

Multimodality

We'll test the following types:

Text (text/plain): Classic books are good text sources of varying lengths and languages.
PDF (application/pdf): Legal agreements are also great examples of complex and dense documents.

Gemini is natively multimodal, which means it can process different types of inputs. Once we've built knowledge graphs from text or PDF inputs, the solution will also naturally support the following formats:

Image (image/*)
Audio (audio/*)
Video (video/*)

General knowledge

⚠️ LLMs are trained on general knowledge, which becomes part of their "long-term memory". To avoid generating memorized information, we'll explicitly instruct the model to use only the provided inputs.

Multilinguality

Gemini is also natively multilingual, which lets us process inputs and generate outputs in 100+ languages.

To keep things general, we'll use English for prompts and knowledge graphs, but you can use any of the 100+ supported languages, as long as your prompts remain clear and explicit.

Let's define a few data sources and helpers: 🔽

import mimetypes
from collections.abc import Iterator
from enum import Enum
from pathlib import Path

from google.genai.types import Part

GOOGLE_CLOUD_STORAGE_PREFIX = "gs://"
HTTPS_PREFIX = "https://"
FILE_PREFIX = "file://"
LOCAL_FOLDER = "./"


class Source(Enum):
    def yield_contents(self) -> Iterator[Part]:
        file_uri = self.value
        if not client.vertexai:
            file_uri = convert_to_https_url_if_cloud_storage_uri(file_uri)
        mime_type, _ = mimetypes.guess_type(file_uri)
        assert mime_type is not None, f"❌ Could not determine mime type: {file_uri=}"

        if file_uri.startswith((GOOGLE_CLOUD_STORAGE_PREFIX, HTTPS_PREFIX)):
            yield Part.from_uri(file_uri=file_uri, mime_type=mime_type)
            return

        if file_uri.startswith(FILE_PREFIX):
            file = Path(file_uri.removeprefix(FILE_PREFIX))
            assert file.exists(), f"❌ File does not exist: {file=}"
            if mime_type == "text/plain":
                yield Part.from_text(text=file.read_text(encoding="utf-8"))
            else:
                yield Part.from_bytes(data=file.read_bytes(), mime_type=mime_type)
            return

    def yield_source_names(self) -> Iterator[str]:
        yield self.name

    def yield_source_links(self) -> Iterator[str]:
        file_uri = convert_to_https_url_if_cloud_storage_uri(self.value)
        if file_uri.startswith(HTTPS_PREFIX):
            yield file_uri
            return
        if file_uri.startswith(FILE_PREFIX):
            yield file_uri.removeprefix(FILE_PREFIX)
            return


def convert_to_https_url_if_cloud_storage_uri(uri: str) -> str:
    return (
        f"{HTTPS_PREFIX}storage.googleapis.com/{uri.removeprefix(GOOGLE_CLOUD_STORAGE_PREFIX)}"
        if uri.startswith(GOOGLE_CLOUD_STORAGE_PREFIX)
        else uri
    )


def local_file(filename: str) -> str:
    return f"{FILE_PREFIX}{LOCAL_FOLDER}{filename}"


# You can find public domain books on Project Gutenberg: https://gutenberg.org/ebooks
def project_gutenberg_txt_url(id: int) -> str:
    return f"{HTTPS_PREFIX}gutenberg.org/cache/epub/{id}/pg{id}.txt"


class Classic(Source):
    en_hugo_les_misérables = project_gutenberg_txt_url(135)
    en_dumas_count_of_monte_cristo = project_gutenberg_txt_url(1184)
    fr_zola_thérèse_raquin = project_gutenberg_txt_url(7461)
    fr_dumas_trois_mousquetaires = project_gutenberg_txt_url(13951)
    fr_dumas_vingt_ans_après = project_gutenberg_txt_url(13952)
    fr_dumas_comte_de_monte_cristo_1 = project_gutenberg_txt_url(17989)
    fr_dumas_comte_de_monte_cristo_2 = project_gutenberg_txt_url(17990)
    fr_dumas_comte_de_monte_cristo_3 = project_gutenberg_txt_url(17991)
    fr_dumas_comte_de_monte_cristo_4 = project_gutenberg_txt_url(17992)


class Document(Source):
    en_pharma_dev_agreement = "gs://cloud-samples-data/documentai/ContractDocAI/CUAD_v1/Part_I/Development/PhasebioPharmaceuticalsInc_20200330_10-K_EX-10.21_12086810_EX-10.21_Development Agreement.pdf"


class Collection(Source):
    fr_dumas_comte_de_monte_cristo = [
        Classic.fr_dumas_comte_de_monte_cristo_1,
        Classic.fr_dumas_comte_de_monte_cristo_2,
        Classic.fr_dumas_comte_de_monte_cristo_3,
        Classic.fr_dumas_comte_de_monte_cristo_4,
    ]
    fr_dumas_trois_mousquetaires_vingt_ans_après = [
        Classic.fr_dumas_trois_mousquetaires,
        Classic.fr_dumas_vingt_ans_après,
    ]

    def yield_contents(self) -> Iterator[Part]:
        for source in self.value:
            yield from source.yield_contents()

    def yield_source_names(self) -> Iterator[str]:
        for source in self.value:
            yield from source.yield_source_names()

    def yield_source_links(self) -> Iterator[str]:
        for source in self.value:
            yield from source.yield_source_links()


def display_input_data_caption(source: Source) -> None:
    names = list(source.yield_source_names())
    links = list(source.yield_source_links())
    links = ", ".join(
        f"[{name}](<{link}>)" for name, link in zip(names, links, strict=True)
    )
    md = f"**Input data** ({links})"
    display_markdown(md)


print("✅ Data helpers defined")

✅ Data helpers defined

🧠 Gemini model

Gemini comes in different versions and sizes (Flash-Lite, Flash, and Pro).

Let's get started with Gemini 3.1 Flash-Lite, as it offers high performance, low latency, and very high output speed:

GEMINI_3_1_FLASH_LITE = "gemini-3.1-flash-lite"

⚙️ Gemini configuration

Gemini can be used in different ways, ranging from factual to creative modes. We're essentially dealing with a data-extraction use case. We want the results to be as factual and deterministic as possible. To achieve this, we can adjust the content generation parameters.

We'll set the temperature, top_p, and seed parameters to minimize randomness:

temperature=0.0
top_p=0.0
seed=42 (arbitrary fixed value)

🛠️ Helpers

Now, let's add core helper classes and functions: 🔽

from enum import StrEnum, auto

import IPython.display
import tenacity
from google.genai.errors import ClientError
from google.genai.types import (
    FinishReason,
    GenerateContentConfig,
    GenerateContentResponse,
    ThinkingConfig,
    ThinkingLevel,
)


class Model(Enum):
    GEMINI_3_1_FLASH_LITE = "gemini-3.1-flash-lite"
    GEMINI_3_5_FLASH = "gemini-3.5-flash"
    GEMINI_2_5_FLASH = "gemini-2.5-flash"
    GEMINI_2_5_PRO = "gemini-2.5-pro"
    # Preview
    GEMINI_3_1_PRO = "gemini-3.1-pro-preview"
    # Default model
    DEFAULT = GEMINI_3_1_FLASH_LITE


# Default configuration for more deterministic outputs
DEFAULT_CONFIG = GenerateContentConfig(
    temperature=0.0,
    top_p=0.0,
    seed=42,  # Arbitrary fixed value
)


class ShowAs(StrEnum):
    DONT_SHOW = auto()
    TEXT = auto()
    MARKDOWN = auto()


def generate_content(
    prompt: str,
    source: Source | str | None = None,
    *,
    model: Model | None = None,
    config: GenerateContentConfig | None = None,
    system_instruction: str | None = None,
    show_prompt: ShowAs = ShowAs.DONT_SHOW,
    show_response: ShowAs = ShowAs.MARKDOWN,
    only_show_prompt: bool = False,
    return_response: bool = False,
) -> GenerateContentResponse | None:
    disable_colab_cell_scrollbar()

    model = model or Model.DEFAULT
    model_id = model.value
    prompt_contents = get_prompt_contents(prompt, source, show_prompt, only_show_prompt)
    if only_show_prompt:
        return None
    config = config or get_generate_content_config(model, system_instruction)
    client = check_client_for_model(model)

    response = None
    display_request_header(model_id, source)
    for attempt in get_retrier():
        with attempt:
            response = client.models.generate_content(
                model=model_id,
                contents=prompt_contents,  # type: ignore
                config=config,
            )
            display_response_info(response)
            display_response(response, show_response)

    return response if return_response else None


def get_prompt_contents(
    prompt: str,
    source: Source | str | None,
    show_prompt: ShowAs,
    only_show_prompt: bool,
) -> list[str | Part]:
    def yield_prompt_contents() -> Iterator[str | Part]:
        if not source:
            yield prompt.strip()
            return
        yield "==Start of input data==\n"
        if isinstance(source, str):
            yield f"{source.strip()}\n"
        else:
            yield from source.yield_contents()
        yield "==End of input data==\n"
        yield f"==Start of user prompt==\n{prompt.strip()}\n==End of user prompt=="

    prompt_contents = list(yield_prompt_contents())
    display_prompt(prompt_contents, show_prompt, only_show_prompt)

    return prompt_contents


def get_generate_content_config(
    model: Model,
    system_instruction: str | None = None,
) -> GenerateContentConfig:
    thinking_config = get_thinking_config_for_model(model)

    return GenerateContentConfig(
        system_instruction=system_instruction,
        temperature=DEFAULT_CONFIG.temperature,
        top_p=DEFAULT_CONFIG.top_p,
        seed=DEFAULT_CONFIG.seed,
        thinking_config=thinking_config,
    )


def get_thinking_config_for_model(model: Model) -> ThinkingConfig | None:
    # Use minimal thinking configurations since our prompt will directly provide a chain of thought
    match model:
        case Model.GEMINI_2_5_FLASH:
            return ThinkingConfig(thinking_budget=0)
        case Model.GEMINI_2_5_PRO:
            return ThinkingConfig(thinking_budget=128, include_thoughts=False)
        case Model.GEMINI_3_1_FLASH_LITE | Model.GEMINI_3_5_FLASH:
            return ThinkingConfig(thinking_level=ThinkingLevel.MINIMAL)
        case Model.GEMINI_3_1_PRO:
            return ThinkingConfig(thinking_level=ThinkingLevel.LOW)
        case _:
            return None  # Default (dynamic thinking is generally enabled)


def check_client_for_model(model: Model) -> genai.Client:
    if (
        model.value.endswith("-preview")
        and client.vertexai
        and client._api_client.location != "global"
    ):
        # Preview models are only available on the "global" location
        return genai.Client(
            enterprise=client.vertexai,
            project=client._api_client.project,
            location="global",
        )

    return client


def get_retrier() -> tenacity.Retrying:
    return tenacity.Retrying(
        stop=tenacity.stop_after_attempt(7),
        wait=tenacity.wait_incrementing(start=10, increment=1),
        retry=tenacity.retry_if_exception(should_retry_request),
        reraise=True,
    )


def should_retry_request(err: BaseException) -> bool:
    if not isinstance(err, ClientError):
        return False
    print(f"❌ ClientError {err.code}: {err.message}")

    retry = False
    match err.code:
        case 400 if err.message is not None and " try again " in err.message:
            # Workshop: project accessing Cloud Storage for the first time (service agent provisioning)
            retry = True
        case 429:
            # Workshop: temporary project with 1 QPM quota
            retry = True
    print(f"🔄 Retry: {retry}")

    return retry


PRINT_COLUMNS = 80
PRINT_SEPARATOR_CHAR = "-"
PRINT_SEPARATOR = PRINT_COLUMNS * PRINT_SEPARATOR_CHAR


def print_caption(caption: str) -> None:
    print(f" {caption} ".center(PRINT_COLUMNS, PRINT_SEPARATOR_CHAR))


def print_separator() -> None:
    print(PRINT_SEPARATOR)


def display_response_info(response: GenerateContentResponse) -> None:
    if usage_metadata := response.usage_metadata:
        if usage_metadata.prompt_token_count:
            print(f"Input tokens    : {usage_metadata.prompt_token_count:9,d}")
        if usage_metadata.cached_content_token_count:
            print(f"Cached tokens   : {usage_metadata.cached_content_token_count:9,d}")
        if usage_metadata.candidates_token_count:
            print(f"Output tokens   : {usage_metadata.candidates_token_count:9,d}")
        if usage_metadata.thoughts_token_count:
            print(f"Thoughts tokens : {usage_metadata.thoughts_token_count:9,d}")
    if not response.candidates:
        print("❌ No `response.candidates`")
        return
    if (finish_reason := response.candidates[0].finish_reason) != FinishReason.STOP:
        print(f"❌ {finish_reason = }")
    if not response.text:
        print("❌ No `response.text`")
        return


def display_prompt(
    contents: list[str | Part],
    show_as: ShowAs,
    only_show_prompt: bool,
) -> None:
    def yield_prompt_strings() -> Iterator[str]:
        for content in contents:
            if isinstance(content, Part):
                yield f"{content!r}\n"
                continue
            yield content

    if only_show_prompt and show_as == ShowAs.DONT_SHOW:
        show_as = ShowAs.TEXT
    if show_as == ShowAs.DONT_SHOW:
        return

    separator = "\n" if show_as == ShowAs.MARKDOWN else ""
    prompt = separator.join(yield_prompt_strings())
    print_caption("Prompt")
    match show_as:
        case ShowAs.TEXT:
            print(prompt)
        case ShowAs.MARKDOWN:
            display_markdown(prompt)
    if only_show_prompt:
        print_separator()


def display_request_header(model_id: str, source: Source | str | None = None) -> None:
    print_caption(f"Request / {model_id}")


def display_response(response: GenerateContentResponse, show_as: ShowAs) -> None:
    if show_as == ShowAs.DONT_SHOW or not (response_text := response.text):
        return
    print_caption("Start of Response")
    response_text = response_text.strip()
    match show_as:
        case ShowAs.TEXT:
            print(response_text)
        case ShowAs.MARKDOWN:
            display_markdown(response_text)
    print_caption("End of Response")


def display_markdown(markdown: str) -> None:
    IPython.display.display(IPython.display.Markdown(markdown))


print("✅ Helpers defined")

✅ Helpers defined

🏗️ Prototyping

Before diving into a solution, it helps to start by prototyping to build some intuition about the natural behavior of the model.

Let's define a short text of a few sentences:

text = """
- Henry Jones is a famous archaeologist. He is actually a "Junior" because he is named after his father.
- Sophie is Henry's daughter, shares his last name, and works as a software engineer.
- William Smith is an aerospace engineer and Sophie's lifelong friend. Everybody calls him Bill and Beau is his dog.
- Short Round met Henry as a child. They first became close friends, and Henry officially adopted him a few years later.
- Sophie and Bill both work at Acme Aerospace.
"""

📋 Listing characters

🧪 Let's see if Gemini can spot our characters…

prompt = """
Using only the input data, list all people and animals mentioned.
"""
generate_content(prompt, text)

----------------------- Request / gemini-3.1-flash-lite ------------------------
Input tokens    :       148
Output tokens   :        67
------------------------------ Start of Response -------------------------------
Based on the input data provided, here are the people and animals mentioned:

**People:**
*   Henry Jones (also known as Henry Jones Junior)
*   Sophie Jones
*   William Smith (also known as Bill)
*   Short Round

**Animals:**
*   Beau (Bill's dog)
------------------------------- End of Response --------------------------------

💡 All people and animals are detected as expected.

📋 Listing characters & relationships

🧪 Now, let's see if it can connect the dots and figure out who's who…

prompt = """
Using only the input data, list all people and animals mentioned, and how they relate to each other.
"""
generate_content(prompt, text)

----------------------- Request / gemini-3.1-flash-lite ------------------------
Input tokens    :       156
Output tokens   :       168
------------------------------ Start of Response -------------------------------
Based on the input data provided, here are the people and animals mentioned and their relationships:

**People:**
*   **Henry Jones (Junior):** A famous archaeologist. He is the father of Sophie, the adoptive father of Short Round, and is named after his own father.
*   **Sophie Jones:** A software engineer at Acme Aerospace. She is the daughter of Henry Jones and a lifelong friend of Bill (William Smith).
*   **William (Bill) Smith:** An aerospace engineer at Acme Aerospace. He is a lifelong friend of Sophie and the owner of Beau.
*   **Short Round:** The adopted son of Henry Jones. He met Henry as a child and they became close friends before the adoption.

**Animals:**
*   **Beau:** A dog owned by William (Bill) Smith.
------------------------------- End of Response --------------------------------

💡 Notes

The relationships between the characters can also be neatly consolidated.
Henry's father is not listed as a named character. Here, we can see the difference between explicit and implicit mentions. By default, only explicit entities are detailed (most likely because models are trained to summarize information).
Implied characters (like "he is named after his father") represent a level of indirection. We'll see in the next section how to detect them as well.

📚 Domain terminology

We're not domain experts in the field we're exploring (yet!).

An LLM processes instructions based on the given prompt and its training knowledge. This knowledge is part of its long-term memory, and we can learn a lot directly from the model itself.

🧪 Let's ask Gemini:

prompt = """
What is the terminology used when building a knowledge graph?
Please provide a simple data example in JSON.
"""
generate_content(prompt)

----------------------- Request / gemini-3.1-flash-lite ------------------------
Input tokens    :        21
Output tokens   :       581
------------------------------ Start of Response -------------------------------
Building a knowledge graph involves representing information as a network of interconnected entities. Here is the core terminology and a simple data example.

### Core Terminology

1.  **Entity (Node):** The "things" in your graph (e.g., a person, a place, a product).
2.  **Relationship (Edge/Link):** The connection between two entities. It describes how they interact (e.g., "works at," "lives in," "is a friend of").
3.  **Property (Attribute):** Key-value pairs that provide more detail about an entity or a relationship (e.g., a person's "age" or a relationship's "start_date").
4.  **Label:** A category assigned to a node or edge to define its type (e.g., a node might have the label "Person").
5.  **Schema (Ontology):** The formal structure or "blueprint" that defines the types of entities allowed and the rules for how they can be connected.
6.  **Triple:** The fundamental unit of a knowledge graph, consisting of a **Subject → Predicate → Object** (e.g., *Alice* → *works at* → *Google*).

---

### Simple Data Example (JSON)

In a knowledge graph, data is often represented as a collection of **Nodes** and **Edges**.

```json
{
"nodes": [
    {
    "id": "1",
    "label": "Person",
    "properties": {
        "name": "Alice",
        "age": 30
    }
    },
    {
    "id": "2",
    "label": "Company",
    "properties": {
        "name": "Google",
        "industry": "Technology"
    }
    }
],
"edges": [
    {
    "id": "e1",
    "source": "1",
    "target": "2",
    "label": "WORKS_AT",
    "properties": {
        "since": 2020
    }
    }
]
}
```

### Breakdown of the Example:

*   **Nodes:** We have two entities: "Alice" (a Person) and "Google" (a Company).
*   **Edge:** We have one relationship: "WORKS_AT" connecting Alice to Google.
*   **Properties:** We stored specific details like Alice's age and the year she started working at Google.
*   **Triple representation:** This JSON effectively encodes the triple: **(Alice) —[WORKS_AT]—> (Google)**.
------------------------------- End of Response --------------------------------

💡 We learn that knowledge graphs are made of entities and relationships, also called nodes and edges, and we get a nice introduction to the field. Using domain terminology will help make our prompts explicit and precise.

⛏️ Tabular extraction

To extract knowledge graphs, we'll reason in terms of entities and relationships, adopting domain terminology.

If we think of the final result as a database, our goal is to generate two linked tables, allowing us to reason in terms of data and fields.

Here is a conceptual view of what we want to achieve:

Entities

`id`	`name`	`label`
0	Henry Jones Jr.	person
1	Henry Jones Sr.	person

Relationships

`source_id`	`link`	`target_id`
0	child_of	1

Let's call this approach "tabular extraction" and split our instructions to output two successive tables, while still using a single request…

💠 Entities

In our prototype text, the entities we want to extract are characters (like people or animals). We can define an entity data schema with the fields id (0, 1, 2…), name (full name of the entity), and label (person|animal).

🧪 Let's extract the entities:

prompt = """
**Data Schema**

Entity:
- `id`: Unique integer identifier (0, 1, 2…).
- `name`: Full name of the entity.
- `label`: `person`|`animal`.

**Instructions**

1. Entity Extraction:
   - Extract every distinct entity from the input data that matches an allowed `label`.
   - Include entities that are explicitly named as well as implied entities whose names can be determined from the context.
2. Output the results as a JSON array inside a fenced code block.
"""

generate_content(prompt, text)

----------------------- Request / gemini-3.1-flash-lite ------------------------
Input tokens    :       249
Output tokens   :       195
------------------------------ Start of Response -------------------------------

[
  {
    "id": 0,
    "name": "Henry Jones Jr.",
    "label": "person"
  },
  {
    "id": 1,
    "name": "Henry Jones Sr.",
    "label": "person"
  },
  {
    "id": 2,
    "name": "Sophie Jones",
    "label": "person"
  },
  {
    "id": 3,
    "name": "William Smith",
    "label": "person"
  },
  {
    "id": 4,
    "name": "Beau",
    "label": "animal"
  },
  {
    "id": 5,
    "name": "Short Round",
    "label": "person"
  }
]

------------------------------- End of Response --------------------------------

💡 Remarks

Every entity is dynamically assigned a unique sequential identifier.
The name "Henry Jones Sr." is extracted even though he's not explicitly mentioned in the text. This stems from the instruction to "include entities that are explicitly named as well as implied entities whose names can be determined from the context".

🔗 Relationships

🧪 Now, let's extract both the entities and their relationships:

prompt = """
**Data Schema**

Entity:
- `id`: Unique integer identifier (0, 1, 2…).
- `name`: Full name of the entity.
- `label`: `person`|`animal`.

Relationship:
- `source_id`: `id` of the subject entity.
- `link`: `snake_case` predicate describing the relationship.
- `target_id`: `id` of the object entity.

**Instructions**

1. Entity Extraction:
   - Extract every distinct entity from the input data that matches an allowed `label`.
   - Include entities that are explicitly named as well as implied entities whose names can be determined from the context.
2. Relationship Extraction:
   - Extract every distinct relationship between them.
   - If a relationship changes over time, make sure to include every distinct stage of the relationship.
3. Output a JSON object with keys `entities` and `relationships` inside a fenced code block.
"""

response = generate_content(prompt, text, return_response=True)

----------------------- Request / gemini-3.1-flash-lite ------------------------
Input tokens    :       340
Output tokens   :       456
------------------------------ Start of Response -------------------------------

{
  "entities": [
    {
      "id": 0,
      "name": "Henry Jones Jr.",
      "label": "person"
    },
    {
      "id": 1,
      "name": "Henry Jones Sr.",
      "label": "person"
    },
    {
      "id": 2,
      "name": "Sophie Jones",
      "label": "person"
    },
    {
      "id": 3,
      "name": "William Smith",
      "label": "person"
    },
    {
      "id": 4,
      "name": "Beau",
      "label": "animal"
    },
    {
      "id": 5,
      "name": "Short Round",
      "label": "person"
    }
  ],
  "relationships": [
    {
      "source_id": 0,
      "link": "child_of",
      "target_id": 1
    },
    {
      "source_id": 2,
      "link": "child_of",
      "target_id": 0
    },
    {
      "source_id": 3,
      "link": "friend_of",
      "target_id": 2
    },
    {
      "source_id": 4,
      "link": "pet_of",
      "target_id": 3
    },
    {
      "source_id": 5,
      "link": "friend_of",
      "target_id": 0
    },
    {
      "source_id": 0,
      "link": "adopted",
      "target_id": 5
    },
    {
      "source_id": 5,
      "link": "child_of",
      "target_id": 0
    }
  ]
}

------------------------------- End of Response --------------------------------

💡 Remarks

The output now includes the relationships array.
We get the expected number of relationships, notably the two relationships between Henry and Short Round.
The link predicates are completely dynamic (a level of flexibility we left in the prompt). While it's interesting to see this natural behavior, we'll want to make it more deterministic for production since our prompt has too many degrees of freedom.
Our open prompt extracts relationships in only one direction. For example, "[Animal] pet_of [Person]" is an asymmetric relationship that could also be extracted as "[Person] owner_of [Animal]". This is another area where the prompt is too open-ended. In the finalization section, we'll see an example that asks the model to extract symmetric and asymmetric relationships in both directions.

💎 Structured output

We've concluded our prototyping stage with promising results using a data schema.

To move to production, the next step is to control the generation with a specific structured output.

🗂️ JSON

The JSON format has industry-wide support and serves as a core or intermediate format in many use cases.

For the next step, we would typically define classes using the Pydantic library and request a pure JSON output with a response schema in the config parameters:

response_mime_type="application/json"
response_schema="CLASS_DERIVED_FROM_PYDANTIC_BASE_MODEL" (docs)

⚠️ However, JSON is a pretty verbose format, designed for interoperability but not optimized for size. Even if we generate compact JSON (also called minified JSON), it still has inherent verbosity due to:

repeated keys (object fields) for each object instance
opening and closing brackets
enclosing quotes for each key and string value

ℹ️ When using LLMs, once the first token is generated, the remaining generation time is roughly proportional to the number of output tokens. Similarly, the cost of a request is based on token usage (input + output), with output tokens being significantly more expensive than input tokens.

💡 A better output structure will positively impact both generation speed and request cost.

Let's explore an alternative…

📅 TSV

Our tabular-extraction problem clearly calls for table outputs. An interesting possibility is to ask for Tab-Separated Values (TSV) outputs. For example, we can define our output to be formatted as two consecutive TSV tables.

Example output format

```tsv filename="entities.tsv"
id{TAB}name{TAB}label
[rows]
```

```tsv filename="relationships.tsv"
source_id{TAB}link{TAB}target_id
[rows]
```

Will this work?

Generating structured outputs like TSV will work seamlessly, as Gemini excels at patterns. We just need to be explicit about what's expected.

Will this be efficient?

💡 For our use case, this structure looks optimal:

Field names are stated once in the header row.
Field values are raw, without outer quotes.
Tab and newline separators take at most a single token each and do not collide with field values (unless we're dealing with very special input data, but we could then prompt the model to escape these using different characters or strings).

ℹ️ CSV could be another alternative, but common separators like commas are everywhere in natural language and frequently appear in names and descriptions (e.g., if we decide to extend entity fields). If you're interested in this topic, check out the TOON format, which proposes a JSON alternative using a YAML+CSV mix.

To make an informed decision, we should actually compare the number of tokens needed to represent the same data…

🗜️ Token comparison

Let's define some helpers to compare the same data using pretty-printed JSON, compact JSON, and TSV: 🔽

import csv
import io
import json
import re
from typing import Literal


def get_data_from_response(response: GenerateContentResponse) -> dict:
    response_text = response.text or ""
    pattern = r"```

json\s*(.*?)\s*

```"
    match = re.search(pattern, response_text, re.DOTALL)
    json_str = match.group(1) if match else response_text
    try:
        data = json.loads(json_str)
        if not isinstance(data, dict):
            print("❌ Returning empty dict (could not parse response as dict)")
            data = {}
    except (json.JSONDecodeError, TypeError):
        print("❌ Returning empty dict (failed parsing the JSON string)")
        data = {}
    return data


def get_tsv_string_from_data(data: dict) -> str:
    output = ""
    for key, items in data.items():
        rows = ""
        if items:
            with io.StringIO() as out:
                headers = list(items[0].keys())
                writer = csv.DictWriter(
                    out,
                    fieldnames=headers,
                    delimiter="\t",
                    lineterminator="\n",
                )
                writer.writeheader()
                writer.writerows(items)
                rows = out.getvalue()
        if output:
            output += "\n"
        output += f'```
{% endraw %}
tsv filename="{key}.tsv"\n{rows}
{% raw %}
```\n'
    return output


def print_text_excerpt(title: str, text: str, max_chars: int = 400) -> None:
    assert max_chars > 0
    chars = len(text)
    if chars <= 0:
        print_caption("❌ Empty text")
        return
    if chars <= max_chars:
        print_caption(f"{title} ({chars} chars)")
        print(text)
        return
    print_caption(
        f"{title} - First {max_chars}/{chars} chars ({max_chars / chars:.0%})"
    )
    print(f"{text[:max_chars]}…")


def compare_json_vs_tsv(
    response: GenerateContentResponse | None,
    only_show_excerpts: bool = False,
) -> None:
    def get_gain(rows: list[tuple[int, int]], col: Literal[0, 1]) -> str:
        val_0, val_1 = rows[0][col], rows[1][col]
        return f"**{1 - val_1 / val_0:.1%}**" if val_0 > 0 else "?"

    def yield_table_string_rows(
        source_title: str,
        target_title: str,
        rows: list[tuple[int, int]],
    ) -> Iterator[list[str]]:
        yield ["", "Chars", "Tokens"]
        yield ["-", "-:", "-:"]
        for caption, values in zip([source_title, target_title], rows):
            yield [caption, *[str(value) for value in values]]
        yield ["**Gain**", get_gain(rows, 0), get_gain(rows, 1)]

    def display_gain_table(
        source_title: str,
        target_title: str,
        source_text: str,
        target_text: str,
    ) -> None:
        print_caption(f"{source_title} → {target_title}")
        model = Model.DEFAULT
        model_id = model.value
        client = check_client_for_model(model)

        rows: list[tuple[int, int]] = []
        for s in [source_text, target_text]:
            chars = len(s)
            tokens = client.models.count_tokens(model=model_id, contents=s).total_tokens
            rows.append((chars, tokens or 0))
        markdown = "\n".join(
            "| " + " | ".join(row) + " |"
            for row in yield_table_string_rows(source_title, target_title, rows)
        )
        display_markdown(markdown)

    if response is None:
        print("❌ response is None")
        return
    data = get_data_from_response(response)
    formatted_json = f"```
{% endraw %}
json\n{json.dumps(data, indent=2)}\n
{% raw %}
```"
    compact_json = f"```
{% endraw %}
json\n{json.dumps(data, separators=(',', ':'))}\n
{% raw %}
```"
    tsv = get_tsv_string_from_data(data)

    if only_show_excerpts:
        max_chars = len(tsv)
        print_text_excerpt("Formatted JSON", formatted_json, max_chars)
        print_text_excerpt("Compact JSON", compact_json, max_chars)
        print_text_excerpt("TSV", tsv, max_chars)
        return

    display_gain_table("Formatted JSON", "Compact JSON", formatted_json, compact_json)
    display_gain_table("Compact JSON", "TSV", compact_json, tsv)
    display_gain_table("Formatted JSON", "TSV", formatted_json, tsv)
    print_separator()


print("✅ JSON vs TSV helpers defined")

✅ JSON vs TSV helpers defined

🧪 First, let's compare how much data we can represent for the same number of characters based on our latest API response:

compare_json_vs_tsv(response, only_show_excerpts=True)

----------------- Formatted JSON - First 335/1122 chars (30%) ------------------
```json
{
  "entities": [
    {
      "id": 0,
      "name": "Henry Jones Jr.",
      "label": "person"
    },
    {
      "id": 1,
      "name": "Henry Jones Sr.",
      "label": "person"
    },
    {
      "id": 2,
      "name": "Sophie Jones",
      "label": "person"
    },
    {
      "id": 3,
      "name": "William Smith",
     …
------------------- Compact JSON - First 335/665 chars (50%) -------------------
```json
{"entities":[{"id":0,"name":"Henry Jones Jr.","label":"person"},{"id":1,"name":"Henry Jones Sr.","label":"person"},{"id":2,"name":"Sophie Jones","label":"person"},{"id":3,"name":"William Smith","label":"person"},{"id":4,"name":"Beau","label":"animal"},{"id":5,"name":"Short Round","label":"person"}],"relationships":[{"source_i…
------------------------------- TSV (335 chars) --------------------------------
```tsv filename="entities.tsv"
id  name    label
0   Henry Jones Jr. person
1   Henry Jones Sr. person
2   Sophie Jones    person
3   William Smith   person
4   Beau    animal
5   Short Round person
```

```tsv filename="relationships.tsv"
source_id   link    target_id
0   child_of    1
2   child_of    0
3   friend_of   2
4   pet_of  3
5   friend_of   0
0   adopted 5
5   child_of    0
```

💡 Notice how much more information can be represented in the same number of text characters. This will apply similarly to token counts.

🧪 And now, let's compare the gains, especially for token counts:

compare_json_vs_tsv(response)

------------------------ Formatted JSON → Compact JSON -------------------------

	Chars	Tokens
Formatted JSON	1122	456
Compact JSON	665	216
Gain	40.7%	52.6%

------------------------------ Compact JSON → TSV ------------------------------

	Chars	Tokens
Compact JSON	665	216
TSV	335	137
Gain	49.6%	36.6%

----------------------------- Formatted JSON → TSV -----------------------------

	Chars	Tokens
Formatted JSON	1122	456
TSV	335	137
Gain	70.1%	70.0%

--------------------------------------------------------------------------------

💡 Savings in output tokens:

30 to 40% compared to compact JSON
60 to 70% compared to standard JSON

With a double-digit percentage reduction in output tokens, building knowledge graphs with TSV outputs is significantly faster (and cheaper)!

Now, let's finalize our code with optimized structures…

🚀 Finalization

🧩 Structure

First, it helps to define a structured prompt template, so we can focus on specific parts of our solution using a divide-and-conquer approach.

Here's a possible prompt template:

KNOWLEDGE_GRAPH_PROMPT_TEMPLATE = """
**Data Schema**

{data_schema}

**Instructions**

{instructions}

**Output Format**

{output_format}
"""

Then, here are some possible Entity, Relationship, and KnowledgeGraph data classes with the matching output format:

from dataclasses import dataclass, field


@dataclass
class Entity:
    id: int
    name: str
    label: str


@dataclass
class Relationship:
    source_id: int
    link: str
    target_id: int


@dataclass
class KnowledgeGraph:
    entities: list[Entity] = field(default_factory=list)
    relationships: list[Relationship] = field(default_factory=list)


TAB = "\t"
KNOWLEDGE_GRAPH_OUTPUT_FORMAT = f"""
Format the output strictly as two TSV code blocks (including the header row):

```tsv filename="entities.tsv"
id{TAB}name{TAB}label
[data_rows]
```

```tsv filename="relationships.tsv"
source_id{TAB}link{TAB}target_id
[data_rows]
```
"""

💡 While the Gen AI SDK natively supports Pydantic models for JSON structured outputs, we're using standard Python data classes here and TSV outputs to maximize our token efficiency.

ℹ️ If you use multiple entity or relationship data classes in your solution, you can dynamically generate the output format specification using features of the dataclasses package (like class docstrings and field descriptions).

Now, let's define a few helpers to generate a knowledge graph: 🔽

from dataclasses import fields, is_dataclass
from typing import get_args, get_origin, get_type_hints

def generate_knowledge_graph(
    data_schema: str,
    instructions: str,
    source: Source | str,
    *,
    model: Model | None = None,
    config: GenerateContentConfig | None = None,
    system_instruction: str | None = None,
    show_prompt: ShowAs = ShowAs.DONT_SHOW,
    show_response: ShowAs = ShowAs.DONT_SHOW,
) -> KnowledgeGraph:
    prompt = get_prompt_for_data_schema_and_instructions(data_schema, instructions)
    response = generate_content(
        prompt,
        source,
        model=model,
        config=config,
        system_instruction=system_instruction,
        show_prompt=show_prompt,
        show_response=show_response,
        return_response=True,
    )

    knowledge_graph = (
        parse_list_dataclass(KnowledgeGraph, response)
        if isinstance(response, GenerateContentResponse)
        else KnowledgeGraph()
    )
    display_knowledge_graph_info(knowledge_graph)

    return knowledge_graph


def show_knowledge_graph_prompt(
    data_schema: str,
    instructions: str,
    source: Source | str,
    *,
    model: Model | None = None,
    config: GenerateContentConfig | None = None,
    system_instruction: str | None = None,
    show_as: ShowAs = ShowAs.TEXT,
) -> None:
    prompt = get_prompt_for_data_schema_and_instructions(data_schema, instructions)
    generate_content(
        prompt,
        source,
        model=model,
        config=config,
        system_instruction=system_instruction,
        show_prompt=show_as,
        only_show_prompt=True,
    )


def get_prompt_for_data_schema_and_instructions(
    data_schema: str,
    instructions: str,
) -> str:
    return KNOWLEDGE_GRAPH_PROMPT_TEMPLATE.format(
        data_schema=data_schema.strip(),
        instructions=instructions.strip(),
        output_format=KNOWLEDGE_GRAPH_OUTPUT_FORMAT.strip(),
    ).strip()


def parse_list_dataclass[T](cls: type[T], response: GenerateContentResponse) -> T:
    assert is_dataclass(cls)
    if not (response_text := response.text):
        return cls()

    data = {}
    for f in fields(cls):
        origin, list_types = get_origin(f.type), get_args(f.type)
        assert (
            origin is list
        ), f"❌ Field {f.name} must be a list[dataclass] parameterized list"
        assert len(list_types) == 1, f"❌ Expected 1 single type: {len(list_types)=}"
        data[f.name] = parse_tsv_block(list_types[0], response_text, f.name)

    return cls(**data)


def parse_tsv_block[T](cls: type[T], data: str, tsv_filestem: str) -> list[T]:
    rows = []
    valid_fields = get_type_hints(cls)
    tsv_string = extract_tsv_block(data, tsv_filestem)
    for row in csv.DictReader(io.StringIO(tsv_string), delimiter="\t"):
        casted_data = {}
        for key, value in row.items():
            if key not in valid_fields or value is None:
                continue
            field_type = valid_fields[key]
            try:  # Note: Works only for directly castable types such as int, float, str, enum (e.g., not bool)
                casted_data[key] = field_type(value)
            except (ValueError, TypeError):
                print(f'❌ Could not cast "{value}" to {field_type} → Skipping {row=}')
                break
        else:
            try:
                rows.append(cls(**casted_data))
            except TypeError as e:
                print(f"❌ Could not instantiate {cls.__name__}: {e} → Skipping {row=}")

    return rows


def extract_tsv_block(data: str, filestem: str) -> str:
    pattern = rf'```
{% endraw %}
tsv filename="{re.escape(filestem)}.tsv"\s*\n(.*?)
{% raw %}
```'
    if not (match := re.search(pattern, data, flags=re.DOTALL)):
        print(f'❌ Could not find a TSV block for "{filestem=}"')
        return ""
    return match.group(1)


def display_knowledge_graph_info(kg: KnowledgeGraph) -> None:
    print_caption("Knowledge Graph Info")
    print(f"Entities      : {len(kg.entities):3,d}")
    print(f"Relationships : {len(kg.relationships):3,d}")
    print_separator()


print("✅ Knowledge graph generation helpers defined")

✅ Knowledge graph generation helpers defined

And here is a possible data schema with some instructions to generate a knowledge graph for our book analysis use case:

from enum import StrEnum, auto


class BookAnalysisEntityLabel(StrEnum):
    PERSON = auto()
    ANIMAL = auto()
    ORGANIZATION = auto()


def pipe_delimited_union(enum: type[StrEnum]) -> str:
    return "|".join(f"`{e.value}`" for e in enum)


BOOK_ANALYSIS_DATA_SCHEMA = f"""
Entity:
- `id`: Unique integer identifier (0, 1, 2…).
- `name`: Most complete name as exclusively determined from the input data.
- `label`: {pipe_delimited_union(BookAnalysisEntityLabel)}.

Relationship:
- `source_id`: `id` of the subject entity.
- `link`: `snake_case` predicate.
- `target_id`: `id` of the object entity.
"""

BOOK_ANALYSIS_INSTRUCTIONS = """
- Extract every distinct entity:
  - Treat distinct pseudonyms/identities as separate entities.
  - Include implied entities whose names can be exclusively determined from the input data.
- Extract every distinct relationship between them:
  - Use specific `link` predicates in `snake_case` as needed (e.g., `alias_of`, `son_of`, `fiancée_of`, `friend_of`, `murderer_of`, `employer_of`, `in_love_with`, `rival_of`).
  - If a relationship changes over time, make sure to include every distinct stage of the relationship.
  - For every asymmetric relationship extracted, make sure to include the logical inverse relationship (e.g., `A husband_of B` AND `B wife_of A`, `A employer_of B` AND `B employee_of A`).
  - For every symmetric relationship extracted, make sure to include both directions (e.g., `A friend_of B` AND `B friend_of A`).
"""

Verify the structured prompt:

show_knowledge_graph_prompt(
    BOOK_ANALYSIS_DATA_SCHEMA,
    BOOK_ANALYSIS_INSTRUCTIONS,
    text,
    show_as=ShowAs.TEXT,
)

------------------------------------ Prompt ------------------------------------
==Start of input data==
- Henry Jones is a famous archaeologist. He is actually a "Junior" because he is named after his father.
- Sophie is Henry's daughter, shares his last name, and works as a software engineer.
- William Smith is an aerospace engineer and Sophie's lifelong friend. Everybody calls him Bill and Beau is his dog.
- Short Round met Henry as a child. They first became close friends, and Henry officially adopted him a few years later.
- Sophie and Bill both work at Acme Aerospace.
==End of input data==
==Start of user prompt==
**Data Schema**

Entity:
- `id`: Unique integer identifier (0, 1, 2…).
- `name`: Most complete name as exclusively determined from the input data.
- `label`: `person`|`animal`|`organization`.

Relationship:
- `source_id`: `id` of the subject entity.
- `link`: `snake_case` predicate.
- `target_id`: `id` of the object entity.

**Instructions**

- Extract every distinct entity:
  - Treat distinct pseudonyms/identities as separate entities.
  - Include implied entities whose names can be exclusively determined from the input data.
- Extract every distinct relationship between them:
  - Use specific `link` predicates in `snake_case` as needed (e.g., `alias_of`, `son_of`, `fiancée_of`, `friend_of`, `murderer_of`, `employer_of`, `in_love_with`, `rival_of`).
  - If a relationship changes over time, make sure to include every distinct stage of the relationship.
  - For every asymmetric relationship extracted, make sure to include the logical inverse relationship (e.g., `A husband_of B` AND `B wife_of A`, `A employer_of B` AND `B employee_of A`).
  - For every symmetric relationship extracted, make sure to include both directions (e.g., `A friend_of B` AND `B friend_of A`).

**Output Format**

Format the output strictly as two TSV code blocks (including the header row):

```tsv filename="entities.tsv"
id  name    label
[data_rows]
```

```tsv filename="relationships.tsv"
source_id   link    target_id
[data_rows]
```
==End of user prompt==
--------------------------------------------------------------------------------

🧪 Let's generate a knowledge graph:

knowledge_graph = generate_knowledge_graph(
    BOOK_ANALYSIS_DATA_SCHEMA,
    BOOK_ANALYSIS_INSTRUCTIONS,
    text,
    show_response=ShowAs.TEXT,
)

print(knowledge_graph.entities)
print(knowledge_graph.relationships)

----------------------- Request / gemini-3.1-flash-lite ------------------------
Input tokens    :       534
Output tokens   :       244
------------------------------ Start of Response -------------------------------
```tsv filename="entities.tsv"
id  name    label
0   Henry Jones Jr. person
1   Henry Jones Sr. person
2   Sophie Jones    person
3   William Smith   person
4   Bill    person
5   Beau    animal
6   Short Round person
7   Acme Aerospace  organization
```

```tsv filename="relationships.tsv"
source_id   link    target_id
0   son_of  1
1   father_of   0
0   father_of   2
2   daughter_of 0
2   friend_of   3
3   friend_of   2
3   alias_of    4
4   alias_of    3
3   employee_of 7
7   employer_of 3
2   employee_of 7
7   employer_of 2
3   owner_of    5
5   pet_of  3
6   friend_of   0
0   friend_of   6
0   adopted_father_of   6
6   adopted_son_of  0
```
------------------------------- End of Response --------------------------------
----------------------------- Knowledge Graph Info -----------------------------
Entities      :   8
Relationships :  18
--------------------------------------------------------------------------------
[Entity(id=0, name='Henry Jones Jr.', label='person'), Entity(id=1, name='Henry Jones Sr.', label='person'), Entity(id=2, name='Sophie Jones', label='person'), Entity(id=3, name='William Smith', label='person'), Entity(id=4, name='Bill', label='person'), Entity(id=5, name='Beau', label='animal'), Entity(id=6, name='Short Round', label='person'), Entity(id=7, name='Acme Aerospace', label='organization')]
[Relationship(source_id=0, link='son_of', target_id=1), Relationship(source_id=1, link='father_of', target_id=0), Relationship(source_id=0, link='father_of', target_id=2), Relationship(source_id=2, link='daughter_of', target_id=0), Relationship(source_id=2, link='friend_of', target_id=3), Relationship(source_id=3, link='friend_of', target_id=2), Relationship(source_id=3, link='alias_of', target_id=4), Relationship(source_id=4, link='alias_of', target_id=3), Relationship(source_id=3, link='employee_of', target_id=7), Relationship(source_id=7, link='employer_of', target_id=3), Relationship(source_id=2, link='employee_of', target_id=7), Relationship(source_id=7, link='employer_of', target_id=2), Relationship(source_id=3, link='owner_of', target_id=5), Relationship(source_id=5, link='pet_of', target_id=3), Relationship(source_id=6, link='friend_of', target_id=0), Relationship(source_id=0, link='friend_of', target_id=6), Relationship(source_id=0, link='adopted_father_of', target_id=6), Relationship(source_id=6, link='adopted_son_of', target_id=0)]

💡 This is looking good!

Now, let's go to the next stage and build a network graph from our data…

🪢 Network graph

Now that we have our entities and relationships neatly packed into data classes, let's bring them to life. We'll use networkx to build a network graph. Using domain terminology, entities become nodes and relationships become directed edges. We'll also calculate node centralities to identify key nodes and use the Louvain method to detect communities (clusters of closely related nodes)…

Helpers 🔽

import textwrap
from typing import cast

import networkx as nx
import numpy as np
from networkx.algorithms.community.louvain import louvain_communities

NODE_CENTRALITY = "node_centrality"
NODE_COMMUNITY_INDEX = "node_community_index"
NODE_COLOR = "node_color"
EDGE_COLOR = "edge_color"
# Wrap names over multiple lines (avoids long node labels)
MULTILINE_NAMES = True
MULTILINE_CHARS = 12


def build_graph(kg: KnowledgeGraph, remove_orphan_nodes: bool) -> nx.DiGraph:
    # For simplicity, we build a DiGraph (adapt to MultiDiGraph if needed)
    graph = nx.DiGraph()

    # Nodes ← Entities
    node_name_from_id: dict[int, str] = {}
    for entity in kg.entities:
        node_name, display_name = get_node_and_display_names_for_entity(entity)
        node_name_from_id[entity.id] = node_name
        graph.add_node(node_name, name=display_name)

    # Edges ← Relationships
    for relationship in kg.relationships:
        source_node = node_name_from_id.get(relationship.source_id, "")
        target_node = node_name_from_id.get(relationship.target_id, "")
        if not source_node or not target_node:
            print(f"❌ Skipping relationship due to empty node:\n{relationship}")
            continue
        # For simplicity, each link has the same weight
        # For better community detection, define your own weight mappings (e.g., family members have stronger bonds)
        weight = 1
        edge_label = relationship.link
        if graph.has_edge(source_node, target_node):
            existing_data = graph[source_node][target_node]
            existing_data["link"] += f"\n{edge_label}"
            existing_data["weight"] += weight
        else:
            graph.add_edge(source_node, target_node, link=edge_label, weight=weight)

    if remove_orphan_nodes:
        graph.remove_nodes_from(list(nx.isolates(graph)))

    return graph


def get_node_and_display_names_for_entity(entity: Entity) -> tuple[str, str]:
    snake_case_name = "_".join(map(str.lower, entity.name.split()))
    node_name = f"{entity.id}_{snake_case_name}"

    display_name = entity.name
    if MULTILINE_NAMES:
        display_name = "\n".join(textwrap.wrap(display_name, width=MULTILINE_CHARS))

    return node_name, display_name


def color_gen(color_count: int) -> Iterator[str]:
    B50, R50, Y50, G50 = ("#4285F4", "#EA4335", "#FBBC04", "#34A853")
    B20, R20, Y20, G20 = ("#AECBFA", "#F6AEA9", "#FDE293", "#A8DAB5")
    B05, R05, Y05, G05 = ("#E8F0FE", "#FCE8E6", "#FEF7E0", "#E6F4EA")
    COLORS = [B50, R50, Y50, G50, B20, R20, Y20, G20, B05, R05, Y05, G05]
    for i in range(color_count):
        yield COLORS[i % len(COLORS)]


Node = str
Positions = dict[Node, np.ndarray]
Community = set[Node]
Communities = list[Community]
Nodes = list[Node]
INTER_COMMUNITY_EDGE_COLOR = "#8888"


def init_graph_data(graph: nx.Graph) -> Nodes:
    def node_centrality(node: Node) -> float:
        return graph.nodes[node][NODE_CENTRALITY]

    def community_max_centrality(community: Community) -> float:
        return max(node_centrality(node) for node in community)

    def nodes_sorted_by_community(communities: Communities) -> Nodes:
        entities = []
        for community in communities:
            sorted_entities = sorted(community, key=node_centrality, reverse=True)
            entities.extend(sorted_entities)
        return entities

    # Node centralities
    centralities = nx.betweenness_centrality(graph, endpoints=True)
    for node_key in graph.nodes:
        graph.nodes[node_key][NODE_CENTRALITY] = centralities[node_key]

    # Communities
    communities = cast(Communities, louvain_communities(graph, seed=42))
    sorted_communities = sorted(communities, key=community_max_centrality, reverse=True)

    # Community colors
    community_count = len(sorted_communities)
    community_colors = list(color_gen(community_count))

    # Node community indices and colors
    for community_index, community in enumerate(sorted_communities):
        for node_key in community:
            node = graph.nodes[node_key]
            node[NODE_COMMUNITY_INDEX] = community_index
            node[NODE_COLOR] = community_colors[community_index]

    # Edge colors
    for node_key_i, node_key_j, edge_data in graph.edges(data=True):
        node_i = graph.nodes[node_key_i]
        node_j = graph.nodes[node_key_j]
        same_community = node_i[NODE_COMMUNITY_INDEX] == node_j[NODE_COMMUNITY_INDEX]
        edge_data[EDGE_COLOR] = (
            node_i[NODE_COLOR] if same_community else INTER_COMMUNITY_EDGE_COLOR
        )

    return nodes_sorted_by_community(sorted_communities)


def compute_node_positions(graph: nx.DiGraph, entities: Nodes) -> Positions:
    # Derive an undirected graph (preserving the entity order)
    undirected_graph = nx.Graph()
    for entity in entities:
        undirected_graph.add_node(entity)
    undirected_graph.add_edges_from(graph.edges())

    if len(entities) < 10:
        positions = nx.circular_layout(undirected_graph)
    else:
        positions = nx.kamada_kawai_layout(undirected_graph)
        positions = nx.arf_layout(undirected_graph, positions, seed=42)

    return positions


@dataclass
class GraphData:
    knowledge_graph: KnowledgeGraph
    remove_orphan_nodes: bool = True
    graph: nx.DiGraph = field(init=False)
    nodes: Nodes = field(init=False)
    positions: Positions = field(init=False)

    def __post_init__(self) -> None:
        self.graph = build_graph(self.knowledge_graph, self.remove_orphan_nodes)
        self.nodes = init_graph_data(self.graph)
        self.positions = compute_node_positions(self.graph, self.nodes)


print("✅ Network graph helpers defined")

✅ Network graph helpers defined

Let's test this:

graph_data = GraphData(knowledge_graph)

print(f"{graph_data.graph = !s}")
print(f"{graph_data.nodes = }")

graph_data.graph = DiGraph with 8 nodes and 16 edges
graph_data.nodes = ['2_sophie_jones', '3_william_smith', '7_acme_aerospace', '5_beau', '4_bill', '0_henry_jones_jr.', '1_henry_jones_sr.', '6_short_round']

🎨 Data visualization

The extracted data is much easier to understand when you can actually see it! We can use matplotlib to draw our network graphs. We'll size the nodes based on their centrality and color-code them by community. To make it even easier to digest, we'll generate an animated sequence highlighting each character's connections one by one…

Let's define our data visualization helpers: 🔽

import typing
from io import BytesIO

import matplotlib.pyplot as plt
from IPython import display
from matplotlib.axes import Axes
from matplotlib.backends.backend_agg import FigureCanvasAgg
from matplotlib.figure import Figure
from PIL import Image as PilImage


class AnimationFormat(StrEnum):
    # Matches PIL.Image.Image.format
    WEBP = auto()
    PNG = auto()
    GIF = auto()


FIGURE_DPI = 200
FIGURE_FACTOR = 1.0
ANIMATION_INTRO_DURATION = 2500
ANIMATION_FRAME_DURATION = 250
EDGE_STYLE = "arc3,rad=0.2"
NodeSizes = dict[Node, int]

if running_in_colab_env():
    ANIMATION_FORMAT = AnimationFormat.WEBP
else:
    ANIMATION_FORMAT = AnimationFormat.GIF


def init_figure(title: str, subtitle: str) -> tuple[Figure, Axes]:
    figsize = (16 * FIGURE_FACTOR, 9 * FIGURE_FACTOR)
    fig, ax = plt.subplots(figsize=figsize, dpi=FIGURE_DPI)
    ax.set_title(title, loc="left")
    ax.set_title(subtitle, loc="right")
    ax.axis("off")
    fig.tight_layout(pad=2)

    return fig, ax


def draw_nodes(graph: nx.Graph, positions: Positions, ax: Axes) -> NodeSizes:
    node_view = graph.nodes(data=True)
    node_sizes = {
        node: max(500, int(10000 * data[NODE_CENTRALITY])) for node, data in node_view
    }
    node_colors = [str(data[NODE_COLOR]) for _, data in node_view]
    border_width = 3.0

    nx.draw_networkx_nodes(
        graph,
        pos=positions,
        node_size=list(node_sizes.values()),
        node_color=node_colors,
        alpha=0.95,
        ax=ax,
        linewidths=border_width,
    )
    labels = {node: data.get("name", str(node)) for node, data in node_view}
    nx.draw_networkx_labels(graph, positions, labels=labels, ax=ax)

    if len(node_sizes) >= 10:
        # Extend axis limits (default rounding changes them too much otherwise)
        (x_min, x_max), (y_min, y_max) = ax.get_xlim(), ax.get_ylim()
        x_min, x_max = int(x_min - 1.0), int(x_max + 1.0)
        y_min, y_max = int(y_min - 1.0), int(y_max + 1.0)
        ax.set_xlim(x_min, x_max)
        ax.set_ylim(y_min, y_max)

    return node_sizes


def draw_edges(
    graph: nx.DiGraph,
    positions: Positions,
    node_sizes: NodeSizes,
    ax: Axes,
    *,
    focused_node: Node | None = None,
) -> None:
    # Draw specific out-edges or all edges
    if focused_node:
        out_edges = graph.edges([focused_node], data=True)
    else:
        out_edges = graph.edges(data=True)
    edge_colors = [data[EDGE_COLOR] for _, _, data in out_edges]

    edge_list = [(u, v) for u, v, _ in out_edges]
    ordered_sizes = [node_sizes[n] for n in graph.nodes()]
    nx.draw_networkx_edges(
        graph,
        positions,
        edge_list,
        edge_color=edge_colors,
        style=":",
        alpha=0.9,
        arrowstyle="-|>",
        arrowsize=20,
        ax=ax,
        node_size=ordered_sizes,
        connectionstyle=EDGE_STYLE,
    )

    edge_labels = {(u, v): data["link"] for u, v, data in out_edges}
    nx.draw_networkx_edge_labels(
        graph,
        positions,
        edge_labels,
        font_size=8,
        font_family="monospace",
        bbox=dict(ec="#FFF8", fc="#FFF8"),
        ax=ax,
        node_size=ordered_sizes,  # type: ignore
        connectionstyle=EDGE_STYLE,  # type: ignore
    )


def display_graph(title: str, subtitle: str, graph_data: GraphData) -> None:
    _, ax = init_figure(title, subtitle)

    node_graph = graph_data.graph
    edge_graph = graph_data.graph
    positions = graph_data.positions
    node_sizes = draw_nodes(node_graph, positions, ax)
    draw_edges(edge_graph, positions, node_sizes, ax)

    plt.show()


def yield_images(
    title: str,
    subtitle: str,
    graph_data: GraphData,
) -> Iterator[PilImage.Image]:
    fig, ax = init_figure(title, subtitle)
    canvas = FigureCanvasAgg(fig)

    positions = graph_data.positions
    node_graph = graph_data.graph
    node_sizes = draw_nodes(node_graph, positions, ax)
    edge_graph = graph_data.graph

    for focused_node in [None, *graph_data.nodes]:
        if focused_node is not None:
            draw_edges(edge_graph, positions, node_sizes, ax, focused_node=focused_node)
        canvas.draw()
        image_size = canvas.get_width_height()
        image_bytes = canvas.buffer_rgba()
        yield PilImage.frombytes("RGBA", image_size, image_bytes).convert("RGB")
    plt.close(fig)


def generate_animation(
    title: str,
    subtitle: str,
    graph_data: GraphData,
    format: AnimationFormat,
) -> BytesIO:
    frames = list(yield_images(title, subtitle, graph_data))
    assert len(frames) >= 1

    if format == AnimationFormat.GIF:
        # Dither all frames with the same palette to optimize the animation
        # The animation is cumulative, so most colors are in the last frame
        method = PilImage.Quantize.MEDIANCUT
        palettized = frames[-1].quantize(method=method)
        frames = [frame.quantize(method=method, palette=palettized) for frame in frames]

    # The animation will be played in a loop: start cycling with the most complete frame
    first_frame = frames[-1]
    next_frames = frames[:-1]
    durations = [ANIMATION_INTRO_DURATION]
    durations += [ANIMATION_FRAME_DURATION] * len(next_frames)
    params: dict[str, typing.Any] = dict(
        save_all=True,
        append_images=next_frames,
        duration=durations,
        loop=0,
    )
    match format:
        case AnimationFormat.GIF:
            params.update(optimize=False)
        case AnimationFormat.PNG:
            params.update(optimize=True)
        case AnimationFormat.WEBP:
            params.update(lossless=True)

    image_io = BytesIO()
    first_frame.save(image_io, str(format).upper(), **params)

    return image_io


def display_graph_animation(title: str, subtitle: str, graph_data: GraphData) -> None:
    image_io = generate_animation(title, subtitle, graph_data, ANIMATION_FORMAT)
    ipython_image = display.Image(data=image_io.getvalue())
    display.display(ipython_image)


def get_graph_title_and_subtitle(
    source: Source | str,
    model: Model | None = None,
    domain: str = "",
) -> tuple[str, str]:
    if isinstance(source, str):
        source_name = f'"{source.strip()[:20]}…"'
    else:
        source_name = source.name
    model_id = model.value if model else Model.DEFAULT.value
    model_id = model_id.removesuffix("-preview")
    separator = " • "
    title_parts = ["Knowledge Graph"]
    if domain:
        title_parts.append(domain)
    subtitle_parts = [source_name, model_id]
    return separator.join(title_parts), separator.join(subtitle_parts)


def display_knowledge_graph(
    knowledge_graph: KnowledgeGraph,
    source: Source | str,
    model: Model | None = None,
    animated: bool = False,
    domain: str = "",
    remove_orphan_nodes: bool = True,
) -> None:
    if not knowledge_graph.entities:
        return
    title, subtitle = get_graph_title_and_subtitle(source, model, domain)
    graph_data = GraphData(knowledge_graph, remove_orphan_nodes)

    if animated:
        display_graph_animation(title, subtitle, graph_data)
    else:
        display_graph(title, subtitle, graph_data)


def extract_knowledge_graph(
    data_schema: str,
    instructions: str,
    source: Source | str,
    model: Model | None = None,
    *,
    domain: str = "",
    animated: bool = False,
) -> None:
    if isinstance(source, Source):
        display_input_data_caption(source)
    knowledge_graph = generate_knowledge_graph(
        data_schema,
        instructions,
        source,
        model=model,
    )
    display_knowledge_graph(knowledge_graph, source, model, animated, domain)


print("✅ Data visualization helpers defined")

✅ Data visualization helpers defined

Let's test this:

display_knowledge_graph(knowledge_graph, text)

💡 We can now quickly visualize and understand our knowledge graphs. This helps us iterate faster when refining prompts.

ℹ️ While this simple approach is great for a quick overview, you might want to swap it out for more specialized libraries if you want to explore the graph interactively.

✅ Challenge completed

Let's define a book analysis function:

def analyze_book(
    source: Source,
    model: Model | None = None,
    animated: bool = False,
) -> None:
    extract_knowledge_graph(
        BOOK_ANALYSIS_DATA_SCHEMA,
        BOOK_ANALYSIS_INSTRUCTIONS,
        source,
        model,
        domain="Character Connections",
        animated=animated,
    )

🧪 First, let's build a knowledge graph based on Zola's Thérèse Raquin:

analyze_book(Classic.fr_zola_thérèse_raquin)

Input data (fr_zola_thérèse_raquin)

----------------------- Request / gemini-3.1-flash-lite ------------------------
Input tokens    :   119,518
Cached tokens   :   118,753
Output tokens   :       357
----------------------------- Knowledge Graph Info -----------------------------
Entities      :  11
Relationships :  28
--------------------------------------------------------------------------------

💡 We can extract and understand the book's narrative in seconds: The love triangle between Thérèse, Camille, and Laurent clearly stands out. Despite being over 200 pages long (and 100k+ tokens), this novel is incredibly minimalistic, revolving around a limited set of characters, which is reflected in the network graph. Adding locations to the extracted entities would also reveal the claustrophobic atmosphere of the book in the resulting knowledge graph.

ℹ️ We used the original French version with English instructions, which works seamlessly. If you translate the instructions, you can also generate dynamic relationship links in different languages (see the 100+ supported languages).

🧪 Let's see how the model handles the interconnected cast of Hugo's Les Misérables:

analyze_book(
    Classic.en_hugo_les_misérables,
    model=Model.GEMINI_3_5_FLASH,
)

Input data (en_hugo_les_misérables)

-------------------------- Request / gemini-3.5-flash --------------------------
Input tokens    :   803,328
Output tokens   :     2,908
----------------------------- Knowledge Graph Info -----------------------------
Entities      :  65
Relationships : 225
--------------------------------------------------------------------------------

💡 We quickly get the gist of the novel.

ℹ️ Donald Knuth famously used Les Misérables as an example back in 1994 (see the Stanford GraphBase). It's been consistently used in Natural Language Processing (NLP) because of its massive scale, linguistic complexity, and multiple high-quality translations. At 800k+ tokens, this book is still a true stress test. Note that we used Gemini 3.5 Flash (instead of 3.1 Flash-Lite). Larger models can infer more and perform deeper consolidation.

🧪 Now, let's process just the first volume of Le Comte de Monte-Cristo (in French):

analyze_book(Classic.fr_dumas_comte_de_monte_cristo_1)

Input data (fr_dumas_comte_de_monte_cristo_1)

----------------------- Request / gemini-3.1-flash-lite ------------------------
Input tokens    :   215,582
Output tokens   :       713
----------------------------- Knowledge Graph Info -----------------------------
Entities      :  36
Relationships :  36
--------------------------------------------------------------------------------

💡 The graph shows the initial setup of Edmond Dantès' world, his friends, and his betrayers. The love triangle between Edmond, Mercédès, and Fernand also comes to light.

ℹ️ Note that the antagonist is referred to only as Fernand, which is expected. We learn his full name, Fernand Mondego, only in volume three. In the prompt, we asked the model to "extract" entities and determine names "from the input data" to avoid generating memorized knowledge. To detect regressions when updating the prompt, we can use this as a unit test to ensure that our response actually provides extracted data and not memorized information.

🧪 And finally, let's analyze the entire saga of Le Comte de Monte-Cristo, all four volumes at once (800k+ tokens):

analyze_book(
    Collection.fr_dumas_comte_de_monte_cristo,
    model=Model.GEMINI_3_5_FLASH,
    animated=True,
)

Input data (fr_dumas_comte_de_monte_cristo_1, fr_dumas_comte_de_monte_cristo_2, fr_dumas_comte_de_monte_cristo_3, fr_dumas_comte_de_monte_cristo_4)

-------------------------- Request / gemini-3.5-flash --------------------------
Input tokens    :   840,385
Cached tokens   :   835,551
Output tokens   :     2,659
----------------------------- Knowledge Graph Info -----------------------------
Entities      :  50
Relationships : 212
--------------------------------------------------------------------------------

💡 The multiple aliases of Edmond Dantès (and other characters) are nicely extracted. The complex plot of this book is built on characters juggling multiple identities in a psychological chess match.

🎉 This is an example of how we can complete the challenge in a single request, using the fewest tokens possible and zero thinking tokens, while providing multiple levels of consolidation.

ℹ️ Note that this is a proof of concept. For an exhaustive extraction in a fully professional solution, we would probably set up a multi-step workflow and consider the following:

As a preliminary step, cache the input data (cached tokens get a 90% discount; learn more about context caching).
In the first request, based on the cached content, focus on extracting the entities (possibly extending the entity labels with location, date…), and then save them to a database.
In the second request, based on the cached content and the entity table, focus on extracting the stated relationships using additional fields (such as evidence or excerpt to serve as direct proof, or chapter or page for source attribution). In this step, we could also dig deeper and differentiate literal/figurative relationships (e.g., biological vs. figurative parents).
In subsequent requests, focus on additional extractions (e.g., interactions between entities) or specific consolidations (e.g., adding inverse relationships, classifying the relationships or interactions into categories).

🔭 Generalization

With our current setup, generalizing to other types of content is as simple as adapting our data schema and instructions.

For example, legal agreements are among the densest types of documents. They're usually made up of many articles and clauses, with every sentence carrying specific legal weight, outlining obligations, or providing definitions. But what kind of knowledge graph do we want to build from a legal agreement?

🧪 Let's test a minimalistic, open-ended prompt:

source = Document.en_pharma_dev_agreement
model = Model.GEMINI_3_5_FLASH

AGREEMENT_OPEN_DATA_SCHEMA = """
Entity:
- `id`: Unique integer identifier (0, 1, 2…).
- `name`: Name of the entity.
- `label`: Entity type.

Relationship:
- `source_id`: `id` of the subject entity.
- `link`: `snake_case` predicate.
- `target_id`: `id` of the object entity.
"""

AGREEMENT_OPEN_INSTRUCTIONS = """
Perform a comprehensive, highly-granular entity/relationship extraction.
"""

extract_knowledge_graph(
    AGREEMENT_OPEN_DATA_SCHEMA,
    AGREEMENT_OPEN_INSTRUCTIONS,
    source,
    model,
    domain="Agreement High-Level Extraction",
)

Input data (en_pharma_dev_agreement)

-------------------------- Request / gemini-3.5-flash --------------------------
Input tokens    :    51,276
Output tokens   :       221
----------------------------- Knowledge Graph Info -----------------------------
Entities      :  10
Relationships :   9
--------------------------------------------------------------------------------

💡 Remarks

Notice how we just passed a PDF directly! Gemini processes the document natively, automatically extracting text and images while performing OCR along the way.
This minimalistic prompt offers the greatest degree of freedom, as the model can't guess exactly what data you want or how you plan to use it. As a result, only high-level entities and relationships are extracted, providing a nice summary of the agreement.
LLMs are trained to synthesize information. With such highly open-ended instructions, the default behavior helps avoid generating unnecessary tokens.

🧪 Then, let's try semi-open instructions focusing on legal obligations:

class AgreementEntityLabel(StrEnum):
    PARTY = auto()
    PERSON = auto()
    ROLE = auto()
    LOCATION = auto()
    JURISDICTION = auto()
    FINANCIAL_AMOUNT = auto()
    DATE = auto()
    EVENT = auto()
    ASSET = auto()
    PRODUCT = auto()
    INTELLECTUAL_PROPERTY = auto()
    OBLIGATION_TYPE = auto()


AGREEMENT_SEMI_OPEN_DATA_SCHEMA = f"""
Entity:
- `id`: Unique integer identifier (0, 1, 2…).
- `name`: Name of the entity.
- `label`: {pipe_delimited_union(AgreementEntityLabel)}.

Relationship: Obligation, right, or transfer from a source entity to a target entity.
- `source_id`: The `id` of the subject entity.
- `link`: Specific `snake_case` predicate.
- `target_id`: The `id` of the object entity.
"""

AGREEMENT_SEMI_OPEN_INSTRUCTIONS = """
- Analyze the input data for every covenant (e.g., "shall", "will", "must", "is obligated to") and perform an exhaustive extraction.
- Make sure to deconstruct complex obligations: For complex clauses (e.g., "A shall pay B $X Million within Y days of the Effective Date"), extract:
  - The primary obligation: `[A] is_obligated_to_pay [B]`
  - The value: `[A's obligation] subject_to [$X Million]`
  - The trigger: `[A's obligation] triggered_by [Effective Date]`
"""

extract_knowledge_graph(
    AGREEMENT_SEMI_OPEN_DATA_SCHEMA,
    AGREEMENT_SEMI_OPEN_INSTRUCTIONS,
    source,
    model,
    domain="Agreement Obligations",
)

Input data (en_pharma_dev_agreement)

-------------------------- Request / gemini-3.5-flash --------------------------
Input tokens    :    51,466
Output tokens   :     2,002
----------------------------- Knowledge Graph Info -----------------------------
Entities      :  85
Relationships :  93
--------------------------------------------------------------------------------

💡 Remarks

This semi-open prompt lists the types of entities to extract, which naturally increases the number of extracted entities and their specificity.
The resulting graph is denser and reflects the legal complexity of the document, rather than just giving us a high-level summary.
The extracted relationships are still rather high-level due to how open-ended this part of our prompt is. For such specific extractions, it's possible to extend the entity and relationship data schemas with additional fields, or even to request specific tabular outputs.

What if we don't care about legal obligations, but rather the document's architecture? Let's shift our focus to the structure itself and extract how sections, clauses, and defined terms are hierarchically organized…

🧪 And now, let's test closed instructions focusing on the document structure:

class AgreementStructureEntityLabel(StrEnum):
    DEFINED_TERM = auto()
    DOCUMENT_SECTION = auto()
    DOCUMENT = auto()


class AgreementStructureRelationshipType(StrEnum):
    DEFINED_IN = auto()
    CONTAINS = auto()


AGREEMENT_STRUCTURAL_DATA_SCHEMA = f"""
Entity:
- `id`: Unique integer identifier (0, 1, 2…).
- `name`: Name of the entity.
- `label`: {pipe_delimited_union(AgreementStructureEntityLabel)}.

Relationship: Connection from a source entity to a target entity.
- `source_id`: The `id` of the subject entity.
- `link`: {pipe_delimited_union(AgreementStructureRelationshipType)}.
- `target_id`: The `id` of the object entity.
"""

AGREEMENT_STRUCTURAL_OPEN_INSTRUCTIONS = """
- Extract every distinct entity that matches an allowed `label`.
- Extract every distinct relationship representing a structural connection (hierarchical organization) between these entities:
  - You must be comprehensive and highly granular. If multiple distinct relationships exist between the same pair of entities, create a separate entry for each.
"""

extract_knowledge_graph(
    AGREEMENT_STRUCTURAL_DATA_SCHEMA,
    AGREEMENT_STRUCTURAL_OPEN_INSTRUCTIONS,
    source,
    model,
    domain="Agreement Structure",
)

Input data (en_pharma_dev_agreement)

-------------------------- Request / gemini-3.5-flash --------------------------
Input tokens    :    51,351
Output tokens   :     9,108
----------------------------- Knowledge Graph Info -----------------------------
Entities      : 314
Relationships : 513
--------------------------------------------------------------------------------

💡 If you extract hundreds of entities from a massive document, your graph will quickly turn into an unreadable hairball. For larger datasets, you'll want to export your nodes and edges to a dedicated graph database, which typically comes with its own visualization and exploration tools.

🏁 Conclusion

We successfully extracted data and built knowledge graphs from documents by following these steps:

Prototyping with open prompts to develop an intuition for the model's natural strengths
Crafting increasingly specific prompts using a tabular-extraction strategy
Structuring our inputs to move towards production-ready and generalizable code
Structuring and optimizing our outputs for faster and cheaper generation
Adding data visualization for easier interpretation of responses and smoother iterations
Conducting more tests, iterating, and enriching the extracted data

These principles apply to many other data-extraction domains and will allow you to solve your own complex problems. Have fun and happy solving!

➕ More!

Try it yourself: Use the companion notebook (or run the notebook on Colab) to reproduce all results in this article.
Get inspired: Explore typical use cases in the Agent Platform Prompt Gallery.
Stay updated: Follow the Agent Platform Release Notes.
Follow me: Connect with me (@PicardParis) on LinkedIn or Twitter / X for more cloud, applied AI, and Python explorations…

Detecting and Editing Visual Objects with Gemini

Laurent Picard — Tue, 03 Mar 2026 17:00:11 +0000

A few notes before we start:

The complete source code for this article, including future updates, is available in this notebook under the Apache 2.0 license.
All new images in this article were generated with Gemini Nano Banana using the explored proof-of-concept. All source images are either in the public domain or free to use (reference links are provided in the code output).
You can experiment with Gemini models for free in Google AI Studio. For programmatic API access, please note that while a free tier is available for some models (i.e., you can perform object detection), image generation is a pay-as-you-go service.

✨ Overview

Traditional computer vision models are typically trained to detect a fixed set of object classes, like "person", "cat", or "car". If you want to detect something specific that wasn't in the training set, such as an "illustration" in a book photograph, you usually have to gather a dataset, label it manually, and train a custom model, which can take hours or even days.

In this exploration, we'll test a different approach using Gemini. We will leverage its spatial understanding capabilities to perform open-vocabulary object detection. This allows us to find objects based solely on a natural language description, without any training.

Once the visual objects are detected, we'll extract them and then use Gemini's image editing capabilities (specifically the Nano Banana models) to restore and creatively transform them.

🔥 Challenge

We are dealing with unstructured data: photos of books, magazines, and objects in the wild. These images present several difficulties for traditional computer vision:

Variety: The objects we want to find (illustrations, engravings, and any visuals in general) vary wildly in style and content.
Distortion: Pages are curved, photos are taken at angles, and lighting is uneven.
Noise: Old books have stains, paper grain, and text bleeding through from the other side.

Our challenge is to build a robust pipeline that can detect these objects despite the distortions, extract them cleanly, and edit them to look like high-quality digital assets… all using simple text prompts.

🏁 Setup

🐍 Python packages

We'll use the following packages:

google-genai: the Google Gen AI Python SDK lets us call Gemini with a few lines of code
pillow for image management
matplotlib for result visualization

We'll also use these packages (dependencies of google-genai):

pydantic for data management
tenacity for request management

pip install --quiet "google-genai>=1.64.0" "pillow>=11.3.0" "matplotlib>=3.10.0"

🔗 Gemini API

To use the Gemini API, we have two main options:

Via Vertex AI with a Google Cloud project
Via Google AI Studio with a Gemini API key

The Google Gen AI SDK provides a unified interface to these APIs, and we can use environment variables for the configuration. 🔽

🛠️ Option 1 - Gemini API via Vertex AI

Requirements:

A Google Cloud project
The Vertex AI API must be enabled for this project: ▶️ Enable the Vertex AI API

Gen AI SDK environment variables:

GOOGLE_GENAI_USE_VERTEXAI="True"
GOOGLE_CLOUD_PROJECT="<PROJECT_ID>"
GOOGLE_CLOUD_LOCATION="<LOCATION>"

💡 For preview models, the location must be set to global. For generally available models, we can choose the closest location among the Google model endpoint locations.

ℹ️ Learn more about setting up a project and a development environment.

🛠️ Option 2 - Gemini API via Google AI Studio

Requirement:

A Gemini API key

Gen AI SDK environment variables:

GOOGLE_GENAI_USE_VERTEXAI="False"
GOOGLE_API_KEY="<API_KEY>"

ℹ️ Learn more about getting a Gemini API key from Google AI Studio.

💡 You can store your environment configuration outside of the source code:

Environment	Method
IDE	`.env` file (or equivalent)
Colab	Colab Secrets (🗝️ icon in left panel, see code below)
Colab Enterprise	Google Cloud project and location are automatically defined
Vertex AI Workbench	Google Cloud project and location are automatically defined

Define the following environment detection functions. You can also define your configuration manually if needed. 🔽

# @title {display-mode: "form"}
import os
import sys
from collections.abc import Callable

from google import genai

# Manual setup (leave unchanged if setup is environment-defined)

# @markdown **Which API: Vertex AI or Google AI Studio?**
GOOGLE_GENAI_USE_VERTEXAI = True  # @param {type: "boolean"}

# @markdown **Option A - Google Cloud project [+location]**
GOOGLE_CLOUD_PROJECT = ""  # @param {type: "string"}
GOOGLE_CLOUD_LOCATION = "global"  # @param {type: "string"}

# @markdown **Option B - Google AI Studio API key**
GOOGLE_API_KEY = ""  # @param {type: "string"}


def check_environment() -> bool:
    check_colab_user_authentication()
    return check_manual_setup() or check_vertex_ai() or check_colab() or check_local()


def check_manual_setup() -> bool:
    return check_define_env_vars(
        GOOGLE_GENAI_USE_VERTEXAI,
        GOOGLE_CLOUD_PROJECT.strip(),  # Might have been pasted with line return
        GOOGLE_CLOUD_LOCATION,
        GOOGLE_API_KEY,
    )


def check_vertex_ai() -> bool:
    # Workbench and Colab Enterprise
    match os.getenv("VERTEX_PRODUCT", ""):
        case "WORKBENCH_INSTANCE":
            pass
        case "COLAB_ENTERPRISE":
            if not running_in_colab_env():
                return False
        case _:
            return False

    return check_define_env_vars(
        True,
        os.getenv("GOOGLE_CLOUD_PROJECT", ""),
        os.getenv("GOOGLE_CLOUD_REGION", ""),
        "",
    )


def check_colab() -> bool:
    if not running_in_colab_env():
        return False

    # Colab Enterprise was checked before, so this is Colab only
    from google.colab import auth as colab_auth  # type: ignore

    colab_auth.authenticate_user()

    # Use Colab Secrets (🗝️ icon in left panel) to store the environment variables
    # Secrets are private, visible only to you and the notebooks that you select
    # - Vertex AI: Store your settings as secrets
    # - Google AI: Directly import your Gemini API key from the UI
    vertexai, project, location, api_key = get_vars(get_colab_secret)

    return check_define_env_vars(vertexai, project, location, api_key)


def check_local() -> bool:
    vertexai, project, location, api_key = get_vars(os.getenv)

    return check_define_env_vars(vertexai, project, location, api_key)


def running_in_colab_env() -> bool:
    # Colab or Colab Enterprise
    return "google.colab" in sys.modules


def check_colab_user_authentication() -> None:
    if running_in_colab_env():
        from google.colab import auth as colab_auth  # type: ignore

        colab_auth.authenticate_user()


def get_colab_secret(secret_name: str, default: str) -> str:
    from google.colab import errors, userdata  # type: ignore

    try:
        return userdata.get(secret_name)
    except errors.SecretNotFoundError:
        return default


def disable_colab_cell_scrollbar() -> None:
    if running_in_colab_env():
        from google.colab import output  # type: ignore

        output.no_vertical_scroll()


def get_vars(getenv: Callable[[str, str], str]) -> tuple[bool, str, str, str]:
    # Limit getenv calls to the minimum (may trigger UI confirmation for secret access)
    vertexai_str = getenv("GOOGLE_GENAI_USE_VERTEXAI", "")
    if vertexai_str:
        vertexai = vertexai_str.lower() in ["true", "1"]
    else:
        vertexai = bool(getenv("GOOGLE_CLOUD_PROJECT", ""))

    project = getenv("GOOGLE_CLOUD_PROJECT", "") if vertexai else ""
    location = getenv("GOOGLE_CLOUD_LOCATION", "") if project else ""
    api_key = getenv("GOOGLE_API_KEY", "") if not project else ""

    return vertexai, project, location, api_key


def check_define_env_vars(
    vertexai: bool,
    project: str,
    location: str,
    api_key: str,
) -> bool:
    match (vertexai, bool(project), bool(location), bool(api_key)):
        case (True, True, _, _):
            # Vertex AI - Google Cloud project [+location]
            location = location or "global"
            define_env_vars(vertexai, project, location, "")
        case (True, False, _, True):
            # Vertex AI - API key
            define_env_vars(vertexai, "", "", api_key)
        case (False, _, _, True):
            # Google AI Studio - API key
            define_env_vars(vertexai, "", "", api_key)
        case _:
            return False

    return True


def define_env_vars(vertexai: bool, project: str, location: str, api_key: str) -> None:
    os.environ["GOOGLE_GENAI_USE_VERTEXAI"] = str(vertexai)
    os.environ["GOOGLE_CLOUD_PROJECT"] = project
    os.environ["GOOGLE_CLOUD_LOCATION"] = location
    os.environ["GOOGLE_API_KEY"] = api_key


def check_configuration(client: genai.Client) -> None:
    service = "Vertex AI" if client.vertexai else "Google AI Studio"
    print(f"✅ Using the {service} API", end="")

    if client._api_client.project:
        print(f' with project "{client._api_client.project[:7]}…"', end="")
        print(f' in location "{client._api_client.location}"')
    elif client._api_client.api_key:
        api_key = client._api_client.api_key
        print(f' with API key "{api_key[:5]}…{api_key[-5:]}"', end="")
        print(f" (in case of error, make sure it was created for {service})")


print("✅ Environment functions defined")

🤖 Gen AI SDK

To send Gemini requests, create a google.genai client:

from google import genai

check_environment()

client = genai.Client()

check_configuration(client)

🖼️ Image test suite

Let's define a list of images for our tests: 🔽

from dataclasses import dataclass
from enum import StrEnum

Url = str


class Source(StrEnum):
    incunable = "https://tile.loc.gov/image-services/iiif/service:rbc:rbc0001:2014:2014rosen0487:0165/full/pct:25/0/default.jpg"
    engravings = "https://tile.loc.gov/image-services/iiif/service:gdc:gdcscd:00:34:07:66:92:1:00340766921:0121/full/pct:50/0/default.jpg"
    museum_guidebook = "https://tile.loc.gov/image-services/iiif/service:rbc:rbc0001:2014:2014gen34181:0033/full/pct:75/0/default.jpg"
    denver_illustrated = "https://tile.loc.gov/image-services/iiif/service:gdc:gdclccn:rc:01:00:04:94:rc01000494:0051/full/pct:50/0/default.jpg"
    physics_textbook = "https://tile.loc.gov/image-services/iiif/service:gdc:gdcscd:00:03:64:87:31:8:00036487318:0103/full/pct:50/0/default.jpg"
    portrait_miniatures = "https://tile.loc.gov/image-services/iiif/service:rbc:rbc0001:2024:2024rosen013592v02:0249/full/pct:50/0/default.jpg"
    wizard_of_oz_drawings = "https://tile.loc.gov/image-services/iiif/service:rbc:rbc0001:2006:2006gen32405:0048/full/pct:25/0/default.jpg"
    paintings = "https://images.unsplash.com/photo-1714146681164-f26fed839692?h=1440"
    alice_drawing = "https://images.unsplash.com/photo-1630595011903-689853b04ee2?h=800"
    book = "https://images.unsplash.com/photo-1643451533573-ee364ba6e330?h=800"
    manual = "https://images.unsplash.com/photo-1623666936367-a100f62ba9b7?h=800"
    electronics = "https://images.unsplash.com/photo-1757397584789-8b2c5bfcdbc3?h=1440"


@dataclass
class SourceMetadata:
    title: str
    webpage_url: Url
    credit_line: str


LOC = "Library of Congress"
LOC_RARE_BOOKS = "Library of Congress, Rare Book and Special Collections Division"
LOC_MEETING_FRONTIERS = "Library of Congress, Meeting of Frontiers"

metadata_by_source: dict[Source, SourceMetadata] = {
    Source.incunable: SourceMetadata(
        "Vergaderinge der historien van Troy (1485)",
        "https://www.loc.gov/resource/rbc0001.2014rosen0487/?sp=165",
        LOC_RARE_BOOKS,
    ),
    Source.engravings: SourceMetadata(
        "Harper's illustrated catalogue (1847)",
        "https://www.loc.gov/resource/gdcscd.00340766921/?sp=121",
        LOC,
    ),
    Source.museum_guidebook: SourceMetadata(
        "Barnum's American Museum illustrated (1850)",
        "https://www.loc.gov/resource/rbc0001.2014gen34181/?sp=33",
        LOC_RARE_BOOKS,
    ),
    Source.denver_illustrated: SourceMetadata(
        "Denver illustrated (1893)",
        "https://www.loc.gov/resource/gdclccn.rc01000494/?sp=51",
        LOC_MEETING_FRONTIERS,
    ),
    Source.physics_textbook: SourceMetadata(
        "Lessons in physics (1916)",
        "https://www.loc.gov/resource/gdcscd.00036487318/?sp=103",
        LOC,
    ),
    Source.portrait_miniatures: SourceMetadata(
        "The history of portrait miniatures (1904)",
        "https://www.loc.gov/resource/rbc0001.2024rosen013592v02/?sp=249",
        LOC_RARE_BOOKS,
    ),
    Source.wizard_of_oz_drawings: SourceMetadata(
        "The wonderful Wizard of Oz (1899)",
        "https://www.loc.gov/resource/rbc0001.2006gen32405/?sp=48",
        LOC_RARE_BOOKS,
    ),
    Source.paintings: SourceMetadata(
        "Open book showing paintings by Vincent van Gogh",
        "https://unsplash.com/photos/9hD7qrxICag",
        "Photo by Trung Manh cong on Unsplash",
    ),
    Source.alice_drawing: SourceMetadata(
        "Open book showing an illustration and text from Alice's Adventures in Wonderland",
        "https://unsplash.com/photos/bewzr_Q9u2o",
        "Photo by Brett Jordan on Unsplash",
    ),
    Source.book: SourceMetadata(
        "Open book showing two botanical illustrations",
        "https://unsplash.com/photos/4IDqcNj827I",
        "Photo by Ranurte on Unsplash",
    ),
    Source.manual: SourceMetadata(
        "Open user manual for vintage camera",
        "https://unsplash.com/photos/aaFU96eYASk",
        "Photo by Annie Spratt on Unsplash",
    ),
    Source.electronics: SourceMetadata(
        "Circuit board with electronic components",
        "https://unsplash.com/photos/Aqa1pHQ57pw",
        "Photo by Albert Stoynov on Unsplash",
    ),
}

print("✅ Test images defined")

🧠 Gemini models

Gemini comes in different versions. We can currently use the following models:

For object detection: Gemini 2.5 or Gemini 3, each available in Flash or Pro versions.
For object editing: Gemini 2.5 Flash Image, Gemini 3 Pro Image, or Gemini 3.1 Flash Image (also known as Nano Banana 🍌, Nano Banana Pro 🍌, and Nano Banana 2 🍌).

🛠️ Helpers

Now, let's add core helper classes and functions: 🔽

from enum import auto
from pathlib import Path
from typing import Any, cast

import IPython.display
import matplotlib.pyplot as plt
import pydantic
import tenacity
from google.genai.errors import ClientError
from google.genai.types import (
    FinishReason,
    GenerateContentConfig,
    GenerateContentResponse,
    PIL_Image,
    ThinkingConfig,
    ThinkingLevel,
)


# Multimodal models with spatial understanding and structured outputs
class MultimodalModel(StrEnum):
    # Generally Available (GA)
    GEMINI_2_5_FLASH = "gemini-2.5-flash"
    GEMINI_2_5_PRO = "gemini-2.5-pro"
    # Preview
    GEMINI_3_FLASH_PREVIEW = "gemini-3-flash-preview"
    GEMINI_3_1_PRO_PREVIEW = "gemini-3.1-pro-preview"
    # Default model used for object detection
    DEFAULT = GEMINI_3_FLASH_PREVIEW


# Image generation and editing models (Nano Banana 🍌 models)
class ImageModel(StrEnum):
    # Generally Available (GA)
    GEMINI_2_5_FLASH_IMAGE = "gemini-2.5-flash-image"  # Nano Banana
    # Preview
    GEMINI_3_PRO_IMAGE_PREVIEW = "gemini-3-pro-image-preview"  # Nano Banana Pro
    GEMINI_3_1_FLASH_IMAGE_PREVIEW = "gemini-3.1-flash-image-preview"  # Nano Banana 2
    # Default model used for image editing
    DEFAULT = GEMINI_2_5_FLASH_IMAGE


Model = MultimodalModel | ImageModel


def generate_content(
    contents: list[Any],
    model: Model,
    config: GenerateContentConfig | None,
    should_display_response_info: bool = False,
) -> GenerateContentResponse | None:
    response = None
    client = check_client_for_model(model)

    for attempt in get_retrier():
        with attempt:
            response = client.models.generate_content(
                model=model.value,
                contents=contents,
                config=config,
            )
    if should_display_response_info:
        display_response_info(response, config)

    return response


def check_client_for_model(model: Model) -> genai.Client:
    if (
        model.value.endswith("-preview")
        and client.vertexai
        and client._api_client.location != "global"
    ):
        # Preview models are only available on the "global" location
        return genai.Client(location="global")

    return client


def display_response_info(
    response: GenerateContentResponse | None,
    config: GenerateContentConfig | None,
) -> None:
    if response is None:
        print("❌ No response")
        return

    if usage_metadata := response.usage_metadata:
        if usage_metadata.prompt_token_count:
            print(f"Input tokens   : {usage_metadata.prompt_token_count:9,d}")
        if usage_metadata.candidates_token_count:
            print(f"Output tokens  : {usage_metadata.candidates_token_count:9,d}")
        if usage_metadata.thoughts_token_count:
            print(f"Thoughts tokens: {usage_metadata.thoughts_token_count:9,d}")

    if (
        config is not None
        and config.response_mime_type == "application/json"
        and response.parsed is None
    ):
        print("❌ Could not parse the JSON response")
        return
    if not response.candidates:
        print("❌ No `response.candidates`")
        return
    if (finish_reason := response.candidates[0].finish_reason) != FinishReason.STOP:
        print(f"❌ {finish_reason = }")
    if not response.text:
        print("❌ No `response.text`")
        return


def generate_image(
    sources: list[PIL_Image],
    prompt: str,
    model: ImageModel,
    config: GenerateContentConfig | None = None,
) -> PIL_Image | None:
    contents = [*sources, prompt.strip()]

    response = generate_content(contents, model, config)

    return check_get_output_image_from_response(response)


def check_get_output_image_from_response(
    response: GenerateContentResponse | None,
) -> PIL_Image | None:
    if response is None:
        print("❌ No `response`")
        return None
    if not response.candidates:
        print("❌ No `response.candidates`")
        if response.prompt_feedback:
            if block_reason := response.prompt_feedback.block_reason:
                print(f"{block_reason = :s}")
            if block_reason_message := response.prompt_feedback.block_reason_message:
                print(f"{block_reason_message = }")
        return None
    if not (content := response.candidates[0].content):
        print("❌ No `response.candidates[0].content`")
        return None
    if not (parts := content.parts):
        print("❌ No `response.candidates[0].content.parts`")
        return None

    output_image: PIL_Image | None = None
    for part in parts:
        if part.text:
            display_markdown(part.text)
            continue
        sdk_image = part.as_image()
        assert sdk_image is not None
        output_image = sdk_image._pil_image
        assert output_image is not None
        break  # There should be a single image

    return output_image


def get_thinking_config(model: Model) -> ThinkingConfig | None:
    match model:
        case MultimodalModel.GEMINI_2_5_FLASH:
            return ThinkingConfig(thinking_budget=0)
        case MultimodalModel.GEMINI_2_5_PRO:
            return ThinkingConfig(thinking_budget=128, include_thoughts=False)
        case MultimodalModel.GEMINI_3_FLASH_PREVIEW:
            return ThinkingConfig(thinking_level=ThinkingLevel.MINIMAL)
        case MultimodalModel.GEMINI_3_1_PRO_PREVIEW:
            return ThinkingConfig(thinking_level=ThinkingLevel.LOW)
        case _:
            return None  # Default


def display_markdown(markdown: str) -> None:
    IPython.display.display(IPython.display.Markdown(markdown))


def display_image(image: PIL_Image) -> None:
    IPython.display.display(image)


def get_retrier() -> tenacity.Retrying:
    return tenacity.Retrying(
        stop=tenacity.stop_after_attempt(7),
        wait=tenacity.wait_incrementing(start=10, increment=1),
        retry=should_retry_request,
        reraise=True,
    )


def should_retry_request(retry_state: tenacity.RetryCallState) -> bool:
    if not retry_state.outcome:
        return False
    err = retry_state.outcome.exception()
    if not isinstance(err, ClientError):
        return False
    print(f"❌ ClientError {err.code}: {err.message}")

    retry = False
    match err.code:
        case 400 if err.message is not None and " try again " in err.message:
            # Workshop: first time access to Cloud Storage (service agent provisioning)
            retry = True
        case 429:
            # Workshop: temporary project with 1 QPM quota
            retry = True
    print(f"🔄 Retry: {retry}")

    return retry


print("✅ Helpers defined")

🔍 Detecting visual objects

To perform visual object detection, craft the prompt to indicate what you'd like to detect and how results should be returned. In the same request, it's possible to also extract additional information about each detected object. This can be virtually anything, from labels such as "furniture", "table", or "chair", to more precise classifications like "mammals" or "reptiles", or to contextual data such as captions, colors, shapes, etc.

For the next tests, we'll experiment with detecting illustrations within book photos. Here's a possible prompt:

OBJECT_DETECTION_PROMPT = """
Detect every illustration within the book photo and extract the following data for each:
- `box_2d`: Bounding box coordinates of the illustration only (ignoring any caption).
- `caption`: Verbatim caption or legend such as "Figure 1". Use "" if not found.
- `label`: Single-word label describing the illustration. Use "" if not found.
"""

Notes:

Bounding boxes are very useful for locating or extracting the detected objects.
Typically, for Gemini models, a box_2d bounding box represents coordinates normalized to a (0, 0, 1000, 1000) space for a (0, 0, width, height) input image.
We're also requesting to extract captions (metadata often present in reference books) and labels (dynamic metadata).

To automate response processing, it's convenient to define a Pydantic class that matches the prompt, such as:

class DetectedObject(pydantic.BaseModel):
    box_2d: list[int]
    caption: str
    label: str

DetectedObjects = list[DetectedObject]

Then, request a structured output with config fields response_mime_type and response_schema:

config = GenerateContentConfig(
    # …,
    response_mime_type="application/json",
    response_schema=DetectedObjects,
    # …,
)

This will generate a JSON response which the SDK can parse automatically, letting us directly use object instances:

detected_objects = cast(DetectedObjects, response.parsed)

Let's add a few object-detection-specific classes and functions: 🔽

import io
import urllib.request
from collections.abc import Iterator
from dataclasses import field
from datetime import datetime

import PIL.Image
from google.genai.types import Part, PartMediaResolutionLevel
from PIL.PngImagePlugin import PngInfo

OBJECT_DETECTION_PROMPT = """
Detect every illustration within the book photo and extract the following data for each:
- `box_2d`: Bounding box coordinates of the illustration only (ignoring any caption).
- `caption`: Verbatim caption or legend such as "Figure 1". Use "" if not found.
- `label`: Single-word label describing the illustration. Use "" if not found.
"""

# Margin added to detected/cropped objects, giving more context for a better understanding of spatial distortions
CROP_MARGIN_PX = 10

# Set to True to save each generated image
SAVE_GENERATED_IMAGES = False
OUTPUT_IMAGES_PATH = Path("./object_detection_and_editing")


# Matching class for structured output generation
class DetectedObject(pydantic.BaseModel):
    box_2d: list[int]
    caption: str
    label: str


# Misc data classes
InputImage = Path | Url
DetectedObjects = list[DetectedObject]
WorkflowStepImages = list[PIL_Image]


class WorkflowStep(StrEnum):
    SOURCE = auto()
    CROPPED = auto()
    RESTORED = auto()
    COLORIZED = auto()
    CINEMATIZED = auto()


@dataclass
class VisualObjectWorkflow:
    source_image: PIL_Image
    detected_objects: DetectedObjects
    images_by_step: dict[WorkflowStep, WorkflowStepImages] = field(default_factory=dict)

    def __post_init__(self) -> None:
        denormalize_bounding_boxes(self)


workflow_by_image: dict[InputImage, VisualObjectWorkflow] = {}


def denormalize_bounding_boxes(self: VisualObjectWorkflow) -> None:
    """Convert the box_2d coordinates.
    - Before: [y1, x1, y2, x2] normalized to 0-1000, as returned by Gemini
    - After:  [x1, y1, x2, y2] in source_image coordinates, as used in Pillow
    """

    def to_image_coord(coord: int, dim: int) -> int:
        return int(coord * dim / 1000 + 0.5)

    w, h = self.source_image.size
    for obj in self.detected_objects:
        y1, x1, y2, x2 = obj.box_2d
        x1, x2 = to_image_coord(x1, w), to_image_coord(x2, w)
        y1, y2 = to_image_coord(y1, h), to_image_coord(y2, h)
        obj.box_2d = [x1, y1, x2, y2]


def detect_objects(
    image: InputImage,
    prompt: str = OBJECT_DETECTION_PROMPT,
    model: MultimodalModel = MultimodalModel.DEFAULT,
    config: GenerateContentConfig | None = None,
    media_resolution: PartMediaResolutionLevel | None = None,
    display_results: bool = True,
) -> None:
    display_image_source_info(image)
    pil_image, content_part = get_pil_image_and_part(image, model, media_resolution)
    prompt = prompt.strip()
    contents = [content_part, prompt]
    config = config or get_object_detection_config(model)

    response = generate_content(contents, model, config)

    if response is not None and response.parsed is not None:
        detected_objects = cast(DetectedObjects, response.parsed)
    else:
        detected_objects = DetectedObjects()

    workflow = VisualObjectWorkflow(pil_image, detected_objects)
    workflow_by_image[image] = workflow
    add_cropped_objects(workflow, image, prompt)

    if display_results:
        display_detected_objects(workflow)


def get_pil_image_and_part(
    image: InputImage,
    model: MultimodalModel,
    media_resolution: PartMediaResolutionLevel | None,
) -> tuple[PIL_Image, Part]:
    if isinstance(image, Path):
        image_bytes = image.read_bytes()
    else:
        headers = {"User-Agent": "Mozilla/5.0"}
        req = urllib.request.Request(image, headers=headers)
        with urllib.request.urlopen(req, timeout=10) as response:
            image_bytes = response.read()

    pil_image = PIL.Image.open(io.BytesIO(image_bytes))
    mime_type = f"image/{pil_image.format.lower()}" if pil_image.format else "image/*"
    content_part = Part.from_bytes(
        data=image_bytes,
        mime_type=mime_type,
        media_resolution=media_resolution,
    )

    return pil_image, content_part


def get_object_detection_config(model: Model) -> GenerateContentConfig:
    # Low randomness for more determinism
    return GenerateContentConfig(
        temperature=0.0,
        top_p=0.0,
        seed=42,
        response_mime_type="application/json",
        response_schema=DetectedObjects,
        thinking_config=get_thinking_config(model),
    )


def add_cropped_objects(
    workflow: VisualObjectWorkflow,
    input: InputImage,
    prompt: str,
    crop_margin: int = CROP_MARGIN_PX,
) -> None:
    cropped_images: list[PIL_Image] = []
    obj_count = len(workflow.detected_objects)
    for obj_order, obj in enumerate(workflow.detected_objects, 1):
        cropped_image, _ = extract_object_image(workflow.source_image, obj, crop_margin)
        cropped_images.append(cropped_image)
        save_workflow_image(
            WorkflowStep.SOURCE,
            WorkflowStep.CROPPED,
            input,
            obj_order,
            obj_count,
            cropped_image,
            dict(prompt=prompt, crop_margin=str(crop_margin)),
        )
    workflow.images_by_step[WorkflowStep.CROPPED] = cropped_images


def extract_object_image(
    image: PIL_Image,
    obj: DetectedObject,
    margin: int = 0,
) -> tuple[PIL_Image, tuple[int, int, int, int]]:
    def clamp(coord: int, dim: int) -> int:
        return min(max(coord, 0), dim)

    x1, y1, x2, y2 = obj.box_2d
    w, h = image.size
    if margin != 0:
        x1, x2 = clamp(x1 - margin, w), clamp(x2 + margin, w)
        y1, y2 = clamp(y1 - margin, h), clamp(y2 + margin, h)

    box = (x1, y1, x2, y2)
    object_image = image.crop(box)

    return object_image, box


def save_workflow_image(
    source_step: WorkflowStep,
    target_step: WorkflowStep,
    input_image: InputImage,
    obj_order: int,
    obj_count: int,
    target_image: PIL_Image | None,
    image_info: dict[str, str] | None = None,
) -> None:
    if not SAVE_GENERATED_IMAGES or target_image is None:
        return
    OUTPUT_IMAGES_PATH.mkdir(parents=True, exist_ok=True)
    time_str = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
    try:
        filename = f"{Source(input_image).name}_"
    except ValueError:
        filename = ""
    filename += f"{obj_order}o{obj_count}_{source_step}_{target_step}_{time_str}.png"
    image_path = OUTPUT_IMAGES_PATH.joinpath(filename)
    params = {}
    if image_info:
        png_info = PngInfo()
        for k, v in image_info.items():
            png_info.add_text(k, v)
        params.update(pnginfo=png_info)
    target_image.save(image_path, **params)


# Matplotlib
FIGURE_FG_COLOR = "#F1F3F4"
FIGURE_BG_COLOR = "#202124"
EDGE_COLOR = "#80868B"
rcParams = {
    "figure.dpi": 300,
    "text.color": FIGURE_FG_COLOR,
    "figure.facecolor": FIGURE_BG_COLOR,
    "figure.edgecolor": FIGURE_FG_COLOR,
    "axes.titlecolor": FIGURE_FG_COLOR,
    "axes.edgecolor": EDGE_COLOR,
    "xtick.color": FIGURE_FG_COLOR,
    "ytick.color": FIGURE_FG_COLOR,
    "xtick.bottom": False,
    "xtick.top": False,
    "ytick.left": False,
    "ytick.right": False,
    "xtick.labelbottom": False,
    "ytick.labelleft": False,
}
plt.rcParams.update(rcParams)


def display_image_source_info(image: InputImage) -> None:
    def get_image_info_md() -> str:
        if image not in Source:
            return f"[[Source Image]({image})]"
        source = Source(image)
        metadata = metadata_by_source.get(source)
        if not metadata:
            return f"[[Source Image]({source.value})]"
        parts = [
            f"[Source Image]({source.value})",
            f"[Source Page]({metadata.webpage_url})",
            metadata.title,
            metadata.credit_line,
        ]
        separator = "•"
        inner_info = f" {separator} ".join(parts)
        return f"{separator} {inner_info} {separator}"

    def yield_md_rows() -> Iterator[str]:
        horizontal_line = "---"
        image_info = get_image_info_md()
        yield horizontal_line
        yield f"_{image_info}_"
        yield horizontal_line

    display_markdown(f"{chr(10)}{chr(10)}".join(yield_md_rows()))


def display_detected_objects(workflow: VisualObjectWorkflow) -> None:
    source_image = workflow.source_image
    detected_objects = PIL.Image.new("RGB", source_image.size, "white")
    for obj in workflow.detected_objects:
        obj_image, box = extract_object_image(source_image, obj)
        detected_objects.paste(obj_image, (box[0], box[1]))

    _, (ax1, ax2) = plt.subplots(1, 2, layout="compressed")
    ax1.imshow(source_image)
    ax2.imshow(detected_objects)

    disable_colab_cell_scrollbar()
    plt.show()


print("✅ Object detection helpers defined")

🧪 Let's start simple: can we detect the single illustration in this incunable from 1485?

detect_objects(Source.incunable)

• Source Image • Source Page • Vergaderinge der historien van Troy (1485) • Library of Congress, Rare Book and Special Collections Division •

💡 This works nicely. The bounding box is very precise, enclosing the hand-colored woodcut illustration very tightly.

🧪 Now, let's check the detection of the multiple visuals in this museum guidebook:

detect_objects(Source.museum_guidebook)

• Source Image • Source Page • Barnum's American Museum illustrated (1850) • Library of Congress, Rare Book and Special Collections Division •

💡 Remarks:

The bounding boxes are again very precise.

The results are perfect: there are no false positives and no false negatives.

The captions below the visuals are not enclosed within the bounding boxes, which was specifically requested. The bounding box granularity can be controlled by changing the prompt.

🧪 What about slightly warped visuals?

detect_objects(Source.paintings)

• Source Image • Source Page • Open book showing paintings by Vincent van Gogh • Photo by Trung Manh cong on Unsplash •

💡 This doesn't make a difference. Notice how the bottom-right painting is partially covered by the orange bookmark. We'll try to fix that in the restoration step.

🧪 What about the tilted visuals in this book about the architecture in Denver?

detect_objects(Source.denver_illustrated)

• Source Image • Source Page • Denver illustrated (1893) • Library of Congress, Meeting of Frontiers •

💡 Each visual is perfectly detected: spatial understanding covers tilted objects.

🧪 Finally, let's check the detection on this significantly warped book page from Alice's Adventures in Wonderland:

detect_objects(Source.alice_drawing)

• Source Image • Source Page • Open book showing an illustration and text from Alice's Adventures in Wonderland • Photo by Brett Jordan on Unsplash •

💡 Page curvature and other distortions don't prevent non-rectangular objects from being detected. In fact, spatial understanding works at the pixel level, which explains this precision for warped objects. If you'd like to work at a lower level, you can also ask for a "segmentation mask" in the prompt and you'll get a base64-encoded PNG (each pixel giving the 0-255 probability it belongs to the object within the bounding box). See the segmentation doc for more details.

🏷️ Text extraction and dynamic labeling

On top of localizing each object with its bounding box, our prompt requested to extract a verbatim caption and to assign a single-word label, when possible.

Let's add a simple function to display the detection data in a table: 🔽

from collections import defaultdict


def display_detection_data(source: Source, show_consolidated: bool = False) -> None:
    def string_with_visible_linebreaks(s: str) -> str:
        return f'''"{s.replace(chr(10), "↩️")}"'''

    def yield_md_rows_consolidated(workflow: VisualObjectWorkflow) -> Iterator[str]:
        yield "| label | count | captions |"
        yield "| :--- | ---: | :--- |"
        stats = defaultdict(list)
        for obj in workflow.detected_objects:
            stats[obj.label].append(string_with_visible_linebreaks(obj.caption))
        for label, captions in stats.items():
            count = len(captions)
            label_captions = " • ".join(sorted(captions))
            yield f"| {label} | {count} | {label_captions} |"

    def yield_md_rows_with_bbox(workflow: VisualObjectWorkflow) -> Iterator[str]:
        yield "| box_2d | label | caption |"
        yield "| :--- | :--- | :--- |"
        for obj in workflow.detected_objects:
            yield f"| {obj.box_2d} | {obj.label} | {string_with_visible_linebreaks(obj.caption)} |"

    workflow = workflow_by_image.get(source)
    if workflow is None:
        print(f'❌ No detection for source "{source.name}"')
        return
    md_rows = list(
        yield_md_rows_consolidated(workflow)
        if show_consolidated
        else yield_md_rows_with_bbox(workflow)
    )
    display_image_source_info(source)
    display_markdown(chr(10).join(md_rows))

In the museum guidebook, the dynamic labeling is precise according to the context, and the captions below each illustration are perfectly extracted:

display_detection_data(Source.museum_guidebook)

• Source Image • Source Page • Barnum's American Museum illustrated (1850) • Library of Congress, Rare Book and Special Collections Division •

box_2d	label	caption
[954, 629, 1338, 1166]	beetle	"The Horned Beetle."
[265, 984, 464, 1504]	armor	"Armor of a Man."
[737, 984, 915, 1328]	armor	"Horse Armor."
[1225, 1244, 1589, 1685]	beetle	"The Goliath Beetle."
[264, 1766, 431, 2006]	mask	"The Mask."
[937, 1769, 1260, 2087]	butterfly	"Painted Lady Butterfly."
[1325, 2170, 1581, 2468]	butterfly	"The Lady Butterfly."

In the book photo showing four paintings, this is perfect too:

display_detection_data(Source.paintings)

• Source Image • Source Page • Open book showing paintings by Vincent van Gogh • Photo by Trung Manh cong on Unsplash •

box_2d	label	caption
[378, 203, 837, 575]	painting	"Hái Ô-liu (Olive Picking), tháng 12 năm 1889, sơn dầu trên toan, 28 3/4 x 35 in. [73 x 89 cm]"
[913, 207, 1380, 563]	painting	"Hẻm núi Les Peiroulets (Les Peiroulets Ravine), tháng 10 năm 1889, sơn dầu trên toan, 28 3/4 x 36 1/4 in. [73 x 92 cm]"
[387, 596, 845, 978]	painting	"Trưa: Nghỉ ngơi (phỏng theo Millet) (Noon: Rest from Work [after Millet]), tháng 1 năm 1890, sơn dầu trên toan, 28 3/4 x 35 7/8 in. [73 x 91 cm]"
[921, 611, 1397, 982]	painting	"Hoa hạnh đào (Almond Blossom), tháng 2 năm 1890, sơn dầu trên toan, 28 3/8 x 36 1/4 in. [73 x 92 cm]"

In the Denver architecture book, the four captions are assigned to the correct illustrations, which was not an obvious task:

display_detection_data(Source.denver_illustrated)

• Source Image • Source Page • Denver illustrated (1893) • Library of Congress, Meeting of Frontiers •

box_2d	label	caption
[203, 224, 741, 839]	building	"ERNEST AND CRANMER BUILDING."
[743, 73, 1192, 758]	building	"PEOPLE'S BANK BUILDING."
[1185, 211, 1787, 865]	building	"BOSTON BUILDING."
[699, 754, 1238, 1203]	building	"COOPER BUILDING."

💡 If you have a closer look at the source image, it's hard to tell which caption belongs to which illustration at a glance. Most of us would need to think about it (and might be wrong). Asking Gemini reveals that the results are intentional and not pure luck: Deciphering vintage layouts can feel a bit like a puzzle, but there is usually a "reading-order" logic at play. In this specific case, the captions are arranged to correspond with the images in a clockwise or Z-pattern starting from the top left.

In the "Alice's Adventures in Wonderland" book page, there was a single illustration accompanying the story text. As expected, the caption is empty (i.e., no false positive):

display_detection_data(Source.alice_drawing)

• Source Image • Source Page • Open book showing an illustration and text from Alice's Adventures in Wonderland • Photo by Brett Jordan on Unsplash •

box_2d	label	caption
[111, 146, 1008, 593]	illustration	""

🔭 Generalizing object detection

We can use the same principles for other object types. We'll generally keep requesting bounding boxes to identify object positions within images. Without changing our current output structure (i.e., no code change), we can use captions and labels to extract different object metadata depending on the input type.

🧪 See how we can detect electronic components by adapting the prompt while keeping the exact same code and output structure:

ELECTRONIC_COMPONENT_DETECTION_PROMPT = """
Exhaustively detect all the individual electronic components in the image and provide the following data for each:
- `box_2d`: bounding box coordinates.
- `caption`: Verbatim alphanumeric text visible on the component (including original line breaks), or "" if no text is present.
- `label`: Specific type of component.
"""

detect_objects(
    Source.electronics,
    ELECTRONIC_COMPONENT_DETECTION_PROMPT,
    media_resolution=PartMediaResolutionLevel.MEDIA_RESOLUTION_ULTRA_HIGH,
)

• Source Image • Source Page • Circuit board with electronic components • Photo by Albert Stoynov on Unsplash •

💡 Remarks:

Large and tiny components are detected, thanks to the specific instruction "exhaustively detect…".

By using the ultra-high media resolution, we ensure more details are tokenized and the "P" component (a visual outlier) gets detected.

Here's a consolidated view of the detected components:

display_detection_data(Source.electronics, show_consolidated=True)

• Source Image • Source Page • Circuit board with electronic components • Photo by Albert Stoynov on Unsplash •

label	count	captions
integrated circuit	3	"49240↩️020S6K" • "8105↩️0:35" • "P4010↩️9NA0"
resistor	4	"" • "" • "105" • "R020"
inductor	1	"n1W"
diode	3	"K" • "L" • "P"
capacitor	6	"" • "" • "" • "" • "" • ""
transistor	1	"41"
connector	1	""

💡 Remarks:

Components are detected along with their text markings, despite the three different text orientations (upright, sideways, and upside down), the blur, and the photo noise.

We removed the degree of freedom for multi-line text by specifying the inclusion of "original line breaks" in the prompt: responses now consistently include the line breaks for the three integrated circuits (displayed with the ↩️ emoji for better visibility).

The last degree of freedom lies in the labeling. While most components have been properly labeled, it is unclear whether the "P" component is a diode, a resistor, or a fuse. Making the instructions more specific (e.g., listing the possible labels, using an enum for the label field in the Pydantic class, or providing guidelines and more details about the expected circuit boards) will make the prompt more "closed" and the results more deterministic and accurate. It's also possible to enable/update the thinking_config configuration, which will trigger a chain of thought before generating the final answer. In all the detections performed, our code used ThinkingLevel.MINIMAL, which didn't consume any thought tokens (with Gemini 3 Flash). Updating the parameter to ThinkingLevel.LOW, ThinkingLevel.MEDIUM, or ThinkingLevel.HIGH will use thought tokens and can lead to better outputs in complex cases.

This demonstrates the versatility of the approach. Without retraining a model, we switched from detecting 15th-century woodcuts and illustrations with vintage layouts to identifying modern electronics just by changing the prompt. Such detections, including caption and label metadata, could be used to auto-crop components for a parts catalog, verify assembly lines, or create interactive schematics… all without a single labeled training image.

🪄 Editing visual objects

Now that we can detect visual objects, we can envision an automation workflow to extract and reuse them. For this, we'll use Gemini 2.5 Flash Image (also known as Nano Banana 🍌) by default, a state-of-the-art image generation and editing model.

Our object editing functions will follow the same template, taking one step as input and generating an edited image for the output step. Let's define core helpers for this: 🔽

from typing import Protocol


class ObjectEditingFunction(Protocol):
    def __call__(
        self,
        image: InputImage,
        prompt: str | None = None,
        model: ImageModel | None = None,
        config: GenerateContentConfig | None = None,
        display_results: bool = True,
    ) -> None: ...


SourceTargetSteps = tuple[WorkflowStep, WorkflowStep]
registered_functions: dict[SourceTargetSteps, ObjectEditingFunction] = {}

DEFAULT_EDITING_CONFIG = GenerateContentConfig(response_modalities=["IMAGE"])
EMPTY_IMAGE = PIL.Image.new("1", (1, 1), "white")


def object_editing_function(
    default_prompt: str,
    source_step: WorkflowStep,
    target_step: WorkflowStep,
    default_model: ImageModel = ImageModel.DEFAULT,
    default_config: GenerateContentConfig = DEFAULT_EDITING_CONFIG,
) -> ObjectEditingFunction:
    def editing_function(
        image: InputImage,
        prompt: str | None = default_prompt,
        model: ImageModel | None = default_model,
        config: GenerateContentConfig | None = default_config,
        display_results: bool = True,
    ) -> None:
        workflow, source_images = get_workflow_and_step_images(image, source_step)
        if prompt is None:
            prompt = default_prompt
        prompt = prompt.strip()
        if model is None:
            model = default_model
        # Note: "config is None" is valid and will use the model endpoint default config

        target_images: list[PIL_Image] = []
        display_image_source_info(image)
        obj_count = len(source_images)
        for obj_order, source_image in enumerate(source_images, 1):
            target_image = generate_image([source_image], prompt, model, config)
            save_workflow_image(
                source_step,
                target_step,
                image,
                obj_order,
                obj_count,
                target_image,
                dict(prompt=prompt),
            )
            target_images.append(target_image if target_image else EMPTY_IMAGE)

        workflow.images_by_step[target_step] = target_images
        if display_results:
            display_sources_and_targets(workflow, source_step, target_step)

    registered_functions[(source_step, target_step)] = editing_function

    return editing_function


def get_workflow_and_step_images(
    image: InputImage,
    step: WorkflowStep,
) -> tuple[VisualObjectWorkflow, list[PIL_Image]]:
    # Objects detected?
    if image not in workflow_by_image:
        detect_objects(image, display_results=False)
    workflow = workflow_by_image.get(image)
    assert workflow is not None

    # Workflow step objects? (single level, could be extended to a dynamical graph)
    operation = (WorkflowStep.CROPPED, step)
    if step not in workflow.images_by_step and operation in registered_functions:
        source_function = registered_functions[operation]
        source_function(image, display_results=False)

    # Source images
    source_images = workflow.images_by_step.get(step)
    assert source_images is not None

    return workflow, source_images


def display_sources_and_targets(
    workflow: VisualObjectWorkflow,
    source_step: WorkflowStep,
    target_step: WorkflowStep,
) -> None:
    source_images = workflow.images_by_step[source_step]
    target_images = workflow.images_by_step[target_step]
    if not source_images:
        print("❌ No images to display")
        return

    fig = plt.figure(layout="compressed")
    if horizontal := (len(source_images) >= 2):
        rows, cols = 2, len(source_images)
    else:
        rows, cols = len(source_images), 2
    gs = fig.add_gridspec(rows, cols)

    for i, (source_image, target_image) in enumerate(
        zip(source_images, target_images, strict=True)
    ):
        for dim, image in enumerate([source_image, target_image]):
            grid_spec = gs[dim, i] if horizontal else gs[i, dim]
            ax = fig.add_subplot(grid_spec)
            ax.set_axis_off()
            ax.imshow(image)

    disable_colab_cell_scrollbar()
    plt.show()


print("✅ Object editing helpers defined")

Now, let's define a first editing step to restore the detected objects that can contain many real-life artifacts…

✨ Restoring visual objects

For this restoration step, we need to craft a prompt that is generic enough (to cover most use cases) but also specific enough (to take into account restoration needs).

An image editing prompt is based on natural language, typically using imperative or declarative instructions. With an imperative prompt, you describe the actions to perform on the input, while with a declarative prompt, you describe the expected output. Both are possible and will provide equivalent results. Your choice is really a matter of preference, as long as the prompt makes sense.

Our test suite is mostly composed of book photos, which can contain various photographic and paper artifacts. The Nano Banana models understand these subtleties and can edit images accordingly, which simplifies the prompt.

Here is a possible restoration function using an imperative prompt:

RESTORATION_PROMPT = """
- Isolate and straighten the visual on a pure white background, excluding any surrounding text.
- Clean up all physical artifacts and noise while preserving every original detail.
- Center the result and scale it to fit the canvas with minimal, symmetrical margins, ensuring no distortion or cropping.
"""

# Default config with low randomness for more deterministic restoration outputs
RESTORATION_CONFIG = GenerateContentConfig(
    temperature=0.0,
    top_p=0.0,
    seed=42,
    response_modalities=["IMAGE"],
)

restore_objects = object_editing_function(
    RESTORATION_PROMPT,
    WorkflowStep.CROPPED,
    WorkflowStep.RESTORED,
    default_config=RESTORATION_CONFIG,
)

print("✅ Restoration function defined")

🧪 Let's try to restore the illustration from the 1485 incunable:

restore_objects(Source.incunable)

• Source Image • Source Page • Vergaderinge der historien van Troy (1485) • Library of Congress, Rare Book and Special Collections Division •

💡 We now have a nice restoration of the hand-colored woodcut illustration. Note that our prompt is generic ("clean up all physical artifacts") and could be made more specific to remove more or fewer artifacts. In this example, there are remaining artifacts, such as the paper discoloration in the sword or the bleeding ink in the armor. We'll see if we can fix these in the colorization step.

🧪 What about the illustrations from the museum guidebook?

restore_objects(Source.museum_guidebook)

• Source Image • Source Page • Barnum's American Museum illustrated (1850) • Library of Congress, Rare Book and Special Collections Division •

💡 All good!

🧪 What about the slightly warped visuals?

restore_objects(Source.paintings)

• Source Image • Source Page • Open book showing paintings by Vincent van Gogh • Photo by Trung Manh cong on Unsplash •

💡 Remarks:

Notice how, on the last painting, the orange bookmark is properly removed and the hidden part inpainted to complete the painting.

We requested to "fit the canvas with minimal, symmetrical margins, without distortion or cropping". Depending on the aspect ratio and type of the visual, this degree of freedom can result in different white margins.

This example shows famous paintings by Vincent Van Gogh. Nano Banana does not fetch any reference images and only uses the provided input. If these were photos of private paintings, they would be restored in the same way.

In the Denver architecture book, the illustrations can be tilted, which our generic prompt does not fully take into account. When several geometric transformations are involved, it can be challenging to craft an imperative prompt that details all the operations to perform. Instead, a descriptive prompt can be more straightforward by directly describing the expected output.

🧪 Here's an example of a descriptive prompt focusing on the restoration of tilted visuals:

tilted_visual_prompt = """
An upright, high-fidelity rendition of the visual isolated against a pure white background, filling the canvas with minimal uniform margins. The output is clean, sharp, and free of physical artifacts.
"""

restore_objects(Source.denver_illustrated, tilted_visual_prompt)

• Source Image • Source Page • Denver illustrated (1893) • Library of Congress, Meeting of Frontiers •

💡 Remarks:

To get these results, the prompt focuses on requesting an "upright" visual "filling the canvas", which proves more straightforward to write than trying to account for all possible geometric corrections.

The native visual understanding automatically identifies the content type (photo, illustration, etc.) and the different artifacts (photographic, paper, printing, scanning…), allowing for precise restorations out of the box.

Notice how the consistency is preserved: the last visual is restored as an illustration, while the first visuals maintain their photographic style.

The results, with this rather generic prompt, are impressive. It is, of course, possible to be more specific and request particular lighting, styles, colors…

In this last test, the input visual has distortions not only from the page curvature but also from the photo perspective.

🧪 Here's an example of a descriptive prompt focusing on restoring warped illustrations:

warped_visual_prompt = """
An edge-to-edge digital extraction of the illustration from the provided book photo, excluding any peripheral text. All page curvature and perspective distortions are corrected, resulting in an image framed in a perfect rectangle, on a pure white canvas with minimal margins.
"""

restore_objects(Source.alice_drawing, warped_visual_prompt)

• Source Image • Source Page • Open book showing an illustration and text from Alice's Adventures in Wonderland • Photo by Brett Jordan on Unsplash •

💡 It is really impressive that such a restoration can be performed in a single step. Note that this prompt is not stable and can generate less optimal results (it would benefit from being more precise). If you have complex transformations, test descriptive prompts iteratively, and you might be pleasantly surprised. In the worst case, it's also possible to process the transformations in successive, easier steps.

Now, let's add a colorization step…

🎨 Colorization

Our restoration step respected the original styles of the input images. Recent image editing models excel at transforming image styles, starting with colors. This can generally be performed directly with a simple, precise instruction.

Here is a possible colorization function using an imperative prompt:

COLORIZATION_PROMPT = """
Colorize this image in a modern book illustration style, maintaining all original details without any additions.
"""

colorize = object_editing_function(
    COLORIZATION_PROMPT,
    WorkflowStep.RESTORED,
    WorkflowStep.COLORIZED,
)

print("✅ Colorization function defined")

🧪 Let's modernize our 1485 illustration:

colorize(Source.incunable)

• Source Image • Source Page • Vergaderinge der historien van Troy (1485) • Library of Congress, Rare Book and Special Collections Division •

💡 All details are preserved, as requested in the prompt. Notice how the colorization can naturally fix some remaining artifacts (e.g., the paper discoloration in the sword or the bleeding ink in the armor).

🧪 Let's colorize our museum guidebook illustrations:

colorize(Source.museum_guidebook)

• Source Image • Source Page • Barnum's American Museum illustrated (1850) • Library of Congress, Rare Book and Special Collections Division •

💡 Our prompt is very open as it only specifies "modern book illustration style". This can generate very creative colorizations, but they all seem to make perfect sense.

🧪 What about our Denver buildings?

colorize(Source.denver_illustrated)

• Source Image • Source Page • Denver illustrated (1893) • Library of Congress, Meeting of Frontiers •

💡 As requested, they all look like modern illustrations, including the first visuals (originating from noisy photos).

It's possible to go further by not only "colorizing" but also "transforming" the image into a significantly different one.

🧪 Let's make our "Alice's Adventures in Wonderland" drawing into a watercolor painting:

watercolor_prompt = """
Transform this visual into a warm, watercolor painting.
"""

colorize(Source.alice_drawing, watercolor_prompt)

• Source Image • Source Page • Open book showing an illustration and text from Alice's Adventures in Wonderland • Photo by Brett Jordan on Unsplash •

🧪 What about making it a traditional painting?

painting_prompt = """
Transform this visual into a traditional painting.
"""

colorize(Source.alice_drawing, painting_prompt)

• Source Image • Source Page • Open book showing an illustration and text from Alice's Adventures in Wonderland • Photo by Brett Jordan on Unsplash •

We can also change image compositions. Depending on the context, some compositions are more or less implied by default. For example, illustrations often have margins, while photos generally have edge-to-edge (full-bleed in the printing world) compositions. When possible, it's interesting to refer to a type of visual (which intrinsically brings a lot of semantics to the context) and adjust the instructions accordingly.

🧪 Let's see how we can detect engravings in this 1847 book, restore them, and transform them into modern digital graphics:

detect_objects(Source.engravings)

• Source Image • Source Page • Harper's illustrated catalogue (1847) • Library of Congress •

restore_objects(Source.engravings)

• Source Image • Source Page • Harper's illustrated catalogue (1847) • Library of Congress •

visual_to_digital_graphic_prompt = """
Transform this visual into a full-color, flat digital graphic, extending the content for a full-bleed effect.
"""

colorize(Source.engravings, visual_to_digital_graphic_prompt)

• Source Image • Source Page • Harper's illustrated catalogue (1847) • Library of Congress •

🧪 We can also transform the same engravings into photos with a very simple prompt:

visual_to_photo_prompt = """
Transform this visual into a high-end, modern camera photograph.
"""

colorize(Source.engravings, visual_to_photo_prompt)

• Source Image • Source Page • Harper's illustrated catalogue (1847) • Library of Congress •

💡 As photos are generally full-bleed, the prompt does not need to specify a composition.

It's really up to our imagination, as Nano Banana seems to grasp every aspect of the visual semantics.

Let's add a final step to see how far we can go, reimagining images as cinematic movie stills…

🎞️ Cinematization

We've used rather "closed" prompts so far, crafting specific instructions and constraints to control the outputs. It's possible to go even further with "open" prompts and generate images in full creative mode. Notably, it can be interesting to refer to photographic or cinematographic terminology as it encompasses many visual techniques.

Here is a possible generic cinematization function to reimagine images as movie stills:

CINEMATIZATION_PROMPT = """
Reimagine this image as a joyful, modern live-action cinematic movie still featuring professional lighting and composition.
"""

cinematize = object_editing_function(
    CINEMATIZATION_PROMPT,
    WorkflowStep.RESTORED,
    WorkflowStep.CINEMATIZED,
)

🧪 Let's cinematize the "Alice's Adventures in Wonderland" drawing:

cinematize(Source.alice_drawing)

• Source Image • Source Page • Open book showing an illustration and text from Alice's Adventures in Wonderland • Photo by Brett Jordan on Unsplash •

💡 This looks like a high-budget movie still. There are lots of degrees of freedom in the prompt, but you're likely to get foreground figures in sharp focus, a gradual background blur, "golden hour" lighting (a magical ingredient for many cinematographers), and detailed textures. Such compositions really evoke different atmospheres compared to the photos generated in the previous test.

🧪 Let's test the workflow on a page from the Wonderful Wizard of Oz containing three drawings:

detect_objects(Source.wizard_of_oz_drawings)

• Source Image • Source Page • The wonderful Wizard of Oz (1899) • Library of Congress, Rare Book and Special Collections Division •

restore_objects(Source.wizard_of_oz_drawings)

• Source Image • Source Page • The wonderful Wizard of Oz (1899) • Library of Congress, Rare Book and Special Collections Division •

cinematize(Source.wizard_of_oz_drawings)

• Source Image • Source Page • The wonderful Wizard of Oz (1899) • Library of Congress, Rare Book and Special Collections Division •

💡 The cast for a new movie is ready ;)

Cinematic images have various use cases:

These cinematized stills can be perfect "reference images" for video generation models like Veo. See Generate Veo videos from reference images.
As they are photorealistic representations, they can also be a source for generating 2D or 3D visuals, in any style, with realistic figures, perfect proportions, advanced lighting, enhanced compositions…
You can use them in many professional contexts or for high-end products: presentations, magazines, posters, storyboards, brainstorming sessions…

🏁 Conclusion

Gemini's native spatial understanding enables the detection of specific visual objects based on a single prompt in natural language.
We tested the detection of illustrations in book photos, which traditional machine learning (ML) models usually miss, as they are typically trained to detect people, animals, vehicles, food, and a finite set of physical object classes.
We tested the detection of straight, tilted, and even significantly warped illustrations, and they were always precisely identified.
The core implementation was straightforward, requiring minimal code using the Python SDK and customized prompts. By comparison, fine-tuning a traditional object detection model is time-consuming: it involves assembling an image dataset, labeling objects, and managing training jobs.
This solution is very flexible: we could switch from detecting illustrations to electronic components, by adapting the prompt, while keeping the code unchanged.
Using structured outputs (with a JSON schema or Pydantic classes, and the Python SDK) makes the code both easy to implement and ready to deploy to production.
Then, Nano Banana allows editing these visual objects in virtually any way imaginable.
We tested a workflow with restoration, colorization, and even cinematization steps, using imperative and descriptive prompts.
The possibilities seem really endless, and the principles in this exploration can be reused in different contexts.

➕ More!

Try it yourself: Use the companion notebook to reproduce the results in this article.
Get inspired: Check out the Nano Banana recipes notebook for more practical examples.
Follow me: Connect with me (@picardparis) on LinkedIn or Twitter-X for more cloud, applied AI, and Python explorations…

Thanks for reading. Let me know if you create something cool!

Counting Gemini text tokens locally with the Vertex AI SDK

Laurent Picard — Thu, 04 Jul 2024 12:43:14 +0000

⚠️ Building with Gemini and calling the Gemini API is now best done with Google Gen AI SDKs. Check out 🆕 How to Count Gemini Tokens Locally 🆕 for an up-to-date version using the Google Gen AI Python SDK.

The Vertex AI SDK for Python now offers local tokenization. This feature allows you to calculate the number of tokens in your text input before sending requests to Gemini. Let’s see how this feature works…

🧩 What is a token?

Large language models (LLMs) use a fundamental unit: the token. In other words, they process input tokens and generate output tokens.

A text token can represent characters, words, or phrases. On average, one token represents approximately four characters in English text. When you send a query to Gemini, your text input is transformed into tokens. This step is called tokenization. Gemini generates output tokens that are then converted back into text using the reverse operation.

📦️ Setting up the Vertex AI SDK

To use Gemini tokenizers, you need both the latest google-cloud-aiplatform package with the tokenization extra:

pip install --upgrade google-cloud-aiplatform[tokenization]

Ensure you have version 1.57.0 or later installed:

pip show google-cloud-aiplatform

Name: google-cloud-aiplatform
Version: 1.57.0
…

🧮 Counting tokens locally

Create a tokenizer for the Gemini model you’re using and call the count_tokens() method on your text input:

from vertexai.preview import tokenization

model_name = "gemini-1.5-flash-001"
tokenizer = tokenization.get_tokenizer_for_model(model_name)

contents = "Hello World!"
result = tokenizer.count_tokens(contents)

print(f"{result.total_tokens = :,}")

result.total_tokens = 3

Now, let’s try with a larger document:

import http.client
import typing
import urllib.request

def download_text_from_url(url: str) -> str:
    with urllib.request.urlopen(url) as response:
        response = typing.cast(http.client.HTTPResponse, response)
        encoding = response.headers.get_content_charset() or "utf-8"
        return response.read().decode(encoding)

url = "https://storage.googleapis.com/dataflow-samples/shakespeare/hamlet.txt"
contents = download_text_from_url(url)
result = tokenizer.count_tokens(contents)

print(f"{result.total_tokens = :,}")

result.total_tokens = 53,824

🚀 Perfect! The number of tokens is computed in a fraction of second.

🖼️ Multimodal inputs

In the tested version (1.57.0), local token counting is only supported for text inputs.

For multimodal inputs (image, video, audio, documents), check out the documentation for details on how different medias account for different token counts:

In all cases, you can send a request using the Vertex AI API as usual:

import vertexai
from vertexai.generative_models import GenerativeModel

# project_id = "PROJECT_ID"
# location = "us-central1"
# vertexai.init(project=project_id, location=location)

model = GenerativeModel(model_name)

response = model.count_tokens(contents)

print(f"{response.total_tokens = :,}")
print(f"{response.total_billable_characters = :,}")

response.total_tokens = 53,824
response.total_billable_characters = 144,443

🔍 Under the hood

Text tokenizers use a fixed data file, also called the LLM vocabulary. This vocabulary determines how to encode a text string into a sequence of tokens and, reversely, how to decode a sequence of tokens into a text string.

Here is what happens under the hood:

When you request a tokenizer for your model for the first time, the LLM vocabulary is downloaded and locally cached. On subsequent calls, the cached data is used.
When you call tokenizer.count_tokens(contents), the text is tokenized (becomes a sequence of tokens) and the corresponding number of tokens is returned.

Key points to remember:

💡 For Gemini 1.0 and 1.5, the LLM vocabulary size is only about 4 MB.
💡 Different Gemini models can use the same tokenizer but this is not guaranteed. Make sure to get the tokenizer for your model.

⚡ Benefits of knowing token counts

Knowing how many tokens your text represents enables the following:

✅ Determine whether your text input will be processed by a model. For example, Gemini 1.0 Pro supports up to 32,760 input tokens, while Gemini 1.5 Pro currently accepts up to 2,097,152 input tokens.
✅ Consolidate or estimate costs on a per-token basis (Gemini pricing is per 1k characters).
✅ Estimate how much time will be needed to process your text input before you start receiving a response (processing times are roughly proportional to the numbers of processed tokens).

You now have a new local tool to manage your inputs before sending requests to Gemini!

🖖 Follow me on Twitter/X or LinkedIn for more cloud explorations

Making AI more Open and Accessible to Cloud Developers with Gemma on Vertex AI

Laurent Picard — Mon, 26 Feb 2024 13:39:01 +0000

Gemma just opened ;)

Gemma is a family of open, lightweight, and easy-to-use models developed by Google Deepmind. The Gemma models are built from the same research and technology used to create Gemini.

This means that we (ML developers & practitioners) now have additional versatile large language models (LLMs) in our toolbox!

If you’d rather read code, you can jump straight to this Python notebook:
→ Finetune Gemma using KerasNLP and deploy to Vertex AI

Availability

Gemma is available today in Google Cloud and the machine learning (ML) ecosystem. You’ll probably find an ML platform you’re already familiar with:

Gemma joins 130+ models in Vertex AI Model Garden
Gemma joins the Kaggle Models
Gemma joins the Hugging Face Models
Gemma joins Google AI for Developers

Here are the Gemma Terms of Use.

Gemma models

The Gemma family launches in two sizes, Gemma 2B and 7B , targeting two typical platform types:

Model	Parameters	Platforms
Gemma 2B	2.5 billion	Mobile devices and laptops
Gemma 7B	8.5 billion	Desktop computers and small servers

These models are text-to-text English LLMs:

Input: a text string, such as a question, a prompt, or a document.
Output: generated English text, such as an answer or a summary.

They were trained on a diverse and massive dataset of 8 trillion tokens, including web documents, source code, and mathematical text.

Each size is available in untuned and tuned versions:

Pretrained (untuned): Models were trained on core training data, not using any specific tasks or instructions.
Instruction-tuned: Models were additionally trained on human language interactions.

That gives us four variants. As an example, here are the corresponding model IDs when using Keras:

gemma_2b_en
gemma_instruct_2b_en
gemma_7b_en
gemma_instruct_7b_en

Use cases

Google now offers two families of LLMs: Gemini and Gemma. Gemma is a complement to Gemini, is based on technologies developed for Gemini, and addresses different use cases.

Examples of Gemini benefits:

Enterprise applications
Multilingual tasks
Optimal qualitative results and complex tasks
State-of-the-art multimodality (text, image, video)
Groundbreaking features (e.g. Gemini 1.5 Pro 1M token context window)

Examples of Gemma benefits:

Learning, research, or prototyping based on lightweight models
Focused tasks such as text generation, summarization, and Q&A
Framework or cross-device interoperability
Offline or real-time text-only processings

The Gemma model variants are useful in the following use cases:

Pretrained (untuned): Consider these models as a lightweight foundation and perform a custom finetuning to optimize them to your own needs.
Instruction-tuned: You can use these models for conversational applications, such as a chatbot, or to customize them even further.

Interoperability

As Gemma models are open, they are virtually interoperable with all ML platforms and frameworks.

For the launch, Gemma is supported by the following ML ecosystem players:

Google Cloud
Kaggle
Keras, which means that Gemma runs on JAX, PyTorch, and TensorFlow
Hugging Face
Nvidia

And, of course, Gemma can run on GPUs and TPUs.

Requirements

Gemma models are lightweight but, in the LLM world, this still means gigabytes.

In my tests, when running inferences in half precision (bfloat16), here are the minimum storage and GPU memory that were required:

Model	Total parameters	Assets size	Min. GPU RAM to run
Gemma 2B	2,506,172,416	4.67 GB	8.3 GB
Gemma 7B	8,537,680,896	15.90 GB	20.7 GB

To experiment with Gemma and Vertex AI, I used Colab Enterprise with a g2-standard-8 runtime, which comes with an NVIDIA L4 GPU and 24 GB of GPU RAM. This is a cost-effective configuration to save time and avoid running out-of-memory when prototyping in a notebook.

Finetuning Gemma

Depending on your preferred frameworks, you’ll find different ways to customize Gemma. Keras is one of them and provides everything you need.

KerasNLP lets you load a Gemma model in a single line:

import keras
import keras_nlp

gemma_lm = keras_nlp.models.GemmaCausalLM.from_preset("gemma_instruct_2b_en")

Then, you can directly generate text:

inputs = [
    "What's the most famous painting by Monet?",
    "Who engineered the Statue of Liberty?",
    'Who were "The Lumières"?',
]

for input in inputs:
    response = gemma_lm.generate(input, max_length=25)
    output = response[len(input) :]
    print(f"{input!r}\n{output!r}\n")

With the instruction-tuned version, Gemma gives you expected answers, as would a good LLM-based chatbot:

# With "gemma_instruct_2b_en"

"What's the most famous painting by Monet?"
'\n\nThe most famous painting by Monet is "Water Lilies," which'

'Who engineered the Statue of Liberty?'
'\n\nThe Statue of Liberty was built by French sculptor Frédéric Auguste Bartholdi between 1'

'Who were "The Lumières"?'
'\n\nThe Lumières were a group of French scientists and engineers who played a significant role'

Now, try the untuned version:

gemma_lm = keras_nlp.models.GemmaCausalLM.from_preset("gemma_2b_en")

You’ll get different results, which is expected as this version was not trained on any specific task and is designed to be finetuned to your own needs:

# With "gemma_2b_en"

"What's the most famous painting by Monet?"
"\n\nWhat's the most famous painting by Van Gogh?\n\nWhat"

'Who engineered the Statue of Liberty?'
'\n\nA. George Washington\nB. Napoleon Bonaparte\nC. Robert Fulton\nD'

'Who were "The Lumières"?'
' What did they invent?\n\nIn the following sentence, underline the correct modifier from the'

If you’d like, for example, to build a Q&A application, prompt engineering may fix some of these issues but, to be grounded on facts and return consistent results, untuned models need to be finetuned with training data.

The Gemma models have billions of trainable parameters. The next step consists of using the Low Rank Adaptation (LoRA) to greatly reduce the number of trainable parameters:

# Number of trainable parameters before enabling LoRA: 2.5B

# Enable LoRA for the model and set the LoRA rank to 4
gemma_lm.backbone.enable_lora(rank=4)

# Number of trainable parameters after enabling LoRA: 1.4M (1,800x less)

A training can now be performed with reasonable time and GPU memory requirements.

For prototyping on a small training dataset, you can launch a local finetuning:

training_data: list[str] = [...]

# Reduce the input sequence length to limit memory usage
gemma_lm.preprocessor.sequence_length = 128

# Use AdamW (a common optimizer for transformer models)
optimizer = keras.optimizers.AdamW(
    learning_rate=5e-5,
    weight_decay=0.01,
)

# Exclude layernorm and bias terms from decay
optimizer.exclude_from_weight_decay(var_names=["bias", "scale"])

gemma_lm.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=optimizer,
    weighted_metrics=[keras.metrics.SparseCategoricalAccuracy()],
)
gemma_lm.fit(training_data, epochs=1, batch_size=1)

After a couple of minutes of training (on 1,000 examples) and using a structured prompt, the finetuned model can now answer questions based on your training data:

# With "gemma_2b_en" before finetuning
'Who were "The Lumières"?'
' What did they invent?\n\nIn the following sentence, underline the correct modifier from the'

# With "gemma_2b_en" after finetuning
"""
Instruction:
Who were "The Lumières"?

Response:
"""
"The Lumières were the inventors of the first motion picture camera. They were"

The prototyping stage is over. Before you proceed to finetuning models in production with large datasets, check out new specific tools to build a responsible Generative AI application.

Responsible AI

With great power comes great responsibility, right?

The Responsible Generative AI Toolkit will help you build safer AI applications with Gemma. You’ll find expert guidance on safety strategies for the different aspects of your solution.

Also, do not miss the Learning Interpretability Tool (LIT) for visualizing and understanding the behavior of your Gemma models. Here’s a video tour showing the tool in action:

Deploying Gemma

We’re in MLOps territory here and you’ll find different LLM-optimized serving frameworks running on GPUs or TPUs.

Here are popular serving frameworks:

vLLM (GPU)
TGI: Text Generation Interface (GPU)
Saxml (TPU or GPU)
TensorRT-LLM (NVIDIA Triton GPU)

These frameworks come with prebuilt container images that you can easily deploy to Vertex AI.

Here is a simple notebook to get you started:

Finetune Gemma using KerasNLP and deploy to Vertex AI

For production finetuning, you can use Vertex AI custom training jobs. Here are detailed notebooks:

Gemma Finetuning (served by vLLM or HexLLM)
Gemma Finetuning (served by TGI)

Those notebooks focus on deployment and serving:

Gemma Deployment (served by vLLM or HexLLM)
Gemma Deployment to GKE using TGI on GPU

For more details and updates, check out the documentation:

All the best to Gemini and Gemma!

After years of consolidation in machine learning hardware and software, it’s exciting to see the pace at which the overall ML landscape now evolves and in particular with LLMs. LLMs are technologies still in their infancy, so we can expect to see more breakthroughs in the near future.

I very much look forward to seeing what the ML community is going to build with Gemma!

All the best to the Gemini and Gemma families!

Follow me on Twitter or LinkedIn for more cloud explorations…

Moderating text with the Natural Language API

Laurent Picard — Tue, 20 Jun 2023 15:46:47 +0000

2023–09–12: text moderation got Generally Available (GA) over the summer + added link to Sep. blog post

Credit: Photo by Da Nina on Unsplash

The Natural Language API lets you extract information from unstructured text using Google machine learning and provides a solution to the following problems:

Sentiment analysis
Entity analysis
Entity sentiment analysis
Syntax analysis
Content classification
Text moderation

🔍 Moderation categories

Text moderation lets you detect sensitive or harmful content. The first moderation category that comes to mind is "toxicity", but there can be many more topics of interest. A PaLM 2-based model powers the predictions and scores 16 categories:


Toxic	Insult	Public Safety	War & Conflict
Derogatory	Profanity	Health	Finance
Violent	Death, Harm & Tragedy	Religion & Belief	Politics
Sexual	Firearms & Weapons	Illicit Drugs	Legal

⚡ Moderating text

Like always, you can call the API through the REST/RPC interfaces or with idiomatic client libraries.

Here is an example using the Python client library (google-cloud-language) and the moderate_text method:

from google.cloud import language

def moderate_text(text: str) -> language.ModerateTextResponse:
    client = language.LanguageServiceClient()
    document = language.Document(
        content=text,
        type_=language.Document.Type.PLAIN_TEXT,
    )
    return client.moderate_text(document=document)

text = (
    "I have to read Ulysses by James Joyce.\n"
    "I'm a little over halfway through and I hate it.\n"
    "What a pile of garbage!"
)
response = moderate_text(text)

🚀 It's fast! The model latency is very low, allowing real-time analyses.

The response contains confidence scores for each moderation category. Let's sort them out:

import pandas as pd

def confidence(category: language.ClassificationCategory) -> float:
    return category.confidence

columns = ["category", "confidence"]
categories = sorted(
    response.moderation_categories,
    key=confidence,
    reverse=True,
)
data = ((category.name, category.confidence) for category in categories)
df = pd.DataFrame(columns=columns, data=data)

print(f"Text analyzed:\n{text}\n")
print(f"Moderation categories:\n{df}")

You may typically ignore scores below 50% and calibrate your solution by defining upper limits (or buckets) for the confidence scores. In this example, depending on your thresholds, you may flag the text as disrespectful (toxic) and insulting:

Text analyzed:
I have to read Ulysses by James Joyce.
I'm a little over halfway through and I hate it.
What a pile of garbage!

Moderation categories:
                 category  confidence
0                   Toxic    0.680873
1                  Insult    0.609475
2               Profanity    0.482516
3                 Violent    0.333333
4                Politics    0.237705
5   Death, Harm & Tragedy    0.189759
6                 Finance    0.176955
7       Religion & Belief    0.151079
8                   Legal    0.100946
9                  Health    0.096305
10          Illicit Drugs    0.083333
11     Firearms & Weapons    0.076923
12             Derogatory    0.073953
13         War & Conflict    0.052632
14          Public Safety    0.051813
15                 Sexual    0.028222

🖖 More

Run this Colab notebook: Using the Natural Language API
See the supported languages
Read more about text moderation
See the latest blog post on Improving Trust in AI and Online Communities
Follow me on Twitter or LinkedIn for more cloud explorations

Using Google Cloud from Colab

Laurent Picard — Fri, 16 Jun 2023 08:49:26 +0000

Google Colab is an amazing tool for Pythonistas. It can be used for a variety of tasks, from quickly experimenting with Python code to sharing extensive data processing notebooks with the world. Colab runs on Google Cloud, which gives you a boost when accessing cloud services because your code is running inside Google’s high performance network, and also offers a simple way to use Google Cloud services, letting you run powerful cloud workloads from your browser.

📦️ Install dependencies

Colab comes with hundreds of preinstalled packages, including pandas, numpy, matplotlib, Flask, Pillow, tensorflow, and pytorch. You can install additional required dependencies as needed.

For example, to analyze text with the power of machine learning using the Natural Language API, install the google-cloud-language client library:

!pip install google-cloud-language>=2.9.1

🔑 Authenticate

To authenticate with your Google Cloud account within the Colab notebook, use the authenticate_user method. A new parameter lets you specify your project ID:

from google.colab import auth

PROJECT_ID = ""  # @param {type:"string"}

auth.authenticate_user(project_id=PROJECT_ID)

After this step:

👍 You are authenticated for gcloud CLI commands.
👍 You are also authenticated when using Google Cloud Python client libraries. Note that you generally won’t need to create a service account.
👍 Your default project is set.
👍 Your notebook can also access your Google Drive, letting you ingest or generate your own files.

🔓 Enable APIs

Ensure required APIs are enabled. In our example, that’s the service language.googleapis.com:

!gcloud services enable language.googleapis.com

🤯 Use Google Cloud services

That’s it! You can now directly use the service by calling its Python client library:

from google.cloud import language

def analyze_text_sentiment(text: str) -> language.AnalyzeSentimentResponse:
    client = language.LanguageServiceClient()
    document = language.Document(
        content=text,
        type_=language.Document.Type.PLAIN_TEXT,
    )
    return client.analyze_sentiment(document=document)

# Input
text = "Python is a very readable language, ..."  # @param {type:"string"}

# Send a request to the API
response = analyze_text_sentiment(text)

# Use the results
print(response)

📊 Benefit from notebook advanced features

Colab is hosting a Jupyter notebook. I personally depict Colab as the “Google Drive of notebooks”. As notebooks have additional superpowers, this lets you nicely process and visualize your data, such as tables, images, or charts.

In our example, the function show_text_sentiment gathers results in a pandas DataFrame, which renders as a table:

✨ Check it out

In these examples, click the “Open in Colab” link:

Using Google Cloud from Colab (short)
Using the Natural Language API with Python (detailed)

💡 You can directly create a new notebook by opening the colab.new URL.

👋 Have fun!

Read the Google Colab blog to learn more about Colab.
Follow me (Twitter / LinkedIn) for more cloud explorations ;)

From pixels to information with Document AI

Laurent Picard — Wed, 15 Mar 2023 17:18:46 +0000

We’re seeing successively difficult problems getting solved thanks to machine learning (ML) models. For example, Natural Language AI and Vision AI extract insights from text and images, with human-like results. They solve problems central to the way we communicate:

✅ Natural language processing (NLP)
✅ Optical character recognition (OCR)
✅ Handwriting recognition (HWR)

What's next? Well, it's already here, with Document AI, and keeps growing:

✅ Document understanding (DU)

Document AI builds on these foundations to let you extract information from documents, in many forms:

Text
Structure
Semantic entities
Normalized values
Enrichments
Qualitative signals

In this article, you’ll see the following:

An overview of Document AI
Practical and visual examples
A document processing demo (source code included)

Processing documents

Processor types

There are as many document types as you can imagine so, to meet all needs, document processors are at the heart of Document AI. They can be of the following types:

General processors handle common tasks such as detecting document tables or parsing forms.
Specialized processors analyze specific documents such as invoices, receipts, pay slips, bank statements, identity documents, official forms, etc.
Custom processors meet other specific needs, letting you automatically train private ML models based on your own documents, and perform tasks such as custom document classification or custom entity detection.

Processor locations

When you create a processor, you specify its location. This helps control where the documents will be processed. Here are the current multi-region locations:

Location	API `location`	API endpoint
European Union	`eu`	`eu-documentai.googleapis.com`
United States	`us`	`us-documentai.googleapis.com`

In addition, some processors are available in single-region locations; this lets you address local regulation requirements. If you regularly process documents in real-time or large batches of documents, this can also help get responses with even lower latencies. As an example, here are the current locations for the "Document OCR" general processor:

`OCR_PROCESSOR` location	API `location`
United States	`us`
European Union	`eu`
India (Mumbai)	`asia-south1`
Singapore (Jurong West)	`asia-southeast1`
Australia (Sydney)	`australia-southeast1`
England (London)	`europe-west2`
Germany (Frankfurt)	`europe-west3`
Québec (Montréal)	`northamerica-northeast1`

Note: API endpoints are named according to the convention {location}-documentai.googleapis.com.

For more information, check out Regional and multi-regional support.

Processor versions

Processors can evolve over time, to offer more precise results, new features, or fixes. For example, here are the current versions available for the "Expense Parser" specialized processor:

`name`	`display_name`	`create_time`
pretrained-expense-v1.1-2021-04-09	Stable	2021-04-09
pretrained-expense-v1.2-2022-02-18	Stable	2022-02-18
pretrained-expense-v1.3-2022-07-15	Stable	2022-07-15
pretrained-expense-v1.4-2022-11-18	Release Candidate	2022-11-18

You may typically do the following:

Use Stable version 1.3 in production.
Use Release Candidate version 1.4 to benefit from new features or test a future version.

To stay updated, follow the Release notes.

Interfaces

Document AI is available to developers and practitioners through the usual interfaces:

REST API (universal JSON-based interface)
RPC API (low-latency gRPC interface, used by all Google services)
Client Libraries (gRPC wrappers, to develop in your preferred programming language)
Cloud Console (web admin user interface)

Requests

There are two ways you can process documents:

Synchronously with online requests, to analyze a single document and directly use the results
Asynchronously with batch requests, to launch a batch processing operation on multiple or larger documents

Interface	Synchronous method	Asynchronous method
REST	`process`	`batchProcess`
RPC	`ProcessDocument`	`BatchProcessDocuments`
Client library: Python	`process_document`	`batch_process_documents`
Client library: Node.js	`processDocument`	`batchProcessDocuments`
Client library: Java,…	…	…

Extracting information from pixels

Let's start by analyzing this simple screenshot with the "Document OCR" general processor, and check how pixels become structured data.

Input image

Document AI response

The response contains a Document instance.
The entire text of the input, detected in natural reading order, is serialized under Document.text.
The structured data is returned on a per page basis (a single page here).
The "Document OCR" processor returns a structural skeleton common to all processors.

{ // document
  "mime_type": "image/png",
  "text": "Willkommen!\nBienvenue !\nWelcome!\nQuestion: What is the answer to the ultimate question of life?\nAnswer: 42\n",
  "pages": [
    {
      "page_number": 1,
      "dimension": {/*…*/},
      "layout": {/*…*/},
      "detected_languages": [/*…*/],
      "blocks": [/*…*/],
      "paragraphs": [/*…*/],
      "lines": [/*…*/],
      "tokens": [/*…*/],
      "image": {/*…*/}
    }
  ]
}

The examples in this article show snake-case field names (like mime_type), using the convention used by gRPC and programming languages like Python. For camel-case environments (like REST/JSON), there is a direct mapping between the two conventions:

mime_type ↔ mimeType
page_number ↔ pageNumber
…

Text levels

For each page, four levels of text detection are returned:

blocks
paragraphs
lines
tokens

In these lists, each item exposes a layout including the item's position relative to the page in bounding_poly.normalized_vertices.

This lets us, for example, highlight the 22 detected tokens:

Here is the last token:

{ // document.pages[0].tokens[21]
  "layout": {
    "text_anchor": {
      // "42\n" ← document.text[104:107]
      "text_segments": [{ "start_index": "104", "end_index": "107" }]
    },
    "confidence": 0.99, // Confident at 99% that the answer is "42" ;)
    "bounding_poly": {
      // The 4 bounding box vertices, relative to the page
      // Page (top, left)..(bottom, right): (0.0, 0.0)..(1.0, 1.0)
      "normalized_vertices": [ { "x": 0.4028, "y": 0.5352 },
                               { "x": 0.4327, "y": 0.5352 },
                               { "x": 0.4327, "y": 0.6056 },
                               { "x": 0.4028, "y": 0.6056 } ]
    } //,…
  } //,…
}

Note: Float values are presented truncated for the sake of readability.

Language support

Documents are often written using one single language, but sometimes use multiple languages. You can retrieve the detected languages at different text levels.

In our previous example, two blocks are detected (pages[0].blocks[]). Let's highlight them:

The left block is a mix of German, French, and English, while the right block is English only. Here is how the three languages are reported at the page level:

{ // document.pages[0]
  "detected_languages": [
    { "language_code": "en", "confidence": 0.77 },
    { "language_code": "de", "confidence": 0.12 },
    { "language_code": "fr", "confidence": 0.11 }
  ]
}

Note: At this level, the language confidence ratios roughly correspond to the proportion of text detected in each language.

Now, let's highlight the five detected lines (pages[0].lines[]):

Each language is also reported at the line level:

{ // document.pages[0]
  "lines": [
    // "Willkommen!"
    { "detected_languages": [{ "language_code": "de", "confidence": 1.0 }] },
    // "Bienvenue !"
    { "detected_languages": [{ "language_code": "fr", "confidence": 1.0 }] },
    // "Welcome!"
    { "detected_languages": [{ "language_code": "en", "confidence": 1.0 }] },
    // "Question: What is the answer to the ultimate question of life?"
    { "detected_languages": [{ "language_code": "en", "confidence": 1.0 }] },
    // "Answer: 42"
    { "detected_languages": [{ "language_code": "en", "confidence": 1.0 }] }
  ]
}

If needed, you can get language info at the token level too. "Question" is the same word in French and in English, and is adequately returned as an English token in this context:

{ // document.pages[0].tokens[6]
  "layout": {
    "text_anchor": {
      // "Question" ← document.text[33:41]
      "text_segments": [{ "start_index": "33", "end_index": "41" }]
    },
    "confidence": 0.99
  },
  // In this context, the token "Question" is English, not French
  "detected_languages": [{ "language_code": "en" }]
}

In the screenshot, did you notice something peculiar in the left block?

Well, punctuation rules can be different between languages. French uses a typographical space for double punctuation marks ("double" as "written in two parts", such as in "!", "?", "«",…). Punctuation is an important part of languages that can get "lost in translation". Here, the space is preserved in the transcription of "Bienvenue !". Nice touch Document AI or, should I say, touché!

{ // document.pages[0].lines[1]
  "layout": {
    "text_anchor": {
      "text_segments": [
        // "Bienvenue !\n" ← document.text[12:24]
        // The French typographical space before "!" is preserved \o/
        { "start_index": "12", "end_index": "24" }
      ]
    },
    "confidence": 0.99
  }
}

For more information, see Language support.

Handwriting detection

Now — a much harder problem — let's check how handwriting is handled.

This example is a mix of printed and handwritten text, where I wrote both a question and an answer. Here are the detected tokens:

I am pleasantly surprised to see my own handwriting transcribed:

{ // document
  "text": "Willkommen!\nBienvenue !\nWelcome!\nQuestion: What is the answer to the ultimate question of life?\nAnswer:\n42\n"
}

I asked my family (used to writing French) to do the same. Each unique handwriting sample also gets correctly detected:

… and transcribed:

{ // document
  "text": [
    "Willkommen!\nBienvenue !\nWelcome!\n",
    "Willkommen!\nBienvenue !\nWelcome!\n",
    "Willkommen!\nBienvenue !\nWelcome!\n",
    "Willkommen!\nBienvenue !\nWelcome!\n",
    "Question: What is the answer to the ultimate question of life?\nAnswer: 42\n",
    "Question: What is the answer to the ultimate question of life?\nAnswer: 42\n",
    "Question: What is the answer to the ultimate question of life?\nAnswer: 42\n",
    "Question: What is the answer to the ultimate question of life?\nAnswer: 42\n"
  ]
}

This can look magical but that's one of the goals of ML models: return results as close as possible to human responses.

Confidence scores

We make mistakes, and so can ML models. To better appreciate the structured data you get, results include confidence scores:

Confidence scores do not represent accuracy.
They represent how confident the model is with the extracted results.
They let you — and your users — ponder the model’s extractions.

Let's overlay confidence scores on top of the previous example:

After grouping them in buckets, confidence scores appear to be generally high, with a few outliers:

The lowest confidence score here is 57%. It corresponds to a handwritten word (token) that is both short (less context given for confidence) and not particularly legible indeed:

{ // document.pages[0].tokens[71].layout
  "text_anchor": {
    // "42\n" ← document.text[351:354]
    "text_segments": [ { "start_index": "351", "end_index": "354" } ]
  },
  // 57% ← Confidence that this token is "42"
  "confidence": 0.57
}

For best results, keep in mind these general rules of thumb:

ML results are best guesses, which can be correct or incorrect.
Results with lower confidence scores are more likely to be incorrect guesses.
If we can't smoothly read a part of a document, or need to think twice about it, it's probably also harder for the ML model.

Although all text is correctly transcribed in the presented examples, this won't always be the case depending on the input document. To build safer solutions — especially with critical business applications — you may consider the following:

Be clear with your users that results are auto-generated.
Communicate the scope and limitations of your solution.
Improve the user experience with interpretable information or filterable results.
Design your processes to trigger human intervention on lower confidence scores.
Monitor your solution to detect changes over time (drift of stable metrics, decline in user satisfaction, etc.).

To learn more about AI principles and best practices, check out Responsible AI practices.

Rotation, skew, distortion

How many times did you scan a document upside down by mistake? Well, this shouldn't be a concern anymore. Text detection is very robust to rotation, skew, and other distortions.

In this example, the webcam input is not only upside down but also skewed, blurry, and with text in unusual orientations:

Before further analysis, Document AI considers the best reading orientation, at the page level, and preprocesses (deskews) each page if needed. This gives you results that can be used and visualized in a more natural way. Once processed by Document AI, the preceding example gets easier to read, without straining your neck:

In the results, each page has an image field by default. This represents the image — deskewed if needed — used by Document AI to extract information. All the page results and coordinates are relative to this image. When a page has been deskewed, a transforms element is present and contains the list of transformation matrices applied to the image:

{ // document.page[0]
  // The image for the page. All coordinates are relative to this image.
  "image": { "content": "…", "mime_type": "image/png", "width": 1150 /*…*/ },
  // 1 transformation matrix was applied to deskew the page
  "transforms": [ { "rows": 2, "cols": 3 /*…*/ } ]
}

Notes:

Page images can be in different formats. For instance, if your input is a JPEG image, the response will include either the same JPEG or a deskewed PNG (as in the earlier example).
Deskewed images have larger dimensions (deskewing adds blank outer areas).
If you don't need visual results in your solution, you can specify a field_mask in your request to receive lighter responses, with only your fields of interest.

Orientation

Documents don't always have all of their text in a single orientation, in which case deskewing alone is not enough. In this example, the sentence is broken out in four different orientations. Each part gets properly recognized and processed in its natural orientation:

Orientations are reported in the layout field:

[ // document.pages[0].blocks
  { "layout": { "orientation": "PAGE_RIGHT" } }, // → 90° clockwise
  { "layout": { "orientation": "PAGE_UP" } },    // ↑ Natural orientation
  { "layout": { "orientation": "PAGE_LEFT" } },  // ← 90° counterclockwise
  { "layout": { "orientation": "PAGE_DOWN" } }   // ↓ 180° from upright
]

Note: Orientations are returned at each OCR level (blocks, paragraphs, lines, and tokens).

Noise

When documents come from our analog world, you can expect … the unexpected. As ML models are trained from real-world samples — containing real-life noise — a very interesting outcome is that Document AI is also significantly robust to noise.

In this example with crumpled paper, the text starts to be difficult to read but still gets correctly transcribed by the OCR model:

Documents can also be dirty or stained. With the same sample, this keeps working after adding some layers of noise:

In both cases, the exact same text is correctly detected:

{ // document
  "text": "Coffee\nCoffee is a brewed beverage made from particular beans. Arthur Dent made himself some coffee on the\nThursday of Earth's destruction; this behaviour was most unusual, as he usually drank tea.\n"
}

You've seen most core features. They are supported by the "Document OCR" general processor as well as the other processors, which leverage these features to focus on more specific document types and provide additional information. Let's check the next-level processor: the "Form Parser" processor.

Form fields

The "Form Parser" processor lets you detect form fields. A form field is the combination of a field name and a field value, also called a key-value pair.

In this example, printed and handwritten text is detected as seen before:

In addition, the form parser returns a list of form_fields:

{ // document
  "text": "My name:\nDEEP THOUGHT\nYour question: What is the answer to the ultimate question?\nMy answer:\n42\n",
  "pages": [
    {
      "page_number": 1,
      "dimension": {/*…*/},
      "layout": {/*…*/},
      "detected_languages": [/*…*/],
      "blocks": [/*…*/],
      "paragraphs": [/*…*/],
      "lines": [/*…*/],
      "tokens": [/*…*/],
      "form_fields": [/* NEW */],
      "image": {/*…*/}
    }
  ]
}

Here is how the detected key-value pairs are returned:

{ // document.pages[0]
  "form_fields": [
    { // "My name:\n"
      "field_name":  { "text_anchor": {/*…*/}, "confidence": 0.96 /*,…*/ },
      // "DEEP THOUGHT\n"
      "field_value": { "text_anchor": {/*…*/}, "confidence": 0.96 /*,…*/ }
    },
    { // "Your question: "
      "field_name":  { "text_anchor": {/*…*/}, "confidence": 0.98 /*,…*/ },
      // "What is the answer to the ultimate question?\n"
      "field_value": { "text_anchor": {/*…*/}, "confidence": 0.98 /*,…*/ }
    },
    { // "My answer:\n"
      "field_name":  { "text_anchor": {/*…*/}, "confidence": 0.76 /*,…*/ },
      // "42\n"
      "field_value": { "text_anchor": {/*…*/}, "confidence": 0.76 /*,…*/ }
    }
  ]
}

And here are their detected bounding boxes:

Note: Form fields can follow flexible layouts. In this example, keys and values are in a left-right order. You'll see a right-left example next. Those are just simple arbitrary examples. It also works with vertical or free layouts where keys and values are logically (visually) related.

Checkboxes

The form parser also detects checkboxes. A checkbox is actually a particular form field value.

This example is a French exam with affirmations that should be checked when exact. To test this, I used checkboxes of different kinds, printed or handmade. All form fields are detected, with the affirmations as field names and the corresponding checkboxes as field values:

When a checkbox is detected, the form field contains an additional value_type field, which value is either unfilled_checkbox or filled_checkbox:

{ // document.pages[0]
  "form_fields": [
    { // [ ] Les cyclines se lient aux CDK…
      "field_name": {/*…*/},
      "field_value": {/*…*/},
      "value_type": "unfilled_checkbox"
    },
    { // [x] Les mutations peuvent être provoquées…
      "field_name": {/*…*/},
      "field_value": {/*…*/},
      "value_type": "filled_checkbox"
    } //,…
  ]
}

Being able to analyze forms can lead to huge time savings, by consolidating — or even autoprocessing — content for you.
The preceding checkbox detection example was actually an evolution of a prior experiment to autocorrect my wife's pile of exam copies.
The proof of concept got better using checkboxes, but was already conclusive enough with True/False handwritten answers. Here is how it can autocorrect and autograde:

Tables

The form parser can also detect another important structural element: tables.

In this example, words are presented in a tabular layout without any borders. The form parser finds a table very close to the (hidden) layout. Here are the detected cells:

In this other example, some cells are filled with text while others are blank. There are enough signals for the form parser to detect a tabular structure:

When tables are detected, the form parser returns a list of tables with their rows and cells. Here is how the table is returned:

{ // document.pages[0]
  "tables": [{
      "layout": {/*…*/},
      "header_rows": [{
        // | ALPHA |       |     |     | EPSILON |     | ETA |       |
        "cells": [/*…*/]
      }],
      "body_rows": [{
        // |       | KAPPA |     | MU  |         | XI  |     | PI    |
        "cells": [/*…*/]
      },
      {
        // | RHO   |       | TAU |     | PHI     |     |     | OMEGA |
        "cells": [/*…*/]
      }]
  }]
}

And here is the first cell:

{ // document.pages[0].tables[0].header_rows[0].cells[0]
  "layout": {
    // ALPHA\n
    "text_anchor": {/*…*/},
    "confidence": 0.9979,
    "bounding_poly": {/*…*/},
    "orientation": "PAGE_UP"
  },
  "row_span": 1,
  "col_span": 1
}

Specialized processors

Specialized processors focus on domain-specific documents and extract entities. They cover many different document types that can currently be classified in the following families:

Procurement — receipts, invoices, utility bills, purchase orders,…
Lending — bank statements, pay slips, official forms,…
Identity — national IDs, driver licenses, passports,…
Contract — legal agreements

For example, procurement processors typically detect the total_amount and currency entities:

{ // document
  "entities": [
    {
      "text_anchor": {/*…*/},
      "type_": "total_amount",
      "mention_text": "15",
      "confidence": 0.96,
      "page_anchor": {/*…*/},
      "id": "0"
    },
    {
      "text_anchor": {/*…*/},
      "type_": "currency",
      "mention_text": "USD",
      "confidence": 0.95,
      "page_anchor": {/*…*/},
      "id": "1"
    }
  ]
}

For more information, check out Fields detected.

Expenses

The "Expense Parser" lets you process receipts of various types. Let's analyze this actual (French) receipt:

A few remarks:

All expected entities are extracted.
Ideally (to be picky), I'd like to get the tax rate too. If there's customer demand for it, this may be a supported entity in a future version.
Due to the receipt's very thin paper, the text on the back side is visible by transparency (another type of noise).
The supplier name is wrong (it's actually mirrored text from the back side) but with a very low confidence (4%). You'd typically handle it with special care (it's shown differently here) or ignore it.
The actual supplier info is hidden (the top part of the receipt is folded) on purpose. You'll see another example with supplier data a bit later.
Receipts are often printed on single (sometimes on multiple) pages. The expense parser supports analyzing expense documents of up to 10 pages.

Procurement documents often list parts of the data in tabular layouts. Here, they're returned as many line_item/* entities. When the entities are detected as part of a hierarchical structure, the results are nested in the properties field of a parent entity, providing an additional level of information. Here's an excerpt:

{ // document.entities[7]
  "text_anchor": {/*…*/},
  "type_": "line_item",
  "mention_text": "*BANANE BIO 1.99",
  "confidence": 1.0,
  "page_anchor": {/*…*/},
  "id": "9",
  "properties": [
    {
      "text_anchor": {/*…*/},
      "type_": "line_item/description",
      "mention_text": "*BANANE BIO",
      "confidence": 0.54613465,
      "page_anchor": {/*…*/},
      "id": "10"
    },
    {
      "text_anchor": {/*…*/},
      "type_": "line_item/amount",
      "mention_text": "1.99",
      "confidence": 0.36692643,
      "page_anchor": {/*…*/},
      "id": "11"
    }
  ]
}

For more information, see the Expense Parser details.

Entity normalization

Getting results is generally not enough. Results often need to be handled in a post-processing stage, which can be both time consuming and a source of errors. To address this, specialized processors also return normalized values when possible. This lets you directly use standard values consolidated from the context of the whole document.

Let's check it with this other receipt:

First, the receipt currency is returned with its standard code under normalized_value:

{ // document.entities[0]
  "type_": "currency",
  "mention_text": "€",
  "normalized_value": {
    "text": "EUR"
  } //,…
}

Then, the receipt is dated 11/12/2022. But is it Nov. 12 or Dec. 11? Document AI uses the context of the document (a French receipt) and provides a normalized value that removes all ambiguity:

{ // document.entities[1]
  "type_": "receipt_date",
  "mention_text": "11/12/2022",
  "normalized_value": {
    "text": "2022-12-11",
    "date_value": {
      "year": 2022,
      "month": 12,
      "day": 11
    }
  } //,…
}

Likewise, the receipt contains a purchase time, written in a non-standard way. The result also includes a canonical value that avoids any interpretation:

{ // document.entities[5]
  "type_": "purchase_time",
  "mention_text": "12h42\n",
  "normalized_value": {
    "datetime_value": {
      "hours": 12,
      "minutes": 42
    }
  } //,…
}

Normalized values simplify the post-processing stage:

They provide standard values that are straightforward to use (e.g. enable direct storage in a data warehouse).
They prevent bugs (esp. the recurring developer mistakes we make when converting data).
They remove ambiguity and avoid incorrect interpretations by using the context of the whole document.

For more information, check out the NormalizedValue structure.

Entity enrichment

Did you notice there was more information in the receipt?

"Maison Jeanne d'Arc, Place de Gaulle" mentions a Joan of Arc's House and a place name. Nonetheless, there is no address, zip code, city, or even country.
This receipt comes most likely from a museum in France, but Joan of Arc lived in a few places (starting with her birthplace in the Domrémy village).
So, which location does this correspond to?
A manual search should give hints to investigate, but can we keep this automated?

Extracting the information behind the data requires extra knowledge or human investigation.
An automated solution would generally ignore this partial data, but Document AI handles this in a unique way.
To understand the world's information, Google has been consistently analyzing the web for over 20 years.
The result is a gigantic up-to-date knowledge base called the Knowledge Graph.
Document AI leverages this knowledge graph to normalize and enrich entities.

First, the supplier is correctly detected and normalized with its usual name:

{ // document.entities[6]
  "type_": "supplier_name",
  "normalized_value": {
    "text": "Maison de Jeanne d'Arc"
  } //,…
}

Then, the supplier city is also returned:

{ // document.entities[8]
  "type_": "supplier_city",
  "normalized_value": {
    "text": "Orléans"
  } //,…
}

Note: Joan of Arc spent some time in Orléans in 1429, as she led the liberation of the besieged city at the age of 17 (but that's another story).

And finally, the complete and canonical supplier address is also part of the results, closing our case here:

{ // document.entities[7]
  "type_": "supplier_address",
  "normalized_value": {
    "text": "3 Pl. du Général de Gaulle\n45000 Orléans\nFrance"
  } //,…
}

Enriched entities bring significant value:

They are canonical results, which avoids information conflicts and discrepancies.
They can add information, which prevents ignoring useful data.
They can complete or fix partially correct data.
In a nutshell, they provide information that is reliable, consistent, and comparable.

Here is a recap of the expected — as well as the non-obvious — entities detected in the receipt:

Note: The slightest characteristic element can be sufficient to detect an entity.
I've been regularly surprised to get entities I wasn't expecting, to eventually realize I had missed clues in the document.
For example, I recently wondered how the expense processor was able to identify a specific store.
The receipt only specified a zip code and the retail chain has several stores in my neighborhood.
Well, a public phone number (hidden in the footer) was enough to uniquely identify the store in question and provide its full address.

To learn more about the Knowledge Graph and possible enrichments, check out Enrichment and normalization.

Invoices

Invoices are the most elaborate type of procurement documents, often spreading over multiple pages.

Here's an example showing entities extracted by the "Invoice Parser":

A few remarks:

The relevant information is extracted (even the supplier tax ID is detected on the 90°-rotated left side).
As seen earlier with the expense parser, the invoice parser extracts many line_item/* entities but also supplier_* and receiver_* info.
This is a typical invoice (issued by the historical French energy provider) with many numbers on the first pages.
The original source is a PDF (digital) that was anonymized, printed, stained, and scanned as a PDF (raster).

For more information, see the Invoice Parser details.

Barcodes

Barcode detection is enabled for some processors. In this example, the invoice parser detects the barcode (a manifest number):

Note: I didn't have an invoice with barcodes at hand, so I used a (slightly anonymized) packing list.

Barcodes are returned, at the page level, like this:

{ // document.pages[0]
  "detected_barcodes": [
    {
      "layout": {/*…*/},
      "barcode": {
        "format_": "CODE_39",
        "value_format": "TEXT",
        "raw_value": "E-58702865"
      }
    }
  ]
}

For more information, check out the DetectedBarcode structure.

Identity documents

Identity processors let you extract identity entities. In this US passport specimen (credit: Bureau of Consular Affairs), the expected fields can be automatically extracted:

For more details about identity processors, you can read my previous article Automate identity document processing.

Document signals

Some processors return document signals — information relative to the document itself.

For example, the "Document OCR" processor returns quality scores for the document, estimating the defects that might impact the accuracy of the results.

The preceding crumpled paper example gets a high quality score of 95%, with glare as a potential defect:

The same example, at a 4x lower resolution, gets a lower quality score of 53%, with blurriness detected as the main potential issue:

Some additional remarks:

A low score lets you flag documents that may need a manual review (or a new better capture).
Surprisingly, the exact same text is correctly extracted in both cases (I checked twice) but that might not always be the case.
The quality scores are returned both as entities (with a quality_score parent entity) and in the image_quality_scores field at the page level (see ImageQualityScores).
This was tested with release candidate pretrained-ocr-v1.1-2022-09-12.

Here's how the image quality scores are returned at the page level:

{ // document.pages[0]
  "image_quality_scores": {
    "quality_score": 0.53,
    "detected_defects": [
      { "confidence": 0.98, "type_": "quality/defect_blurry" },
      { "confidence": 0.77, "type_": "quality/defect_glare" },
      { "confidence": 0.16, "type_": "quality/defect_text_too_small" },
      { "confidence": 0.05, "type_": "quality/defect_noisy" },
      { "confidence": 0.03, "type_": "quality/defect_faint" },
      { "confidence": 0.01, "type_": "quality/defect_dark" }
    ]
  }
}

Likewise, the "Identity Proofing" processor gives you signals about the validity of ID documents.

First, this can help detect whether documents look like IDs. This preceding example is a random document analyzed as NOT_AN_ID:

Here are the corresponding entities:

{ // document
  "entities": [
    {
      "type_": "fraud_signals_is_identity_document",
      "mention_text": "NOT_AN_ID" //,…
    },
    {
      "type_": "fraud_signals_suspicious_words",
      "mention_text": "PASS" //,…
    },
    {
      "type_": "fraud_signals_image_manipulation",
      "mention_text": "PASS" //,…
    },
    {
      "type_": "fraud_signals_online_duplicate",
      "mention_text": "PASS" //,…
    }
  ]
}

The preceding passport example does get detected as an ID but also triggers useful fraud signals:

The results include evidences that #1 it's a specimen, #2 which can be found online:

{ // document
  "entities": [
    {
      "type_": "fraud_signals_is_identity_document",
      "mention_text": "PASS" //,…
    },
    {
      "type_": "fraud_signals_suspicious_words",
      "mention_text": "SUSPICIOUS_WORDS_FOUND" //,…
    },
    {
      "type_": "evidence_suspicious_word",
      "mention_text": "SPECIMEN" //,…
    },
    {
      "type_": "fraud_signals_image_manipulation",
      "mention_text": "POSSIBLE_IMAGE_MANIPULATION" //,…
    },
    {
      "type_": "evidence_thumbnail_url",
      "mention_text": "https://….gstatic.com/images?…" //,…
    },
    {
      "type_": "evidence_hostname",
      "mention_text": "pbs.twimg.com" //,…
    },
    {
      "type_": "fraud_signals_online_duplicate",
      "mention_text": "POSSIBLE_ONLINE_DUPLICATE" //,…
    }
  ]
}

As new use cases appear, it's likely that some processors will extract new document signals. Follow the Release notes to stay updated.

Pre-trained processors

Document AI is already huge and keeps evolving. Here is a screencast showing the current processor gallery and how to create a processor from the Cloud Console:

Custom processors with Workbench

If you have your own business-specific documents, you may wish to extract custom entities not covered by the existing processors. Document AI Workbench can help you solve this by creating your own custom processor, trained with your own documents, in two ways:

With uptraining, you can extend an existing processor.
You can also create a new processor from scratch.

For more information, watch this great introduction video made by my teammate Holt:

Performance

To capture your documents, you'll probably use either:

A scanner
A camera

With scanners, you'll need to choose a resolution in dots per inch (dpi). To optimize the performance (especially the accuracy and consistency of the results), you can keep in mind the following:

300 dpi is often a nice spot
200 dpi is generally a hard minimum

Here are the resolutions needed to capture a sheet of paper:

Note: Dimensions are presented in the horizontal orientation (image sensors generally have a landscape native resolution).

With cameras, the captured dots per inch depend on the camera resolution but also on the way you zoom on the document. Here's an example with an A4 sheet of paper:

This translates into different dpi ranges. Here are indicative values:

To give you an idea, here is how a PNG image size evolves when capturing the same document at different resolutions. The total number of pixels is a surface. Though the PNG compression slightly limits the growth, the size increases almost quadratically:

A few general observations:

Document captures are usually in the 100-600 dpi range, depending on the use case.
100-200 dpi may be enough for some specific end user solutions, for example to process small receipts captured from mobile phone cameras or laptop webcams.
Scans of 300-600 dpi may provide better results but will generate larger files and make your processing pipeline slower.
Scans above 600 dpi are less likely. They are useful for very small documents (such as slides or negative films) or documents that need to be upscaled (e.g. to print a poster).
If your use case involves documents with small text, barcodes, or very noisy (e.g. blurry) inputs, you'll typically need to compensate with a higher capture resolution.
Capture devices have noise intrinsic to the technology they use. They generally have options that will produce different results. Also, image sensors are more noisy in low-light environments.
Every conversion step in your architecture is likely to introduce side effects and a potential loss of information.
Lossy image formats (such as JPEG) can nicely reduce image sizes. On the other hand, too high compression levels (a setting) can generate artifacts with a negative impact on the detection performance (causing a strong degradation of image quality and consequently of result accuracy).
Lossless formats (such as PNG) imply larger file sizes but may bring the benefit of using lower resolutions. They may also be more future-proof (e.g. for archival) by preserving the possibility of later conversions to other formats or to smaller resolutions without losing information.
Storage costs are rather low, so it may also be interesting to consider higher acceptable scanning resolutions, to optimize both the performance and the overall cost of your solution.
In a nutshell, choosing capture devices, resolutions, and formats depends on your use case and is a tradeoff between accuracy, speed, and cost.

To finetune your solution and get more accurate, faster, or consistent results, you may consider the following:

Check the performance of your solution with worst-case documents at different resolutions.
Have a clear understanding of the different formats used in your processing pipeline.
Test your solution with actual documents (as close as possible to what you'll have in production). Also, as the ML models are trained on real documents, you may get better results when testing real documents.
Keep a lossless format upstream when possible; documents can be converted to a lossy format downstream.
Evaluate whether capturing at a higher resolution and then downscaling produces better results.
When using lossy formats, finetune the compression level. There is usually a threshold at which compression artifacts become negligible while maintaining a good compression ratio.
When using cameras, control your lighting environment and avoid low-light conditions.
Make your users aware that blurry captures may decrease the performance.

For example:

The examples in this article can all be analyzed using cameras with resolutions ranging from 100 to 220 dpi. However, results won't always be consistent with low resolutions.
For practical reasons, I mostly use laptop, mobile phone, or dedicated cameras. There are excellent ones (called document cameras) that can sit on your desk and provide solid captures with a fast auto-focus.
For an enterprise-grade solution, I'd probably stick to 300 dpi scans. Multifunction printers generally have scanners that provide excellent PNG, PDF, or TIFF captures.
The barcode (contained in the preceding packing list sample) starts to be detected at 150 dpi with a scanner, and at 200 dpi with a webcam. After being photocopied (i.e. scanned + printed), it's still detected at 150 dpi with a scanner but needs to be zoomed in with the webcam.
While capturing documents from a browser and a webcam, I checked the HTML canvas default format: it's PNG (great!) but in RGBA (with a transparency channel, unnecessary for document images). This made the data larger without any benefit. The optimization was a parameter away (canvas.getContext('2d') → canvas.getContext('2d', { alpha: false })) and made the captures sent to the backend 9 to 18% smaller.

Sample demo

I put myself in your shoes to see what it takes to build a document processing prototype and made the following choices:

A frontend using vanilla JavaScript, responsible for managing user interactions and input documents
A backend using the Python client library, responsible for all document processings and renderings
Synchronous calls to Document AI, to process documents in real-time
No cloud storage, to avoid storing personally identifiable information (analyses are stored client-side)

Here is the chosen software stack based on open-source Python projects:

… and one possible architecture to deploy it to production using Cloud Run:

Here is the core function I use to process a document live in Python:

from typing import BinaryIO

from google.cloud.documentai_v1 import (
    Document,
    DocumentProcessorServiceClient,
    ProcessRequest,
    RawDocument,
)


def process_document(
    file: BinaryIO,
    mime_type: str,
    project: str,
    location: str,
    processor_id: str,
) -> Document:
    """Analyze the input file with Document AI."""
    client_options = {"api_endpoint": f"{location}-documentai.googleapis.com"}
    client = DocumentProcessorServiceClient(client_options=client_options)
    raw_document = RawDocument(content=file.read(), mime_type=mime_type)
    name = client.processor_path(project, location, processor_id)

    request = ProcessRequest(
        raw_document=raw_document,
        name=name,
        skip_human_review=True,
    )
    response = client.process_document(request)

    return response.document

To make sure that "what you see is what you get", the sample images and animations in this article have been generated by the demo:

Note: To avoid making repeated and unnecessary calls to Document AI while developing the app, sample document analyses are cached (JSON serialization). This lets you check the returned structured documents and also explains why responses are immediate in this (real-time, not sped up) screencast.

It takes seconds to analyze a document. Here is an example with the user uploading a PDF scan:

You can also take camera captures from the web app:

Check out the source code and feel free to reuse it. You'll also find instructions to deploy it as a serverless app.

More?

Test documents? → Try Document AI in your browser.
Video series? → Watch The future of documents.
New to Google Cloud? → Check out the $300 free trial offer.
Learn and grow? → Become a Google Cloud Innovator.
Feedback or questions? → Reach out on Twitter (@PicardParis) or LinkedIn (in/PicardParis).

Automate identity document processing with Document AI

Laurent Picard — Thu, 02 Jun 2022 13:07:59 +0000

How many times have you filled out forms requesting personal information? It's probably too many times to count. When online and signed in, you can save a lot of time thanks to your browser’s autofill feature. In other cases, you often have to provide the same data manually, again and again. The first Document AI identity processors are now generally available and can help you solve this problem.

In this post, you’ll see how to…

Process identity documents with Document AI
Create your own identity form autofiller

Use cases

Here are a few situations that you've probably encountered:

Financial accounts: Companies need to validate the identity of individuals. When creating a customer account, you need to present a government-issued ID for manual validation.
Transportation networks: To handle subscriptions, operators often manage fleets of custom identity-like cards. These cards are used for in-person validation, and they require an ID photo.
Identity gates: When crossing a border (or even when flying domestically), you need to pass an identity check. The main gates have streamlined processes and are generally well equipped to scale with the traffic. On the contrary, smaller gates along borders can have manual processes – sometimes on the way in and the way out – which can lead to long lines and delays.
Hotels: When traveling abroad and checking in, you often need to show your passport for a scan. Sometimes, you also need to fill out a longer paper form and write down the same data.
Customer benefits: For benefit certificates or loyalty cards, you generally have to provide personal info, which can include a portrait photo.

In these examples, the requested info – including the portrait photo – is already on your identity document. Moreover, an official authority has already validated it. Checking or retrieving the data directly from this source of truth would not only make processes faster and more effective, but also remove a lot of friction for end users.

Identity processors

Processor types

Each Document AI identity processor is a machine learning model trained to extract information from a standard ID document such as:

Driver license
National ID
Passport

Note: an ID can have information on both sides, so identity processors support up to two pages per document.

Availability

Generally available as of June 2022, you can use two US identity processors in production:

Processor	API `type`
US Driver License Parser	`US_DRIVER_LICENSE_PROCESSOR`
US Passport Parser	`US_PASSPORT_PROCESSOR`

Currently available in Preview:

The Identity Doc Fraud Detector, to check whether an ID document has been tampered with
Three French identity processors

Processor	API `type`
Identity Doc Fraud Detector	`ID_FRAUD_DETECTION_PROCESSOR`
France Driver License Parser	`FR_DRIVER_LICENSE_PROCESSOR`
France National ID Parser	`FR_NATIONAL_ID_PROCESSOR`
France Passport Parser	`FR_PASSPORT_PROCESSOR`

Notes:

More identity processors are in the pipe.
To request access to processors in Preview, please fill out the Access Request Form.

Processor creation

You can create a processor:

Manually from Cloud Console (web admin UI)
Programmatically with the API

Processors are location-based. This helps guarantee where processing will occur for each processor.

Here are the current multi-region locations:

Location	API `location`
United States	`us`
European Union	`eu`

Once you've created a processor, you reference it with its ID (PROCESSOR_ID hereafter).

Note: To manage processors programmatically, see the codelab Managing Document AI processors with Python.

Document processing

You can process documents in two ways:

Synchronously with an online request, to analyze a single document and directly use the results
Asynchronously with a batch request, to launch a batch processing operation on multiple or larger documents

Online requests

Example of a REST online request:

The method is named process.
The input document here is a PNG image (base64 encoded).
This request is processed in the European Union.
The response is returned synchronously.

POST https://eu-documentai.googleapis.com
     /v1/projects/PROJECT_ID/locations/eu/processors/PROCESSOR_ID
     :process

{
  "rawDocument": {
    "content": "iVBORw0KGg…",
    "mimeType": "image/png"
  },
  "skipHumanReview": true
}

Batch requests

Example of a REST batch request:

The method is named batchProcess.
The batchProcess method launches the batch processing of multiple documents.
This request is processed in the United States.
The response is returned asynchronously; output files will be stored under my-storage-bucket/output/.

POST https://us-documentai.googleapis.com
     /v1/projects/PROJECT_ID/locations/us/processors/PROCESSOR_ID
     :batchProcess

{
  "inputDocuments": {
    "gcsDocuments": {
      "documents": [
        {
          "gcsUri": "gs://my-storage-bucket/input/id-doc-1.pdf",
          "mimeType": "application/pdf"
        },
        {
          "gcsUri": "gs://my-storage-bucket/input/id-doc-2.tiff",
          "mimeType": "image/tiff"
        },
        {
          "gcsUri": "gs://my-storage-bucket/input/id-doc-3.png",
          "mimeType": "image/png"
        },
        {
          "gcsUri": "gs://my-storage-bucket/input/id-doc-4.gif",
          "mimeType": "image/gif"
        }
      ]
    }
  },
  "documentOutputConfig": {
    "gcsOutputConfig": {
      "gcsUri": "gs://my-storage-bucket/output/"
    }
  },
  "skipHumanReview": true
}

Interfaces

Document AI is available through the usual Google Cloud interfaces:

The RPC API (low-latency gRPC)
The REST API (JSON requests and responses)
Client libraries (gRPC wrappers, currently available for Python, Node.js, and Java)
Cloud Console (web admin UI)

Note: With the client libraries, you can develop in your preferred programming language. You'll see an example later in this post.

Identity fields

A typical REST response looks like the following:

The text and pages fields include the OCR data detected by the underlying ML models. This part is common to all Document AI processors.
The entities list contains the fields specifically detected by the identity processor.

{
  "text": "…",
  "pages": […],
  "entities": [
    {
      "textAnchor": {…},
      "type": "Family Name",
      "mentionText": "PICHAI",
      "confidence": 0.999945,
      "pageAnchor": {…},
      "id": "4"
    },
    {
      "textAnchor": {…},
      "type": "Given Names",
      "mentionText": "Sundar",
      "confidence": 0.9999612,
      "pageAnchor": {…},
      "id": "5"
    }
    ,…
  ]
}

Here are the detectable identity fields:

Entity type	Comment
`Portrait`	Bounding box (portrait photo)
`Family Name`	String
`Given Names`	String
`Document Id`	String
`Expiration Date`	Date (+ normalization)
`Date Of Birth`	Date (+ normalization)
`Issue Date`	Date (+ normalization)
`Address`	String
`MRZ Code`	String (Machine Readable Zone)

Please note that Address and MRZ Code are optional fields. For example, a US passport contains an MRZ but no address.

Fraud detection

Available in preview, the Identity Doc Fraud Detector helps detect tampering attempts. Typically, when an identity document does not "pass" the fraud detector, your automated process can block the attempt or trigger a human validation.

Here is an example of signals returned:

Entity type	Example
`fraud-signals/is-identity-document`	"NOT_AN_ID"
`fraud-signals/suspicious-words`	"PASS"
`fraud-signals/image-manipulation`	"POSSIBLE_IMAGE_MANIPULATION"

Sample demo

You can process a document live with just a few lines of code.

Here is a Python example:

import google.cloud.documentai_v1 as docai


def process_document(
    file: typing.BinaryIO,
    mime_type: str,
    project_id: str,
    location: str,
    processor_id: str,
) -> docai.Document:
    api_endpoint = {"api_endpoint": f"{location}-documentai.googleapis.com"}
    client = docai.DocumentProcessorServiceClient(client_options=api_endpoint)

    raw_document = docai.RawDocument(content=file.read(), mime_type=mime_type)
    name = client.processor_path(project_id, location, processor_id)

    request = docai.ProcessRequest(
        raw_document=raw_document,
        name=name,
        skip_human_review=True,
    )
    response = client.process_document(request)

    return docai.Document(response.document)

This function uses the Python client library:

The input is a file (any format supported by the processor).
client is an API wrapper (configured for processing to take place in the desired location).
process_document calls the API process method, which returns results in seconds.
The output is a structured Document.

You can collect the detected fields by parsing the document entities:

def id_data_from_document(document: docai.Document) -> dict:
    id_data = defaultdict(dict)

    for entity in document.entities:
        key = entity.type_
        page_index = 0
        value = entity.mention_text
        confidence, normalized = None, None

        if entity.page_anchor:
            page_index = entity.page_anchor.page_refs[0].page
        if not value:  # Send the detected portrait image instead
            image = crop_entity(document, entity)
            value = data_url_from_image(image)
        if entity.confidence != 0.0:
            confidence = int(entity.confidence * 100 + 0.5)
        if entity.normalized_value:
            normalized = entity.normalized_value.text

        id_data[key][page_index] = dict(
            value=value,
            confidence=confidence,
            normalized=normalized,
        )

    return id_data

Note: This function builds a mapping ready to be sent to a frontend. A similar function can be used for other specialized processors.

Finalize your app:

Define your user experience and architecture
Implement your backend and its API
Implement your frontend with a mix of HTML + CSS + JS
Add a couple of features: file uploads, document samples, or webcam captures
That's it; you've built an identity form autofiller

Here is a sample web app in action:

Here is the processing of a French national ID, dropping images from the client:

Note: For documents with multiple pages, you can use a PDF or TIFF container. In this example, the two uploaded PNG images are merged by the backend and processed as a TIFF file.

And this is the processing of a US driver license, captured with a laptop 720p webcam:

Notes:

Did you notice that the webcam capture is skewed and the detected portrait image straight? That's because Document AI automatically deskews the input at the page level. Documents can even be upside down.
Some fields (such as the dates) are returned with their normalized values. This can make storing and processing these values a lot easier – and less error-prone – for developers.

The source code for this demo is available in our Document AI sample repository.

Stay tuned; the family of Document AI processors keeps growing and growing.

🎨 Image processing as a service 🐍

Laurent Picard — Thu, 07 Apr 2022 07:56:59 +0000

👋 Hello

Have you ever written a script to transform an image? Did you share the script with others or did you run it on multiple computers? How many times did you need to update the script or the setup instructions? Did you end up making it a service or an online app? If your script is useful, you’ll likely want to make it available to others. Deploying processing services is a recurring need – one that comes with its own set of challenges. Serverless technologies let you solve these challenges easily and efficiently.

In this post, you’ll see how to…

Create an image processing service that generates coloring pages
Make it available online using minimal resources

…and do it all in less than 200 lines of Python and JavaScript!

🛠️ Tools

To build and deploy a coloring page generator, you’ll need a few tools:

A library to process images
A web application framework
A web server
A serverless solution to make the demo available 24/7

🧱 Architecture

Here is one possible architecture for a coloring page generator using Cloud Run:

And here is the workflow:

1 - The user opens the web app: the browser requests the main page.
2 - Cloud Run serves the app HTML code.
3 - The browser requests the additional needed resources.
4 - Cloud Run serves the CSS, JavaScript, and other resources.
A - The user selects an image and the frontend sends the image to the /api/coloring-page endpoint.
B - The backend processes the input image and returns an output image, which the user can then visualize, download, or print via the browser.

🐍 Software stack

Of course, there are many different software stacks that you could use to implement such an architecture.

Here is a good one based on Python:

It includes:

Gunicorn: A production-grade WSGI HTTP server
Flask: A popular web app framework
scikit-image: An extensive image processing library

Define these app dependencies in a file named requirements.txt:

# https://pypi.org/project/gunicorn
gunicorn==20.1.0

# https://pypi.org/project/flask
Flask==2.1.1

# https://pypi.org/project/scikit-image
# scikit-image dependencies include NumPy and Pillow
scikit-image==0.19.2

🎨 Image processing

How do you remove colors from an image? One way is by detecting the object edges and removing everything but the edges in the result image. This can be done with a Sobel filter, a convolution filter that detects the regions in which the image intensity changes the most.

Create a Python file named main.py, define an image processing function, and within it use the Sobel filter and other functions from scikit-image:

import numpy as np
import skimage
from PIL import Image
from PIL.Image import Image as PilImage


def generate_coloring_page(input: PilImage) -> PilImage:
    # Convert to grayscale if needed
    if input.mode != "L":
        input = input.convert("L")
    np_image = np.asarray(input)

    # Detect the edges
    np_image = skimage.filters.sobel(np_image)
    # Convert to 8 bpp
    np_image = skimage.util.img_as_ubyte(np_image)
    # Invert to get dark edges on a light background
    np_image = 255 - np_image
    # Improve the contrast
    np_image = skimage.exposure.rescale_intensity(np_image)

    return Image.fromarray(np_image)

Note: The NumPy and Pillow libraries are automatically installed as dependencies of scikit-image.

As an example, here is how the Cloud Run logo is processed at each step:

✨ Web app

Backend

To expose both endpoints (GET / and POST /api/coloring-page), add Flask routes in main.py:

import io

import flask
from PIL import Image

app = flask.Flask(__name__, static_url_path="")


@app.get("/")
def index():
    return app.send_static_file("index.html")


@app.post("/api/coloring-page")
def coloring_page():
    file = flask.request.files.get("input-image")
    if file is None:
        return "Missing input-image parameter", 400

    input_image = Image.open(file.stream)
    output_image = generate_coloring_page(input_image)

    image_io = io.BytesIO()
    output_format = "png"
    output_image.save(image_io, format=output_format)
    image_io.seek(0)

    return flask.send_file(image_io, mimetype=f"image/{output_format}")

Frontend

On the browser side, write a JavaScript function that calls the /api/coloring-page endpoint and receives the processed image:

async function fetchColoringPage(inputFile) {
    const formData = new FormData()
    formData.append('input-image', inputFile)

    const url = '/api/coloring-page'
    const init = { method: 'POST', body: formData }
    try {
        const response = await fetch(url, init)
        return response.ok ? response.blob() : null
    } catch (error) {
        console.error(error)
        return null
    }
}

The base of your app is there. Now you just need to add a mix of HTML + CSS + JS to complete the desired user experience.

Local development

To develop and test the app on your computer, once your environment is set up, make sure you have the needed dependencies:

pip install --upgrade -r requirements.txt

Add the following block to main.py. It will only execute when you run your app manually:

import os

# ...

if __name__ == "__main__":
    os.environ["FLASK_ENV"] = "development"
    app.run(host="localhost", port=8080, debug=True)

Run your app:

python main.py

Flask starts a local web server:

 * Serving Flask app 'main' (lazy loading)
 * Environment: development
 * Debug mode: on
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 718-408-327
 * Running on http://localhost:8080/ (Press CTRL+C to quit)

Note: In this mode, you’re using a development web server (one that is not suited for production). You’ll next set up the deployment to serve your app with Gunicorn, a production-grade server.

You're all set. Open localhost:8080 in your browser, test, refine, and iterate.

🚀 Deployment

Once your app is ready for prime time, you can define how it will be served with this single line in a file named Procfile:

web: gunicorn --bind :$PORT --workers 1 --threads 8 --timeout 0 main:app

At this stage, here are the files found in a typical project:

.
├── main.py
├── Procfile
├── requirements.txt
└── static
    ├── favicon.ico
    ├── index.html
    ├── scripts.js
    └── styles.css

That's it, you can now deploy your app from the source folder:

SERVICE="coloring-page-generator"
SOURCE="."

gcloud run deploy $SERVICE --source $SOURCE --allow-unauthenticated

⚙️ Under the hood

The command line output details all the different steps:

This command is equivalent to running `gcloud builds submit --pack image=[IMAGE] SOURCE` and `gcloud run deploy SERVICE --image [IMAGE]`

Building using Buildpacks and deploying container to Cloud Run service [SERVICE] in project [PROJECT_ID] region [REGION]
OK Building and deploying... Done.
  OK Creating Container Repository...
  OK Uploading sources...
  OK Building Container... Logs are available at […].
  OK Creating Revision...
  OK Routing traffic...   
  OK Setting IAM Policy...
Done.
Service [SERVICE] revision [SERVICE-REVISION] has been deployed and is serving 100 percent of traffic.
Service URL: https://SERVICE-PROJECTHASH-REGIONID.a.run.app

Cloud Build is indirectly called to containerize your app. One of its core components is Google Cloud Buildpacks, which automatically builds a production-ready container image from your source code. Here are the main steps:

Cloud Build fetches the source code.
Buildpacks autodetects the app language (Python, in this case) and uses the corresponding secure base image.
Buildpacks installs the app dependencies (defined in requirements.txt for Python).
Buildpacks configures the service entrypoint (defined in Procfile for Python).
Cloud Build pushes the container image to Artifact Registry.
Cloud Run creates a new revision of the service based on this container image.
Cloud Run routes production traffic to it.

Notes:

Buildpacks currently supports the following runtimes: Go, Java, .NET, Node.js, and Python.
The base image is actively maintained by Google, scanned for security vulnerabilities, and patched against known issues. This means that, when you deploy an update, your service is based on an image that is as secure as possible.
If you need to build your own container image, for example with a custom runtime, you can add your own Dockerfile and Buildpacks will use it instead.

💫 Updates

More testing from real-life users shows some issues.

First, the app does not handle pictures taken with digital cameras in non-native orientations. You can fix this using the EXIF orientation data:

-from PIL import Image
+from PIL import Image, ImageOps
...
def generate_coloring_page(input: PilImage) -> PilImage:
    # Convert to grayscale if needed
    if input.mode != "L":
        input = input.convert("L")
+   # Transpose if taken in non-native orientation (rotated digital camera)
+   NATIVE_ORIENTATION = 1
+   if input.getexif().get(0x0112, NATIVE_ORIENTATION) != NATIVE_ORIENTATION:
+       input = ImageOps.exif_transpose(input)
    np_image = np.asarray(input)
    ...

In addition, the app is too sensitive to details in the input image. Textures in paintings, or noise in pictures, can generate many edges in the processed image. You can improve the processing algorithm by adding a denoising step upfront:

...
def generate_coloring_page(input: PilImage) -> PilImage:
    ...
+   # Remove some noise to keep the most visible edges
+   np_image = skimage.restoration.denoise_tv_chambolle(np_image, weight=0.05)
    # Detect the edges
    np_image = skimage.filters.sobel(np_image)
    ...

This additional step makes the coloring page cleaner and reduces the quantity of ink used if you print it:

Redeploy, and the app is automatically updated:

gcloud run deploy $SERVICE --source $SOURCE

🎉 It's alive

The app is visible as a service in Cloud Run:

The service dashboard gives you an overview of app usage:

That's it; your image processing app is in production!

🤯 It's serverless

There are many benefits to using Cloud Run in this architecture:

Your app is available 24/7.
The environment is fully managed: you can focus on your code and not worry about the infrastructure.
Your app is automatically available through HTTPS.
You can map your app to a custom domain.
Cloud Run scales the number of instances automatically and the billing includes only the resources used when your code runs.
If your app is not used, Cloud Run scales down to zero.
If your app gets more traffic (imagine it makes the news), Cloud Run scales up to the number of instances needed.
You can control performance and cost by fine-tuning many settings: CPU, memory, concurrency, minimum instances, maximum instances, and more.
Every month, the free tier offers the first 50 vCPU-hours, 100 GiB-hours, and 2 million requests for no cost.

💾 Source code

The project includes just seven files and less than 200 lines of Python + JavaScript code.

You can reuse this demo as a base to build your own image processing app:

Check out the source code on GitHub.
For step-by-step instructions on deploying the app yourself in a few minutes, see “Deploying from scratch”.

🖖 More

Try the demo and generate your own coloring pages.
Learn more about Cloud Run.
For more cloud content, follow me on Twitter (@PicardParis) or LinkedIn (in/PicardParis), and feel free to get in touch with any feedback or questions.

🕵️ Detecting faces as a service 🐍

Laurent Picard — Tue, 29 Sep 2020 16:00:36 +0000

👋 Hello!

In this article, you'll see the following:

how to detect faces in pictures,
how to automatically anonymize, crop,… a picture with faces,
how to make this a serverless online demo,
in less than 300 lines of Python code.

Here is a famous face that has been automatically anonymized and cropped. Do you guess who this is?

Note: We're talking about face detection, not face recognition. Though technically possible, face recognition can have harmful applications. Responsible companies have established AI principles and avoid exposing such potentially harmful technologies (e.g. Google AI Principles).

🛠️ Tools

A few tools will do:

a machine learning model to analyze images,
a library to process images,
a web application framework,
a serverless solution to keep the demo available 24/7 and at minimal cost.

🧱 Architecture

Here is an architecture using 2 Google Cloud services (App Engine + Vision API):

The workflow is the following:

Open the demo: App Engine serves the home page.
Take a selfie: the frontend sends it to the /analyze-image endpoint.
The backend sends a request to the Vision API: the image is analyzed and the results (annotations) are returned.
The backend returns the annotations, in addition to the number of detected faces (to display the info directly in the web page).
The frontend sends image, annotations, and processing options to the /process-image endpoint.
The backend processes the image with the given options and returns the result image.
Change the options: steps 5 and 6 are repeated.
Get the image with new options.

This is one of many possible architectures. The advantages of this one are the following:

The web browser caches both the selfie and the annotations: no storage is involved and no private images are stored anywhere in the cloud.
The Vision API is only called once per image.

🐍 Python libraries

Google Cloud Vision

The client library that wraps calls to the Vision API.
https://pypi.org/project/google-cloud-vision

Pillow

A very popular imaging library, both extensive and easy to use.
https://pypi.org/project/Pillow

Flask

One of the most popular web app frameworks.
https://pypi.org/project/Flask

Dependencies

Define the dependencies in the requirements.txt file:

google-cloud-vision==2.4.4

Pillow==8.3.2

Flask==2.0.2

Notes:

As a best practice, also specify the dependency versions. This freezes your production environment in a known state and prevents newer versions from potentially breaking future deployments.

App Engine will automatically deploy these dependencies.

🧠 Image analysis

Vision API

The Vision API gives access to state-of-the-art machine learning models for image analysis. One of the multiple features is face detection. Here is a way to detect faces in an image:

from google.cloud import vision_v1 as vision

Annotations = vision.AnnotateImageResponse
MAX_DETECTED_FACES = 50

def detect_faces(image_bytes: bytes) -> Annotations:
    client = vision.ImageAnnotatorClient()
    api_image = vision.Image(content=image_bytes)
    return client.face_detection(api_image, max_results=MAX_DETECTED_FACES)

Backend endpoint

Exposing an API endpoint with Flask consists in wrapping a function with a route. Here is a possible POST endpoint:

import base64

import flask

app = flask.Flask(__name__)

@app.post("/analyze-image")
def analyze_image():
    image_file = flask.request.files.get("image")
    annotations = detect_faces(image_file.read())

    return flask.jsonify(
        faces_detected=len(annotations.face_annotations),
        annotations=encode_annotations(annotations),
    )

def encode_annotations(annotations: Annotations) -> str:
    binary_data = Annotations.serialize(annotations)
    base64_data = base64.urlsafe_b64encode(binary_data)
    base64_annotations = base64_data.decode("ascii")
    return base64_annotations

Frontend request

Here is a javascript function to call the API from the frontend:

async function analyzeImage(imageBlob) {
  const formData = new FormData();
  formData.append("image", imageBlob);
  const params = { method: "POST", body: formData };
  const response = await fetch("/analyze-image", params);

  return response.json();
}

🎨 Image processing

Face bounding box and landmarks

The Vision API provides the bounding box of the detected faces and the position of 30+ face landmarks (mouth, nose, eyes,…). Here is a way to visualize them with Pillow (PIL):

from PIL import Image, ImageDraw

PilImage = Image.Image
ANNOTATION_COLOR = "#00FF00"
ANNOTATION_LANDMARK_DIM_PERMIL = 8
ANNOTATION_LANDMARK_DIM_MIN = 4

def draw_face_landmarks(image: PilImage, annotations: Annotations):
    r_half = min(image.size) * ANNOTATION_LANDMARK_DIM_PERMIL // 1000
    r_half = max(r_half, ANNOTATION_LANDMARK_DIM_MIN) // 2
    border = max(r_half // 2, 1)

    draw = ImageDraw.Draw(image)
    for face in annotations.face_annotations:
        v = face.bounding_poly.vertices
        r = (v[0].x, v[0].y, v[2].x + 1, v[2].y + 1)
        draw.rectangle(r, outline=ANNOTATION_COLOR, width=border)

        for landmark in face.landmarks:
            x = int(landmark.position.x + 0.5)
            y = int(landmark.position.y + 0.5)
            r = (x - r_half, y - r_half, x + r_half + 1, y + r_half + 1)
            draw.rectangle(r, outline=ANNOTATION_COLOR, width=border)

Face anonymization

Here is way to anonymize the faces thanks to the bounding boxes:

ANONYMIZATION_PIXELS=13

def anonymize_faces(image: PilImage, annotations: Annotations):
    for face in annotations.face_annotations:
        v = face.bounding_poly.vertices
        face = image.crop((v[0].x, v[0].y, v[2].x + 1, v[2].y + 1))

        face1_w, face1_h = face.size
        pixel_dim = max(face1_w, face1_h) // ANONYMIZATION_PIXELS
        face2_w, face2_h = face1_w // pixel_dim, face1_h // pixel_dim
        face = face.resize((face2_w, face2_h), Image.NEAREST)
        face = face.resize((face1_w, face1_h), Image.NEAREST)

        image.paste(face, (v[0].x, v[0].y))

Face cropping

Similarly, to focus on the detected faces, you can crop everything around the faces:

CROP_MARGIN_PERCENT = 10

def crop_faces(image: PilImage, annotations: Annotations) -> PilImage:
    mask = Image.new("L", image.size, 0x00)
    draw = ImageDraw.Draw(mask)
    for face in annotations.face_annotations:
        draw.ellipse(face_crop_box(face), fill=0xFF)
    image.putalpha(mask)
    return image

def face_crop_box(face_annotation) -> tuple[int, int, int, int]:
    v = face_annotation.bounding_poly.vertices
    x1, y1, x2, y2 = v[0].x, v[0].y, v[2].x + 1, v[2].y + 1
    w, h = x2 - x1, y2 - y1
    hx, hy = x1 + w / 2, y1 + h / 2
    m = max(w, h) * (100 + CROP_MARGIN_PERCENT) / 100 / 2
    return int(hx - m + 0.5), int(hy - m + 0.5), int(hx + m + 1.5), int(hy + m + 1.5)

🍒 Cherry on Py 🐍

Now, the icing on the cake (or the "cherry on the pie" as we say in French):

Having independent rendering functions lets you combine multiple options at once.
Knowing the bounding box for all faces allows cropping the image to the minimal bounding box.
Using the location of the nose and the mouth, you can add a moustache to everyone.
If your functions have parameters to render a single frame, you can generate animations with a few lines of code.
Once your Flask app works locally, you can deploy and keep it available 24/7 at minimal cost.

Here is what's detected on famous photorealistic paintings:

American Gothic

Girl with a Pearl Earring

Shakespeare

Here are some animated versions:

American Gothic

Girl with a Pearl Earring

Shakespeare

And, of course, this works even better on real pictures:

Personal pictures (aged from 2 to 44)

Yes, I've had a moustache for over 42 years, and my sister too ;)

And, finally, here is our famous anonymous from the beginning:

Mona Lisa

🚀 Source code and deployment

The Python source for the backend takes less than 300 lines of code.
You can deploy this demo in 4 minutes.
See "Deploying from scratch".

🎉 Online demo

Try the demo by yourself:

➡️ https://face-detection.lolo.dev ⬅️

Here is a preview:

🖖 See you

Feedback or questions? I'd love to read from you! Follow me on Twitter for more…

⏳ Updates

2021-10-08: Updated with latest library versions + Python 3.7 → 3.9

🎞️ Video object tracking as a service 🐍

Laurent Picard — Thu, 25 Jun 2020 17:58:35 +0000

👋 Hello!

In this article, following up on the previous one ("Video summary as a service"), you'll see the following:

how to track objects present in a video,
with an automated processing pipeline,
in less than 300 lines of Python code.

Here is an example of an auto-generated object summary for the video :

🛠️ Tools

A few tools will do:

Storage space for videos and results
A serverless solution to run the code
A machine learning model to analyze videos
A library to extract frames from videos
A library to render the objects

🧱 Architecture

Here is a possible architecture using 3 Google Cloud services (Cloud Storage, Cloud Functions, and the Video Intelligence API):

The processing pipeline follows these steps:

You upload a video
The upload event automatically triggers the tracking function
The function sends a request to the Video Intelligence API
The Video Intelligence API analyzes the video and uploads the results (annotations)
The upload event triggers the rendering function
The function downloads both annotation and video files
The function renders and uploads the objects
You know which objects are present in your video!

🐍 Python libraries

Video Intelligence API

To analyze videos
https://pypi.org/project/google-cloud-videointelligence

Cloud Storage

To manage downloads and uploads
https://pypi.org/project/google-cloud-storage

OpenCV

To extract video frames
OpenCV offers a headless version (without GUI features, ideal for a service)
https://pypi.org/project/opencv-python-headless

Pillow

To render and annotate object images
Pillow is a very popular imaging library, both extensive and easy to use
https://pypi.org/project/Pillow

🧠 Video analysis

Video Intelligence API

The Video Intelligence API is a pre-trained machine learning model that can analyze videos. One of its multiple features is detecting and tracking objects. For the 1st Cloud Function, here is a possible core function calling annotate_video() with the OBJECT_TRACKING feature:

from google.cloud import storage, videointelligence

def launch_object_tracking(video_uri: str, annot_bucket: str):
    """ Detect and track video objects (asynchronously)

    Results will be stored in <annot_uri> with this naming convention:
    - video_uri: gs://video_bucket/path/to/video.ext
    - annot_uri: gs://annot_bucket/video_bucket/path/to/video.ext.json
    """
    print(f"Launching object tracking for <{video_uri}>...")
    features = [videointelligence.Feature.OBJECT_TRACKING]
    video_blob = storage.Blob.from_string(video_uri)
    video_bucket = video_blob.bucket.name
    path_to_video = video_blob.name
    annot_uri = f"gs://{annot_bucket}/{video_bucket}/{path_to_video}.json"
    request = dict(features=features, input_uri=video_uri, output_uri=annot_uri)

    video_client = videointelligence.VideoIntelligenceServiceClient()
    video_client.annotate_video(request)

Cloud Function entry point

import os

ANNOTATION_BUCKET = os.getenv("ANNOTATION_BUCKET", "")
assert ANNOTATION_BUCKET, "Undefined ANNOTATION_BUCKET environment variable"

def gcf_track_objects(data, context):
    """Cloud Function triggered by a new Cloud Storage object"""
    video_bucket = data["bucket"]
    path_to_video = data["name"]
    video_uri = f"gs://{video_bucket}/{path_to_video}"
    launch_object_tracking(video_uri, ANNOTATION_BUCKET)

Notes:

This function will be called when a video is uploaded to the bucket defined as a trigger.

Using an environment variable makes the code more portable and lets you deploy the exact same code with different trigger and output buckets.

🎨 Object rendering

Code structure

It's interesting to split the code into 2 main classes:

StorageHelper for managing local files and cloud storage objects
VideoProcessor for all graphical processings

Here is a possible core function for the 2nd Cloud Function:

class VideoProcessor:
    @staticmethod
    def render_objects(annot_uri: str, output_bucket: str):
        """Render objects from video annotations"""
        with StorageHelper(annot_uri, output_bucket) as storage:
            with VideoProcessor(storage) as video_proc:
                print(f"Objects to render: {video_proc.object_count}")
                video_proc.render_object_summary()

Cloud Function entry point

def gcf_render_objects(data, context):
    """Cloud Function triggered by a new Cloud Storage object"""
    annotation_bucket = data["bucket"]
    path_to_annotation = data["name"]
    annot_uri = f"gs://{annotation_bucket}/{path_to_annotation}"
    VideoProcessor.render_objects(annot_uri, OBJECT_BUCKET)

Note: This function will be called when an annotation file is uploaded to the bucket defined as a trigger.

Frame rendering

OpenCV and Pillow easily let you extract video frames and compose over them:

class VideoProcessor:
    def get_frame_with_overlay(
        self, obj: vi.ObjectTrackingAnnotation, frame: vi.ObjectTrackingFrame
    ) -> PilImage:
        def get_video_frame() -> PilImage:
            pos_ms = frame.time_offset.total_seconds() * 1000
            self.video.set(cv.CAP_PROP_POS_MSEC, pos_ms)
            _, cv_frame = self.video.read()
            image = Image.fromarray(cv.cvtColor(cv_frame, cv.COLOR_BGR2RGB))
            image.thumbnail(self.cell_size)  # Makes it smaller if needed
            return image

        def add_caption(image: PilImage) -> PilImage:
            ...

        def add_bounding_box(image: PilImage) -> PilImage:
            ...
            draw.rectangle(r, outline=BBOX_COLOR, width=BBOX_WIDTH_PX)
            return image

        return add_bounding_box(add_caption(get_video_frame()))

Note: It would probably be possible to only use OpenCV but I found it more productive developing with Pillow (code is more readable and intuitive).

🔎 Results

Here are the main objects found in the video :

Notes:

The machine learning model has correctly identified different wildlife species: those are "true positives". It has also incorrectly identified our planet as "packaged goods": this is a "false positive". Machine learning models keep learning by being trained with new samples so, with time, their precision keeps increasing (resulting in less false positives).

The current code filters out objects detected with a confidence below 70% or with less than 10 frames. Lower the thresholds to get more results.

🍒 Cherry on Py 🐍

Now, the icing on the cake (or the "cherry on the pie" as we say in French), you can enrich the architecture to add new possibilities:

Trigger the processing for videos from any bucket (including external public buckets)
Generate individual object animations (in parallel to object summaries)

Architecture (v2)

A - Video object tracking can also be triggered manually with an HTTP GET request
B - The same rendering code is deployed in 2 sibling functions, differentiated with an environment variable
C - Object summaries and animations are generated in parallel

Cloud Function HTTP entry point

def gcf_track_objects_http(request):
    """Cloud Function triggered by an HTTP GET request"""
    if request.method != "GET":
        return ("Please use a GET request", 403)
    if not request.args or "video_uri" not in request.args:
        return ('Please specify a "video_uri" parameter', 400)
    video_uri = request.args["video_uri"]
    launch_object_tracking(video_uri, ANNOTATION_BUCKET)
    return f"Launched object tracking for <{video_uri}>"

Note: This is the same code as gcf_track_objects() with the video URI parameter specified by the caller through a GET request.

🎉 Results

Here are some auto-generated trackings for the video :

The left elephant (a big object ;) is detected:
The right elephant is perfectly isolated too:
The veterinarian is correctly identified:
The animal he's feeding too:

Moving objects or static objects in moving shots are tracked too, as in :

A building in a moving shot:
Neighbor buildings are tracked too:
Persons in a moving shot:
A surfer crossing the shot:

Here are some others for the video :

A butterfly (easy?):
An insect, in larval stage, climbing a moving twig:
An ape in a tree far away (hard?):
A monkey jumping from the top of a tree (harder?):
Now, a trap! If we can be fooled, current machine learning state of the art can too:

🚀 Source code and deployment

The Python source code is less than 300 lines.
You can deploy this architecture in less than 8 minutes.
See "Deploying from scratch".

🖖 See you

Do you want more, do you have questions? I'd love to read your feedback. You can also follow me on Twitter.

⏳ Updates

2021-10-08: Updated with latest library versions + Python 3.7 → 3.9