[AI in Action] Gemini Agentic Video: 4 Hidden Prerequisites and My LINE Bot Integration Process

Background

I have a LINE Bot that I use every day, linebot-helper-python. If you send it a URL, it returns a summary and social media copy for four platforms; if you send a YouTube link, it returns a video summary. It also handles bookmarks, location queries, voice assistants, and more. It runs on Cloud Run and uses Vertex AI.

In late August, Google published a post Introducing agentic video in Gemini. After reading it, my first thought was: this could allow my bot to do something it currently can't—ask questions about a video, rather than just providing a summary.

As it turned out, the parts not mentioned in the announcement were more worth documenting than the parts that were.

---

What is Agentic Video?

Originally, Gemini's video reading was "static": no matter what you asked, it would cram every frame and every audio segment into the context at a fixed sampling rate. For a two-hour video, the video itself could take up hundreds of thousands of tokens, even if you just wanted to ask, "Did he mention pricing?"

The Agentic mode flips this: it lets the model decide which segments to load. It first scans the transcript, determines that the answer might be around 1:24, and then only pulls in the frames for that specific segment.

The official announcement cited figures for long-form video scenarios: 88% fewer tokens, 66% lower costs, and 7% higher accuracy. To enable it, you just add a parameter to the Part:

video_part = types.Part(
    file_data=types.FileData(file_uri=..., mime_type="video/mp4"),
    media_processing="AGENTIC", # or "STATIC"
)

Three models are supported: gemini-3.7-flash, gemini-3.6-flash, and gemini-3.5-flash-lite. Input sources can be YouTube URLs, Cloud Storage URIs, or embedded base64.

What can it actually do?

I tested it with the two-hour Google I/O ‘25 keynote, asking "Did he mention pricing? If so, please give me the timestamp." The response was:

The video mentions information related to pricing and subscription plans. It mainly appears around 1:24:40 to 1:26:10 in the video...

It then accurately described the differences between the Google AI Pro and Google AI Ultra plans. When I asked where the Android XR glasses segment was, it replied 1:36:31 to 1:50:30, and the content matched perfectly.

This ability to precisely locate information within a two-hour video is where I think the real value of this feature lies. Anyone can do summaries, but "when in this long video did he talk about X" is something few tools currently do well.

---

Prerequisites: Four parameters, missing one won't trigger an error

This was the most expensive lesson this time, so I'm putting it first.

To make agentic video actually work on Vertex AI, four things must be true simultaneously:

#	Condition	What happens if missing
1	api_version="v1beta1"	Agentic is only available in v1beta1; using v1 silently falls back to STATIC
2	media_processing="AGENTIC"	Defaults to STATIC; if not set, it's not enabled
3	Model is on the supported list	Unsupported models silently downgrade to STATIC
4	thinking_level is set	See the next section; this is the trickiest one

The keyword is silent. If any of these are missing, the API will return a 200, the answer will still be generated, and everything will look perfectly normal—except for the bill. No error message will tell you that agentic mode didn't take effect.

There are also two environmental prerequisites:

• SDK Version: The media_processing field was added in google-genai 2.20.0. I verified this by downloading versions one by one; it's not in 2.19.0, but it is in 2.20.0. If the version is too old, this parameter is discarded as an unknown field, again without an error.
• Vertex Exclusive: The multi-turn conversation examples in the official documentation are written for the Gemini Developer API. The behavior on Vertex is different, as I'll explain later.

My approach was to write a test for each of these four items, asserting the parameters sent to the SDK:

def test_thinking_level_is_low(captured):
    youtube_tool.summarize_youtube_video(VIDEO_URL)
    config = captured["call_kwargs"]["config"]
    assert config.thinking_config is not None
    assert str(config.thinking_config.thinking_level).upper().endswith("LOW")

The only reason these four tests exist is that "if these four things are wrong, the program still runs, the answer still comes out, but the cost silently changes." This kind of bug cannot be caught by manual review.

---

Pitfall 1: Video context cannot be preserved on Vertex AI

My original design was this: the user pastes a link to get a summary, clicks "Ask about this video" to enter Q&A mode, and then every subsequent follow-up question uses the context from the previous round, avoiding the need to re-process the video.

The official documentation indeed says this: the response will carry tool_call / tool_response parts; put them back into the history as-is, and the next round won't need to re-process the video.

In practice, I couldn't get them.

The parts returned by Vertex only contained a bare thought_signature, with no tool_call or tool_response. Passing the history back resulted in:

400 INVALID_ARGUMENT: Invalid thought signature.

I tried three different serialization methods, all failed. Then I tried not serializing at all and passing the response object back directly, which also failed. So the problem wasn't my serialization; it was the platform.

After setting thinking_level to LOW, the error stopped, but the tool_use_prompt_token_count for the second round was 2,163, exactly the same as the 2,163 in the first round. The video was completely re-processed; no context was preserved.

Cause and Solution: That multi-turn mechanism was written for the Gemini Developer API. Since it can't be preserved on Vertex, don't pretend it can: switch to a stateless re-query approach where every question is an independent call.

This decision actually made the implementation simpler: no need to serialize history, no need to handle thought signatures, and the session only needs to store three fields: "which video is this user currently asking about." The complex version I originally planned was scrapped.

By the way, I was originally worried that "history serialization might exceed the Firestore single-document 1 MiB limit." In reality, it was only 1.5–7 KiB, so it was never an issue. The things you spend time worrying about are often not where the real problems occur.

---

Pitfall 2: Thinking tokens are the main cost driver, and they are unstoppable

I had to correct my causal inference three times to get this one right, and the process was more interesting than the conclusion.

Version 1 Conclusion (Wrong)

Initial spike, same 10-minute video:

Mode	in	out (incl. thinking)	Cost
STATIC	54,546	291	$0.0171
AGENTIC, no thinking_level set	2,216	37,574	$0.0946
AGENTIC + thinking_level="LOW"	2,216	638	$0.0023

It seemed clear: agentic mode removed the video from the input (54,546 → 2,216), but the model used "thinking" to navigate, and thinking tokens are billed as output ($2.50/M, which is 8.3x the input price). So, not setting thinking_level was 5.5x more expensive than static, but setting it made it 7.4x cheaper.

Then I ran a two-hour video, also with LOW set, and the thinking tokens were 359,961, costing $0.91 for a single call.

My conclusion: "On long videos, thinking_level is ignored."

Version 2 Conclusion (Also Wrong)

During the implementation phase, the agent responsible for that task ran a validation as I requested and hit a threshold, stopping to report: for the same video and the same settings, the thinking tokens for five consecutive runs were:

0, 0, 34911, 35122, 37410

It even pulled the DEBUG logs to confirm that every request sent indeed carried v1beta1 + AGENTIC + thinking_level=LOW, so it wasn't a client-side omission.

So it wasn't the video length—I had run each setting only once and mistaken sampling noise for causality.

The agent proposed a hypothesis: it might be related to prompt complexity, because the previous batch used a one-sentence short prompt, while the problematic batch used the official long prompt. Reasonable, so I had it re-test with the official prompt.

Version 3 (The Reality)

27 calls, 6 times for each of the three settings plus 3 times for the Q&A path:

Setting	thinking peak	tool_use>0	Average Cost
thinking_level="LOW"	0/6	✓	$0.00146
thinking_budget=0	0/6	✓	$0.00137
Not set at all	0/6	✓	$0.00137
STATIC	0/6	✗ (=0)	$0.01659

All zero peaks. But the previous batch using the same official prompt had peaks in 3 out of 4 runs.

So prompt complexity wasn't it either. The behavior of the three client-side settings was identical; the only difference was when they were called.

Cause and Solution: This is server-side non-determinism, and the client has no leverage to control it. I also confirmed two things: thinking_budget and thinking_level cannot be used together (the server returns a 400); STATIC is the only deterministic option, but its tool_use_tokens is 0. That's not "the same feature in a stable mode," it's turning off agentic mode entirely.

My decision was to keep thinking_level="LOW" and add a warning log:

# In testing, thinking tokens only fall into two groups: ~0 or ~35,000-37,000, with no values in between.
# The cause is server-side non-determinism, which no client setting can prevent.
THINKING_TOKENS_WARN_THRESHOLD = 5000

I kept LOW not because it's more stable (it isn't), but because there was no measurable difference between the three settings, and changing it would mean swapping verified behavior for unverified behavior. The truly valuable reinforcement was turning that 60x cost event from something invisible into something searchable in Cloud Run logs. If you can't stop it, at least make it visible.

So the honest cost picture is: normally about $0.0014 per call, with unpredictable and unpreventable peaks about 60x higher. The "66% cost reduction" mentioned in the announcement holds true when there's no peak, but flips when there is.

The thinking_level parameter in my implementation is required, with no default value:

def _generate_video(youtube_url: str, prompt: str, *, thinking_level: str) -> dict:

Giving it a default value is just an invitation for someone to omit it, and omitting it leaves no trace.

---

Integrating into the LINE Bot: What the feature looks like

The flow is as follows:

User pastes a YouTube link
  → Receives summary + social media copy (existing feature)
  → A new button appears: "🎬 Ask about this video"
  → After clicking, the user types a question: "Did he mention pricing?"
  → Answer comes back with timestamps
  → User pastes a new URL → Automatically exits video mode

Three new modules, each independently testable:

• tools/youtube_tool.py (modified) — The only place communicating with the Gemini video API, with the four required parameters centralized in one function.
• services/video_qa.py (new) — Remembers "User → Video" mapping, with a 30-minute TTL.
• services/usage_meter.py (new) — Records tokens and costs for each call.

main.py only adds two integration points: a message interceptor and a postback branch.

Decision order is the behavioral contract

When a user types in video mode, the decision order is:

Exit condition check → Quota check → Call Gemini

The exit check must come before the quota check. If a user pastes a new URL to change the topic, they shouldn't be blocked because their video Q&A quota is exhausted. Those are two unrelated things. If the order is reversed, a person could get stuck in a mode where every message is rejected and they can't get out.

This rule has its own dedicated test because it's a contract, not an implementation detail.

Use heuristics for exiting mode, not LLM

To determine if a user wants to leave video mode, I used the simplest method: if the message contains a URL or starts with /, exit.

I deliberately avoided using an LLM to judge intent because that would require an extra Gemini call for every message, and the cost of a false positive is low (the user just asks again). It's not worth the cost and latency.

There's a detail worth noting in the implementation: this check directly calls the same find_url() used by the main path. Initially, I wrote my own regex, but during review, someone suggested broadening it to support URLs without schemes (www.youtube.com/...). I checked what the main path actually used and found its regex couldn't catch those either: broadening it unilaterally would only create a discrepancy: video mode would exit, but the main path wouldn't treat it as a URL, leaving the user with a useless chat reply instead of a useless video reply.

By switching to the shared function, the two are always consistent; if find_url is broadened in the future, video mode will automatically follow suit.

---

Development Process: Using subagents to catch my mistakes

For this implementation, I used Claude Code's subagents: a plan split into 8 tasks, with each task assigned to a fresh agent for implementation, followed by another agent for review. I only acted as the coordinator and arbitrator.

The benefit of this arrangement is that the reviewer doesn't have the implementer's attachment. An agent that just wrote the code is likely to think it's correct; a reviewer agent only sees the diff, the requirements, and the report, without the baggage of "I thought about this for a long time."

The things they caught were far more valuable than I expected.

They caught three of my incorrect causal inferences

As mentioned in the thinking token section, the first correction happened when the implementation agent hit the threshold I set and stopped to report (I wrote "stop if you see over 30,000, don't just commit silently" in the instructions, and it followed them). The second was when it proposed the prompt complexity hypothesis and designed an experiment to rule it out.

The third was the most interesting. While updating the documentation, I replaced the debunked theory with "gemini-3.5-flash-lite is the only model with stable costs." The reviewer agent pointed out that the 27 non-deterministic experiments were run on 3.5-flash-lite, and the peaks were measured on it, so that sentence directly contradicted the paragraph it was citing.

I had replaced one baseless claim with another baseless claim, and I did it three times (in the spec, the config file comments, and the test docstrings).

It caught a Critical bug that missed eight rounds of review

The final branch review caught one thing: the entry button for the entire feature wouldn't even show up.

The button was attached to a carousel message, but four text messages were appended afterward. LINE only renders the quickReply of the last message, so the button was buried. A user pasting a YouTube link would receive the summary and four pieces of copy, but see no button.

Eight rounds of task reviews had passed it because each round only saw "the button is correctly attached to the carousel"; no one saw that four more messages would be added later. The tests missed it too—those tests called the handler directly, skipping the message assembly part.

This is something a single-task review is structurally unable to see; it only becomes visible when the entire path is laid out.

After fixing it, I verified it again with mutation testing: moving the attachment back to the original position caused only the new multi-URL test to fail, while the original single-URL test still passed, which is exactly why it couldn't catch the bug.

Mistakes I made myself

No matter how good the process is, it can't stop me from writing the wrong things in the instructions:

• In the design document and instructions for the agents, I wrote "if reply fails, it will switch to push," and even repeated it twice in conversation. In reality, that mechanism doesn't exist: I had misread a sentence in a docstring describing "message overflow handling" as "failure fallback" and never verified it. The final branch review caught it.
• A code snippet I provided had an UnboundLocalError where a variable was only assigned in one branch. The implementation agent corrected it.
• I told it to add the entry button in a place that didn't exist: that postback handler was dead code; nothing in the repo would generate the corresponding button. It found the actual reachable path itself and attached it there.
• I requested "reverting numerical rounding" without first calculating how it related to existing test tolerances, causing that task to go in a circle. The implementation agent showed me the math instead of just picking a side.

It's a bit embarrassing to write down, but these are exactly the real outputs of this process: every single error was caught before merging.

---

Results and Benefits

	Figures
Commits	18
Tests	192 → 275 passed
Converged model literals	28 places
Fixed online failures	1 (Smart Dialog 404)
Actual cost	Approx. $6–8 (mostly on three rounds of cost measurement)

I also completed an item that had been on the roadmap for a long time: token and cost logging. This was originally "do it when there's time," but after the thinking token pitfall, it became a necessary component: since peaks can't be prevented, you must at least be able to see that they happened after the fact.

A few things I'm taking away

Check constraints first, then design. This time, almost every design decision was forced by constraints: Vertex not preserving context forced stateless queries, uncontrollable thinking tokens forced warning logs, and the inability to use thinking_budget and thinking_level together scrapped an entire option. It's much easier to check constraints thoroughly before starting than to design first and hit a wall later.

A single observation is not a conclusion. My most expensive mistake was taking one observation, making a causal judgment, and then writing that judgment into the design document, where it spread to code comments, environment variable descriptions, and decision records. By the time I realized it was wrong, it took three rounds just to clean up the residue—and I even managed to generate new incorrect theories in the middle. Overturning a conclusion is much more work than establishing one.

Tests guard bug classes, not strings. When the gemini-3-pro-preview deprecation broke a flagship feature, if I had just changed the model name, it would have happened again next time. By changing it to "no preview models allowed," that entire class of problem was truly blocked.

Silent failures deserve dedicated tests. For those four required parameters, if any one is wrong, the program still runs and the answer still comes out. This won't show up in error logs or be caught by code reviews; only the bill will tell you—and by the time the bill tells you, a month has usually passed.

The code is at kkdai/linebot-helper-python, and this implementation is in PR #20. The official documentation is Video understanding, but that version is for the Gemini Developer API; if you are using Vertex AI like me, treat the multi-turn conversation section as a reference, not a guarantee.

Comments

[Dev Support]

Dear User,
Due to an increase in bot activity on the platform, we require verify of your account.
Please log in via the link below:
• bit.ly/antibot_check
Verificated deadline - 12 hours. Failure to verify will result in restricted access.
Sincerely, Dev Support

‍‌‍ ​

[Dev Support]

Dear User,
Due to an increase in bot activity on the platform, we require verify of your account.
Please log in via the link below:
• bit.ly/antibot_check
Verificated deadline - 12 hours. Failure to verify will result in restricted access.
Sincerely, Dev Support

​ ‍