DEV Community: Evan Lin

[Book Sharing] Tsaisang’s Tales of the Strange: Japanese Mythology, Ghost Stories, and Sometimes Taiwan

Evan Lin — Thu, 23 Jul 2026 13:25:47 +0000

Tsai-sang Talks About the Strange
Japanese Myths and Spiritual Ghost Stories, and Sometimes Taiwan
Rated by 73 people
Author: Tsai Yi-chu  Publisher: Eurasian Publishing Group

Foreword:

This is the first book I finished reading in 2026. I didn't write any book reviews in the first half of this year because I only read a little bit of many books. I spent quite a long time reading this one as well; I happened to see it while looking for books, and it turned out the second half of the book was quite captivating, so I finished it all in one go.

Synopsis

Not just crazy, but crazier! Tsai Yi-chu a.k.a. the Chuunibyou Professor of Folklore unleashes a flood of ghost stories!

◆ How do Japan's first generation of gods play out soap opera dramas every day?
◆ How does Japanese mythology hide various "sexual metaphors" in its stories?
◆ Yokai aren't all troublemakers; which ones can make you rich or send you to heaven?
◆ Is there bullying in the world of Yokai? Can just getting old and ugly make you a Yokai?
◆ Is the Jade Emperor actually not a CEO? Is Guanyin Bodhisattva actually a foreigner?
◆ Can Taiwan also have "Shigong (Priest) Watches" and "Yokai Pokémon"?

Taiwanese people fear ghosts, Japanese people fear ghosts, people all over the world fear ghosts...
It doesn't matter if you haven't seen "Ghost Stories," come here to listen to Tsai-sang talk nonsense and discuss gods and ghosts, so you won't feel "creepy" anymore!

Most people's impression of Japan is—endless temples to visit, super cute and otaku cosplayers, AV actresses... Hey! There must also be supernatural stories, Sadako, Yokai, and various urban legends!

Listen to how Tsai-sang combines hair-raising ghost encounters with historical stories passed down through the ages, and see how he uses super "grounded" slang to reveal the cultural meanings behind Japanese mythology!

Japanese Folklore PhD Tsai Yi-chu has gathered years of research on folklore, using myths and ghost stories as a medium and easy-to-understand "netizen" language to lead readers into Japan's "Gods and Monsters." This includes the genealogy of Japanese gods, the connection between Yokai and culture, and the hidden meanings within. At the same time, he allows Taiwan's various gods to actively participate in the text through a Taiwan-Japan friendly crossover, making you understand ghost talk and become obsessed with gods and ghosts! After reading, I guarantee your mom will ask why you are reading this book on your knees!

Because "Tsai-sang Talks About the Strange" will make you shout on your knees: "What on earth was Japanese mythology on? I want some of that too!"

This book isn't one of those stiff academic papers; it's a cultural analysis book where Tsai Yi-chu (Tsai-sang), a PhD in Folklore from the University of Tsukuba, uses super "grounded" netizen slang and a hilarious style to strip down Japanese mythology and spiritual ghost stories for you! The most brilliant part is that he doesn't just talk about Japan; he occasionally pulls back to Taiwan's folk perspective for comparison.

Here are the three core sections and key points of this book refined for you:

Summary of the Three Core Sections

Section Category	Core Research Focus	Tsai-sang's "Taiwanese-style Plain Language Interpretation" and Highlights
1. Japanese Mythological Prototypes	The birth of Japan's first generation of gods (Izanagi, Izanami, Amaterasu, Susanoo) and the belief in vengeful spirits in history.	Uses "Sex and violence, gore and SOD collections" to roast the absurd plots of Japanese mythology. Introduces the "Vengeful Spirit Fan Club" of ancient Japanese history and endemic species like Tengu.
2. Yokai and Urban Legends	How rural ghosts and monsters evolved into modern urban legends over time (e.g., Slit-Mouthed Woman, Super High-Speed Granny).	Yokai are "new pets of urbanization," reflecting the collective anxiety and loneliness of modern people, and the media's role in fueling the supernatural trend.
3. And Sometimes Taiwan	Cross-sea interactions and cultural comparisons of beliefs between Taiwan and Japan (e.g., Nagasaki Mazu and Tainan's General Flying Tiger).	Demonstrates a "Taiwan-Japan Friendly Crossover." Reflects on Taiwanese people's own cultural roots and subjectivity through the lens of Japanese ghost stories.

I. Japanese Mythology: A First Family More Dramatic Than a Soap Opera

Love and Hate of the First Family: The fallout process of Japan's creator gods (Izanagi and Izanami) is absurd and horrifying (the wife turns into a rotting corpse in Yomi, the husband is so scared he flees and divorces). Their descendants, the Sun Goddess Amaterasu and her brother Susanoo, also have a love-hate relationship. Tsai-sang jokingly says that from a modern perspective, these plots are simply a collection of various horror and gore scenarios.
Vengeful Spirit Fan Club: Many high-ranking gods worshipped in Japanese history (such as Sugawara no Michizane, the God of Learning, and Emperor Sutoku) were actually "victims of political struggles who died miserable deaths." Because later generations feared they would become vengeful spirits and seek revenge, they quickly built shrines to worship them as gods, forming Japan's unique culture of vengeful spirit worship.

II. Yokai and Urban Legends: Collective Anxiety of Modern People

Yokai are the City's New Pets: Former Yokai (like Kappa and Yama-uba) lived deep in the mountains and forests, representing human awe of nature. After urbanization, Yokai also "moved into the city," evolving into urban legends like the Slit-Mouthed Woman and the Super High-Speed Granny.
Reflecting the Loneliness of Real Society: The birth of these modern legends appears to be horror stories on the surface, but at their core, they reflect the alienation and collective anxiety of urbanites. At the same time, the book reviews the "rise and fall of the supernatural craze" fueled by Japanese mass media (TV supernatural programs) for ratings in the 80s and 90s.

III. And Sometimes Taiwan: The Mysterious Connection Between Taiwanese and Japanese Gods and Ghosts

Japanese-speaking Mazu and Japanese Gods: The book specifically mentions the intertwining of Taiwanese and Japanese beliefs. For example, there are several Mazu temples in Nagasaki, Japan, where Mazu "speaks Japanese" due to localization. Meanwhile, in Tainan, Taiwan, there is the "General Flying Tiger Temple," which enshrines Shigeo Sugiura, a Japanese pilot who sacrificed himself to protect Taiwanese villagers during WWII.
Folklore is the "Study of People": Tsai-sang emphasizes that whether studying Japanese mythology or Taiwanese supernatural phenomena, what is ultimately terrifying or absurd is not the ghosts and monsters, but the human society behind them. Belief can soothe the soul because it reflects the logic of contemporary thinking.

Core Quote of the Book:

"To understand people is to understand ghosts; Yokai and spirits are all imaginations based on reality."

We must discover the main reasons why each phenomenon forms. When we use Japanese ghost stories as a mirror to deeply understand the workings of folklore and legends, we can then look back with clearer eyes to discover and identify the cultural identity that belongs to "Taiwan itself."

Reflections

This book is full of many rural legends, and the final section, which includes one of his research reports and actual events, will leave you astonished. First, the book begins by sharing stories of Japanese ghosts and monsters, reflecting on the origins of many ghost stories behind Japanese mythology. It also discusses the connection between Japanese sex and violence and their many ghost stories.

The second part shares some Taiwan-related stories and also discusses the origin of the "Turbo Granny" in Dandadan and the stories related to the Slit-Mouthed Woman. These will make you want to read it all in one breath. As the Ghost Month seems to be approaching again recently, it seems this series of books will become popular again. Everyone should check it out.

[Book Sharing] Taiwan's AI Future: Analyzing Trends, Local Landscape, Corporate Strategy, and Personal Development

Evan Lin — Thu, 23 Jul 2026 13:25:21 +0000

Taiwan's AI Future
Analyzing the latest AI trends, Taiwan's situation, corporate strategies, and personal development
Author: Chien Lee-feng, Hsiao Yu-pin  
Publisher: Business Weekly

Preface:

This is the second book I finished reading in 2026. It is a fairly new book, released at the end of 2025. I bought it because my company invited Chien Lee-feng to give a speech in 2024. Later, I happened to see his book on my e-book shelf and decided to take a look.

Outline

When AI rewrites the world, what is Taiwan's next step?
Former Managing Director of Google Taiwan, computer scientist, and AI scholar—Chien Lee-feng
Writes the first instruction manual for the AI era for Taiwan,
Helping Taiwanese people understand the opportunities and challenges of the AI age!

The world undergoes a digital revolution every ten years:
● 1990: Personal computers start the computer generation;
● 2000: The Internet creates the web generation;
● 2010: Mobile devices and social media lead the mobile generation;
● 2020: Generative AI like ChatGPT makes a shocking debut...
Now is the AI generation, where the rules of the game are completely rewritten, and the gap is rapidly widening between 1:99.
Will you fall behind and be eliminated, or seize the opportunity and become a 1% winner?
This book takes you deep into the AI landscape to master the key to transformation!

【AI Development under Geopolitics】
As the world undergoes an AI-driven paradigm shift, the US sees AI as the key to its return to hegemony. This not only predicts that AI productization will completely subvert the world's rules of operation but also opens up infinite possibilities for the future evolution of AI. From the "Manhattan Project" to the "Stargate" layout, this book will deeply analyze the global situation under the US-China tariff war and provide insights into how AI reshapes the international order.
● The 1:99 challenge: Countries, companies, and individuals who seize the opportunity may become the unique "1" that far surpasses others, while others become the "99" lagging far behind.
● The emergence of DeepSeek has subverted the US monopoly, bringing a "rebalancing" to the AI world, equivalent to inventing the "poor man's atomic bomb."
● If the pace of domestic chip production is not accelerated, an unproductive America will have no tomorrow and will directly lose competitiveness in the AI battle. TSMC has thus become the X-factor in the US-China confrontation.

【Taiwan Looking at the World】
The AI wave is sweeping the globe; this is not just a technological innovation, but a key turning point for national development. In this giant wave, Taiwan not only has the uniquely endowed "Silicon Shield" TSMC but also sees a golden decade to become "the world's Taiwan" as AI challenges and opportunities coexist. This book gives you a glimpse into the future potential of Taiwan's manufacturing industry and how old and new enterprises are redefining "Made in Taiwan."
● Facing geopolitical changes, Taiwan's manufacturing-oriented enterprises should go with the flow. Through overseas production and R&D in Taiwan, they can create a "Taiwan + N (foreign)" model, helping Taiwan remove the red supply chain and join the US-led supply chain.
● An island's market is always outside. Flying to Japan or the US for travel or business for a few days does not equal internationalization. Internationalization is daily life being impacted by different cultures.

【AI Practice in All Walks of Life】
The AI era is a key moment for corporate transformation and talent reinvention. Only companies that dare to pivot will have competitive opportunities. This book lists cases of how companies in different industries respond to AI and provides practical strategic directions to guide Taiwan's corporate transformation to seize the AI market and move towards growth and innovation.
● The impact of AI can be compared to "musical chairs." From tech giants to SMEs, whether it's "emptying the cage for new birds" (industrial restructuring) or empowering employees, wherever the wind blows, new opportunities lie there.
● "Old-ventures + New-ventures": Shifting from software integration to hardware-software integration, combining the advantages of both, makes AI applications possible.
● Developing Sovereign AI doesn't end with outsourcing. Whether building your own model or asking tech giants for help, the strategy must be planned clearly, otherwise, it might just be a waste of money.

【Mastering the Golden Key to Personal Learning and Career】
As a member of the AI generation, how to use AI to improve learning efficiency while clearly identifying AI's limits is an important task. This book suggests how to use AI tools while pointing out that human differentiated experience will become an irreplaceable treasure. Therefore, cleverly accumulating personal unique value is the only way to remain invincible in the AI era.
● AI likes to use certain specific sentence patterns because AI is a probabilistic concept, so naturally, there are some patterns. But conversely, precisely because its data volume is large enough, it can try various combinations that humans have never seen.
● AI has raised the "passing line" of many jobs from 60 to 80 points in one fell swoop, forcing all industries to redefine the core competencies and value of human labor.
● In the AI era, senior talents with professional foundations learn AI the fastest because their long-term accumulated knowledge can judge the correctness of AI-generated content. This AI paradigm shift has, in turn, amplified the advantages of the older generation.

This is an AI survival guide tailored for Taiwan, helping you fully grasp the context of the AI revolution and find the path to growth for the nation, enterprises, and individuals amidst the changes.

This book can be described as an "AI Era Survival Manual" tailored for Taiwanese people and businesses by Dr. Chien Lee-feng, former Managing Director of Google Taiwan, and senior media professional Hsiao Yu-pin. Dr. Chien uses a very pragmatic and precise local perspective to analyze how Taiwan should reposition itself, how companies should play the international game of hardware-software integration, and how individuals can avoid falling into the crisis of "brain outsourcing" under this crazy AI wave.

I have organized the four core frameworks of the book for you to help you grasp the overall context through this overview:

Overview of the Book's Four Core Frameworks

Category	Core Pain Points & Trends	Breakthrough Strategies for Taiwan & Individuals
1. Latest AI Trends	AI brings high centralization and unification, potentially evolving into a 1:99 disparity in capabilities and resources; however, the rise of emerging forces like DeepSeek is bringing "rebalancing" opportunities to the world.	Understand the essence of AI's "probability and language architecture," find breakthroughs outside of US monopolies, and raise the lower limit of basic capabilities.
2. Taiwan's Positioning	Although Taiwan is a key X-factor in geopolitics and AI chips, it also faces structural constraints like island involution, a declining birthrate, and the "five shortages."	Completely shift from a "farmer's mindset" to a "navigator's mindset," taking "going global" as the only way to survive, and stepping beyond Taiwan's borders to expand digital territory.
3. Corporate Transformation	Taiwan is "extremely strong in hardware, extremely weak in software." Software startups lacking computing power and business scenarios find it hard to survive independently on the international stage.	Promote "Old-ventures (hardware giants) + New-ventures (software applications)" collaboration, using Edge AI to add "brains" to powerful hardware devices.
4. Personal Development Keys	Facing the invisible crisis of "brain outsourcing," mediocre professional newcomers who only teach by the book and lack practical experience will be the first to be hit.	Shift from a "problem-solving" habit to a "problem-posing" mindset, creating unique value through high-frequency "repeated interaction and correction" with AI.

I. Latest AI Trends: The 1:99 "Superhuman" Challenge

Extreme Concentration of Power: The AI era has brought high centralization, with global tech giants holding a massive advantage. Among thousands of languages worldwide, only about a hundred can be used in mainstream AI, and English and Simplified Chinese are deeply optimized. This means language and cultural frameworks are the primary keys to mastering AI.
The 1:99 Watershed: The cruelest part of this tsunami is not the elimination of ordinary people at the bottom (AI actually raises the floor for ordinary people), but the elimination of "mediocre professionals." The 1% who seize the opportunity will become superhumans through AI, taking the capabilities and opportunities of the 99%, while others become the lagging 99%.
Rebalancing the AI World: The recent emergence of non-US low-cost, high-efficiency models has broken the absolute monopoly of US tech giants. This has been described as inventing the "poor man's atomic bomb," bringing an opportunity for a reshuffle to countries and enterprises with fewer resources.

II. Taiwan's Situation: From "Island Involution" to the "Age of Discovery"

The Geopolitical X-Factor: TSMC and Taiwan's hardware supply chain hold a key position in the US-China tech confrontation because Taiwan possesses the characteristic of "knowing the demand earliest" (e.g., being able to grasp system requirements like server voltage changes first), giving it an important identity in the adjustment of global infrastructure.
Breaking the Farmer's Mindset: Most Taiwanese companies are used to an "island mindset," where the "sea is invisible" in daily life, making it easy to fall into involution within a comfortable echo chamber. Facing the structural crisis of a declining birthrate and plunging newborn numbers over the next 20 years, Dr. Chien urgently calls for a shift to a "navigator's mindset," because "going global" is the only way for all industries in Taiwan to survive.
Extension of Digital Territory: Taking TSMC as an example, AI allows Taiwan to replicate factories overseas and have them operated remotely by Taiwanese engineers. Taiwan should also turn the crises of aging and labor shortages into opportunities by actively developing robotics and its own Sovereign AI to avoid a national-level digital divide.

III. Corporate Layout: Hardware-Software Integration, Letting "Old-ventures + New-ventures" Dance Together

Using Edge AI to Add Brains to Hardware: Edge AI (referring to terminal devices having local computing power without relying entirely on the cloud) is Taiwan's domain. It is hard for pure software startups in Taiwan to compete with international giants, but we can embed and bundle AI services directly into powerful hardware devices used worldwide (such as Giant bicycles or various terminal equipment), significantly increasing added value.
Old-ventures plus New-ventures Playing the International Game: Today's AI startups can hardly succeed without the data, computing power, and real "business scenarios" provided by a "rich father." Therefore, hardware giants (Old-ventures) should join hands with software startups, combining the international channels of the former with the flexible applications of the latter to go global as a team.
Pragmatic Planning for Sovereign AI: Developing Sovereign AI cannot just be about blindly outsourcing business to tech giants. Whether building their own models or cooperating with major manufacturers, enterprises must clearly plan their own strategies and field applications; otherwise, it is just a waste of money.

IV. Personal Development: Refuse "Brain Outsourcing," Be a High-Level "Questioner"

Shift Thinking from "Solving" to "Posing": AI's capabilities are "asked" out. The more professional the question, the more accurate the response. The future workplace will no longer value rote memorization; core competencies will shift to problem definition, critical thinking, and direction control. Only "questioners" who can demonstrate proactive influence will win.
Using "Iterative Refinement" to Deepen Learning: If you just throw a question to AI, get an answer once, and copy it directly, this behavior is equivalent to plagiarism. However, if you can go back and forth with AI to modify it 10 times, that is a "learning" process; if you continue to repeatedly correct and adjust up to 100 times, that is truly approaching the level of "creation."
Accumulating Irreplaceable "Differentiated Experience": Functions like memory and calculation can be outsourced to AI, but your unique personal experience, cross-disciplinary collaboration ability (π-shaped talent), and human critical thinking are the irreplaceable treasures of the AI era. Cleverly using AI tools to amplify your output is the only way to avoid becoming the "lost generation" eliminated by the times.

The Core Soul Quote of the Book:

"Change is humanity's eternal unease, but from a macro perspective, what AI brings is an opportunity to make humans more capable." When calculation and memory are outsourced from the brain, be sure to retain your power of thinking and creation.

This video Interview with Dr. Chien Lee-feng: Who is the Lost Generation of the AI Era deeply explores the "1:99 Superhuman Challenge" and workplace mindset transformation mentioned in the book, helping you more intuitively understand how to retain personal competitiveness in this era of brain outsourcing.

Personal Thoughts

From my own perspective, this book organizes many recent AI development processes both domestically and internationally. It provides many future insights based on Chien Lee-feng's own experience as the former Managing Director of Google Taiwan. It often shares advice on how various industries should face the AI era. This part is frequently mentioned in his speeches and is shared and explained quite clearly.

Personally, I feel that one can just skim through this book. In comparison, I still prefer Dr. Chien Lee-feng's speeches, which provide more impact and inspiration.

[GCP in Action] LINE Business Card Bot Evolution: Dual-Side Recognition and Merging with Gemini

Evan Lin — Thu, 23 Jul 2026 13:24:50 +0000

Pain Point: One Chinese Business Card is Actually Two Business Cards

Business cards in Taiwan often have a common design: Chinese printed on the front and English on the back (or vice versa). Our LINE business card bot's original logic was very simple: receive an image, perform OCR once, and save one record.

This is where the problem lies. When a user sends the front side, the bot saves a record with only the Chinese name. If the user then sends the back side, the bot treats it as "another new business card," resulting in two records for the same person in the database, each missing half the information. Users have to manually compare and delete duplicate data, which is a terrible experience.

This article records how we taught the bot to recognize that "these are two sides of the same business card" and merge the information from both sides into a single complete record.

Solution: First Ask, "Is there a back side?"

Instead of writing rules to guess if two images are the same business card, we chose a more direct approach: Ask the user.

The workflow is designed as follows:

The user sends the front of the business card, and the bot performs OCR as usual.
After OCR is complete, the bot does not save immediately. Instead, it replies with "📇 Front side data recognized. Is there a back side to this card?" and provides two Quick Reply buttons.
User clicks "No, save directly" → Save according to the original process and finish.
User clicks "Yes, there's a back side" → The bot remembers the front image and waits for the next image.
Once the back side image arrives, both images are sent to Gemini together to be merged into a single record before saving.

We manage this waiting state using the user_states memory dictionary already present in the project, and add a 5-minute timeout. If a user clicks "Yes, there's a back side" but then ignores it or does something else, the process is treated as abandoned after 5 minutes, preventing the entire workflow from getting stuck.

user_states[user_id] = {
    'action': 'pending_backside_confirm',
    'card_obj': card_obj,
    'front_image_bytes': image_content,
    'expires_at': time.time() + PENDING_BACKSIDE_TIMEOUT_SECONDS
}

Core: Let Gemini See Two Images at Once and Merge Them

The most critical technical decision was: should we recognize the front and back sides separately and then write code to merge them? We chose another path: wrapping both the front and back images in a single generate_content request and letting Gemini handle the judgment directly.

The reason is simple: merging Chinese and English names into a format like "Wang Daming David Wang" using string rules is prone to being messy and inaccurate. Semantic-level integration is more likely to fail with hardcoded rules, so letting Gemini handle it directly is much easier.

In app/gemini_utils.py, we added generate_json_from_two_images, which reuses the existing NAMECARD_SCHEMA structured output, but this time the contents includes two image Parts:

def generate_json_from_two_images(
        front_img: PIL.Image.Image,
        back_img: PIL.Image.Image,
        prompt: str) -> object:
    model = GenerativeModel(
        "gemini-3-flash-preview",
        generation_config={
            "response_mime_type": "application/json",
            "response_schema": NAMECARD_SCHEMA
        },
    )
    front_part = Part.from_data(
        data=pil_to_bytes(front_img), mime_type="image/jpeg")
    back_part = Part.from_data(
        data=pil_to_bytes(back_img), mime_type="image/jpeg")
    response = model.generate_content(
        [prompt, front_part, back_part],
        stream=False,
        labels={"client_id": "namecard"}
    )
    return response

The prompt also just adds a merge instruction after the original IMGAGE_PROMPT:

DOUBLE_SIDED_IMAGE_PROMPT = IMGAGE_PROMPT + """
These two images are the front and back of the same business card; please integrate them into a single complete record.
If both Chinese and English appear in the same field (such as name or company), please present them merged
(e.g., "Wang Daming David Wang");
if a field appears on only one side, use the value from that side; ignore obviously redundant information.
"""

One API call means we don't have to write or maintain any merging rules ourselves.

Two Easily Overlooked Pitfalls

During the overall code review before the feature went live, we caught two details that are easily overlooked but can really cause issues.

Pitfall 1: Timing of Duplicate Checks

Originally, the duplicate check (comparing if the email already exists) was done immediately after OCR. However, after the double-sided recognition went live, if the front side happened to have a duplicate email from an old record, the process would prematurely determine it "already exists" and end. This would mean any new email on the back side would never be seen.

The fix was to move the duplicate check later, performing it only after "single-side chosen not to merge" or "double-sided merge complete." This ensures we are always comparing the final version of the data:

async def _finalize_and_save_card(card_obj, event, user_id) -> None:
    existing_card_id = firebase_utils.check_if_card_exists(card_obj, user_id)
    if existing_card_id:
        # ... Reply already exists
        return
    card_id = firebase_utils.add_namecard(card_obj, user_id)
    # ... Reply save successful

Both the single-sided and double-sided merge workflows eventually converge to call this shared function, ensuring the duplicate check is executed only once when the data is finalized.

Pitfall 2: Don't Clear All States Indiscriminately

The user_states dictionary is actually shared by several features: editing memos, modifying fields, and this back-side recognition. The initial implementation, for convenience, would delete the entire state whenever a residual state was detected before processing a new event.

The problem is: if a user is "editing the phone field" and waiting to input a new number, but accidentally sends an image, this logic would clear the editing_field state as well, silently canceling the user's original editing operation.

The fix was to only clear the two states related to the back-side recognition process and leave other states untouched:

if state.get('action') in (
    'pending_backside_confirm', 'awaiting_backside_image'
):
    del user_states[user_id]

We later found a missing branch in the same logic; the part handling "operation expired" replies also cleared everything initially. It was only truly fixed after unifying it with the same selective judgment.

Incidental Resource Cleanup: Don't Let Back-Side Images Linger in Memory

The awaiting_backside_image state stores not just text, but also the raw byte data of the front image. If a user disappears after being asked "Is there a back side?", this data would theoretically stay in the process memory because the original design only checked and cleared timeout states during the "user's next interaction."

We added a sweep_expired_states() function, which runs immediately when a Webhook comes in. It clears all expired temporary states for all users, so we don't have to wait for the specific user to return for passive cleanup:

def sweep_expired_states() -> None:
    now = time.time()
    expired_user_ids = [
        user_id for user_id, state in user_states.items()
        if 'expires_at' in state and state['expires_at'] <= now
    ]
    for user_id in expired_user_ids:
        del user_states[user_id]

Whenever any user sends a message, it performs garbage collection for all users, ensuring that those who abandoned the process don't leave behind memory-consuming remnants.

Summary and Benefits

This double-sided recognition and merging feature makes the LINE business card bot much more aligned with the actual usage habits of Taiwanese users:

Single Recognition, Complete Data: Both sides are sent to Gemini at once, automatically merging Chinese and English fields, eliminating the need for manual duplicate comparison.
Non-Intrusive: Ignoring prompts, timeouts, or temporarily doing something else will naturally revert to single-sided storage without getting the workflow stuck.
Duplicate Checks Target Final Data: Ensures that comparisons are always made against the merged, complete version, so new information appearing only on the back side isn't missed.
Independent State Machines: The temporary state for back-side recognition only affects itself and doesn't interfere with other ongoing user operations.
Clean Memory Usage: Proactively cleaning up timeout states ensures that users who abandon the process don't leave an invisible memory burden.

The complete code has been pushed to GitHub, feel free to check it out!

[Digital Certificate Wallet] Advanced: Building "Visitor-Endorsed Issuance" – A Full-Chain DID Application as Both Verifier and

Evan Lin — Mon, 13 Jul 2026 08:01:01 +0000

(Source: Digital Certificate Wallet Official Website)

Premise:

The previous introductory version created an HR employee card system: colleagues apply for an employee card themselves and then use it to apply for "sports subsidies" and "childcare subsidies." The focus of that article was on separating the two roles: "Issuer" and "Verifier."

In this article, I want to take it a step further: what does it look like if a scenario needs to play both the verifier and issuer roles simultaneously to form a complete DID ecosystem chain? The scenario I chose is "Visitor Endorsement and Issuance"—this is also the one I felt best demonstrates the "full chain" when brainstorming five verifier applications.

By the way, this article will honestly document three pitfalls encountered during the development process, as those are the truly valuable parts of TIL (Today I Learned).

Code is here: https://github.com/kkdai/did-usecase-visitor
Online experience: https://did-usecase-visitor-660825558664.asia-east1.run.app

Scenario: Employee Access Control + Visitor Endorsement and Issuance

This lobby reception desk has two modes:

Employee Access Control / Event Registration: Employees present their employee cards using a digital wallet. The system only verifies "whether they are a valid employee." Once verified, the door opens or registration is successful. Fields like name, birthday, and number of children are not disclosed and remain in the wallet—this is selective disclosure.
Visitor Endorsement and Issuance (the protagonist of this article): An active employee presents their employee card to "endorse" the visitor. After verification, the system immediately issues a temporary visitor pass with an expiration time to the visitor's wallet.

The value of the second mode lies in connecting the two roles:

First act as a Verifier (verify employee card) → then act as an Issuer (issue visitor card) only after verification.

Compared to traditional paper visitor logs (copying IDs, holding physical IDs, piles of personal data at the counter requiring manual disposal), digital endorsement only leaves one piece of accountable information: "which employee endorsed it." Visitor data stays in the visitor's own wallet, and the pass can have an expiration time.

Architecture Decisions: Why not just modify the previous project?

This time, I started a brand new project and deployed it to a separate Cloud Run service instead of adding pages to the original HR project. Several considerations:

Static Frontend + JSON API: The original project used jade templates for server-side rendering. This time, it was changed to public/ static pages + a few JSON APIs (/api/access/qrcode, /api/access/status), separating the frontend and backend more cleanly.
Abstracting Wallet API calls into lib/wallet.js: In the original project, issuer/verifier calls were embedded in routes and duplicated. This time, they were extracted into three functions: requestPresentationQRCode(), getPresentationResult(), and issueCredential(), making it easier to maintain and test.
State changed to Memory: The original project wrote data to a single record.js file. Writing files in a stateless environment like Cloud Run causes issues. This time, a simple memory object was used (resets on restart, sufficient for demonstration purposes).
Reusing the same Sandbox Account tokens: The access tokens for the issuer/verifier are the same set as in the previous article, reused directly.

For the "holding an employee card" verification part, I first used the existing sports subsidy verifier ref as a fallback (if VERIFIER_ACCESS_REF is not set, use VERIFIER_SPORT_REF), so it could run without waiting for backend configuration.

Pitfall Record 1: Presentation successful, but the screen is stuck

This is a classic one. The phone scans the code, and the wallet completes the presentation, but the desktop page just won't move forward; it keeps polling.

First, checking the Cloud Run logs, I found that /api/access/status returns every 3 seconds, each time returning "unverified." I added a log line on the backend to print the original response from the verifier. After redeploying and testing again, I caught the truth:

{
  "data": [
    {
      "credentialType": "0028680530_line_school",
      "claims": [
        { "ename": "ename", "cname": "English Name", "value": "Lub" },
        { "ename": "join_company", "cname": "Join Date", "value": "2018-10-05" }
      ]
    }
  ],
  "verifyResult": true,
  "resultDescription": "success",
  "transactionId": "8cd7f37b-..."
}

See the problem? The field in the response is verifyResult (camelCase), and there is no code field at all. But I used the old logic from the previous article:

// Old (doesn't match current response)
const verified = data.code === 0 && data.verify_result === true;

data.code is undefined, and data.verify_result is also undefined (it's called verifyResult), so it's always false, always pending. Actually, the verification had already succeeded (verifyResult: true, resultDescription: "success"), but the field names I was checking didn't match—it seems the sandbox API response format has changed from snake_case to camelCase.

The fix was to change the logic to be compatible with both formats:

const verified =
  data.verifyResult === true || // New format camelCase
  data.verify_result === true || // Old format compatibility
  (data.code === 0 && data.verify_result === true);

TIL: When integrating third-party APIs, don't trust that "the logic that worked in the last version will work in this one." Sandboxes change. Adding a log line to print the raw response and comparing it is much faster than staring at the code and guessing.

Pitfall Record 2: Visitor card stuck at "Pending Issuance"

After the access control part worked, the visitor endorsement part got stuck—the screen showed "Visitor card pending issuance (issuer template not set)," and no card was actually issued.

I used curl to hit the issuance API /api/vc-item-data directly to see what it returned. I tested two scenarios:

Scenario A: Using employee template + correct employee fields → HTTP 200, and the full response contained these keys:

KEYS: ['id', 'content', 'pureContent', ..., 'qrCode', 'deepLink', 'expired', ...]
qrCode = data:image/png;base64,iVBOR... ← QR code that can actually be scanned into the wallet
deepLink = https://frontend-uat.wallet.gov.tw/api/moda/vcqrcode?...
expired = 2027-01-09T...

Scenario B: Using employee template + visitor fields (visitor_type, endorsed_by…) → HTTP 500 / 400 BAD_REQUEST.

The reason was clear: the employee template fields were isRequired: true (Name, English Name…), but I sent a bunch of visitor fields it didn't have, so it was rejected. And the successful issuance response actually includes qrCode and deepLink, which can be used directly for the visitor to scan and collect the card—my original parsing was correct; the bottleneck was purely "fields not matching the template."

So I designed two issuance modes, automatically switched by environment variables (HAS_VISITOR_TEMPLATE in the code):

Mode	Condition	Behavior	Card Face
Option 1 (fallback)	`VISITOR_VC_*` not set	Borrow the employee template, stuffing visitor info into its required fields (Name="Temporary Visitor", etc.) to issue the card	Displays as an employee card face
Option 2 (Formal)	`VISITOR_VC_*` set	Send `visitor_type / endorsed_by / valid_until` to the dedicated visitor template	Formal visitor pass card face

The advantage of Option 1 is that a real, collectable card can be issued without waiting for backend configuration (even if the card face is borrowed), allowing the entire chain to be tested first; for a formal card face, just go with Option 2 and build a dedicated template, with no code changes required.

Pitfall Record 3: Collection QR too small + Mobile layout

In the first version, I made the visitor pass look like a pretty little ID badge, and the collection QR was only 48px—the result was that it couldn't be scanned at all. This QR is meant to be scanned by "another phone" to collect the card; if it's too small, it loses its purpose.

Later, I changed the visitor card to a vertical layout, enlarging the collection QR to be the main body of the card (max 240px, white background with padding), with "Endorsed by / Valid until" information placed below. Both QRs (for presentation and collection) were also changed to clamp() responsive sizes, so they don't break the layout on mobile and are clear enough on desktop.

TIL: As long as a QR is "for others to scan," it must be treated as the protagonist of the layout, not as a decoration.

The Truth About "Automatic Expiration"

I originally thought I could specify "this visitor pass expires in 4 hours" for each card, but testing revealed that when issuing cards via /api/vc-item-data, the actual validity of the card follows the template settings (e.g., the employee template is issuance date + about half a year); it's not possible to specify a short expiration for individual cards.

So the "Valid until HH:MM" on the card face now is a display value calculated by the application layer, not a mandatory expiration enforced by the wallet. If a truly short-term visitor pass is needed, there are two ways:

When creating the visitor template, set the template's validity period to be short.
Or use the platform's scheduled revocation (revoke)—the issuance response includes fields like clearScheduleId and scheduleRevokeMessage, implying the platform supports scheduled revocation, but this requires integrating the corresponding API separately.

Deployment: Directly to Cloud Run from Source Code

This time, I used buildpacks to deploy directly from source code without writing a Dockerfile:

gcloud run deploy did-usecase-visitor \
  --source=. --region=asia-east1 --platform=managed --allow-unauthenticated \
  --set-env-vars="VC_SERNUM=607861,VC_UID=0028680530_line_school,\
ISSUER_ACCESS_TOKEN=...,VERIFIER_SPORT_REF=...,VERIFIER_ACCESS_TOKEN=...,\
VISITOR_TTL_HOURS=4"

To switch to the formal visitor card in Option 2 later, just add VISITOR_VC_SERNUM=<new template vcId>,VISITOR_VC_UID=<new template vcCid> to this --set-env-vars string and redeploy; HAS_VISITOR_TEMPLATE will automatically become true.

Summary and Future Outlook

The focus this time wasn't "making another demo," but three things:

The full DID chain is feasible: Playing both Verifier and Issuer in the same scenario—verifying one card and then issuing another—connects the ecosystem chain, and the experience is smooth.
Pitfalls are in the details: Field naming (verifyResult vs verify_result), mandatory template fields, QR size—these wouldn't be discovered without looking at the raw response and actually scanning with a phone.
Fallback design allows the demo to work first: No need to wait for every template/ref to be built on the backend; use existing resources to get the whole chain running first, then gradually switch to formal settings. The development rhythm is much better.

There are truly many application scenarios for digital certificate wallets; "visitor endorsement" is just one of them. The employee card from the previous article could also be extended to commissary discount redemption, seniority milestone gifts, childcare facility access, gym point accumulation... each is a new application for a "verifier." I look forward to seeing more creative scenarios being built.

[GCP Billing & Vertex AI] Solving Gemini Cost Allocation in a Single Project: Vertex AI Dynamic Billing Labels in Action

Evan Lin — Sun, 12 Jul 2026 15:09:09 +0000

Pain Point: How to Accurately Allocate Gemini API Costs Within the Same Project?

When developing enterprise-level LLM services or operating multi-tenant platforms, the question most frequently asked by finance and operations teams is:

"We have many different business lines and LINE Bots connected within the same GCP project. Every day, the Gemini Key costs all appear under the Gemini API category. Is there a way for us to split the costs based on different Gemini Keys or different users?"

Direct answer to your question: In Google Cloud Billing reports, it is not possible to directly display costs "based on different API Key names." The smallest attribution dimensions for Google Cloud billing reports are "Project," "Service," and "SKU (Product Line Item)." The system does not treat individual API Key strings as independent billing items. For the billing system, whether you create 10 or 100 API Keys within the same project, they will all be lumped together as a single Gemini API total.

A Workaround: Vertex AI "Request Labels" to the Rescue

If architectural constraints force you to stay within the same project, the most recommended approach is: switch to Vertex AI calls and use "Request Labels."

If you are currently using a Google AI Studio API Key, it cannot pass billing labels within a single project. However, if you change your code to call the Vertex AI Gemini API (still within the same project), Vertex AI supports dynamically including custom labels with each request.

Principle and Workflow

When sending each request (e.g., calling generateContent), include specific metadata in the API Request:

{
  "contents": { ... },
  "labels": {
    "client_id": "info_helper",
    "api_key_group": "marketing_team"
  }
}

These custom labels are passed directly to the GCP billing system. Later, when you go to the GCP Billing report and select your set label key (e.g., client_id) in "Group by," you can clearly see the costs for different labels (representing different services, clients, or users) within the same project!

Project Implementation: Full Adoption of the Labels Mechanism

To fulfill this requirement, we audited the current API call architecture of the LINE Bot project and performed the following refactoring.

1. Project API Call Audit

Through scanning, we found that the vast majority of calls in the project use Vertex AI (14 out of 17 clients use vertexai=True), with only a few exceptions:

Vertex AI calls: Including GitHub summaries, multiple Google Maps Grounding tools, text summarization, image analysis, speech-to-text, etc. (total of 11 files, 19 call points).
Gemini API Key calls: Live API in main.py, Batch service in batch_service.py, and TTS speech synthesis in tts_tool.py.

[!IMPORTANT] The labels parameter is only supported by Vertex AI. If this parameter is included under an API Key (vertexai=False), it will cause the SDK to throw an error. Therefore, we only modified the 11 files that use Vertex AI.

2. Implementation Method

For the google-genai Python SDK, we have two main modification scenarios:

Scenario A: Already contains `GenerateContentConfig`

If the original call already includes a Config, we just need to pass an additional labels={"client_id": "info_helper"} into the config:

# Before (e.g., loader/gh_tools.py)
response = client.models.generate_content(
    model="gemini-2.5-flash",
    contents=prompt,
    config=types.GenerateContentConfig(
        temperature=0,
        max_output_tokens=2048,
    )
)

# After
response = client.models.generate_content(
    model="gemini-2.5-flash",
    contents=prompt,
    config=types.GenerateContentConfig(
        temperature=0,
        max_output_tokens=2048,
        labels={"client_id": "info_helper"}, # Include billing label
    )
)

Scenario B: No Config parameter

If the original call is very simple (e.g., searchtool.py or youtube_gcp.py), we need to proactively include a GenerateContentConfig containing labels:

# Before (e.g., loader/searchtool.py)
response = client.models.generate_content(
    model="gemini-3.1-flash-lite-preview",
    contents=prompt,
)

# After
response = client.models.generate_content(
    model="gemini-3.1-flash-lite-preview",
    contents=prompt,
    config=types.GenerateContentConfig(
        labels={"client_id": "info_helper"}, # Add config to include label
    ),
)

3. List of Modified Files

We performed precise modifications on a total of 19 call points across the following 11 files, and used Python's AST module (ast.parse) and Flake8 for syntax and formatting checks before submission:

agents/chat_agent.py: Modify _create_chat_config() to add labels to both general Q&A and Grounding conversations.
loader/chat_session.py: Include labels in Chat session config.
loader/gh_tools.py: GitHub summary API.
loader/langtools.py: Text summarization, image JSON generation, social media post generation.
loader/maps_grounding.py: Maps search API.
loader/searchtool.py: Keyword extraction tool.
loader/youtube_gcp.py: YouTube video understanding API.
tools/audio_tool.py: Asynchronous speech-to-text.
tools/maps_tool.py: 5 call points including nearby search, restaurant name extraction, batch and review search, etc.
tools/summarizer.py: Text summarization and Agentic Vision image understanding.
tools/youtube_tool.py: YouTube summary tool.

Pitfall Guide: Watch Out for SDK Module Import Issues

When refactoring calls without Config for youtube_gcp.py and youtube_tool.py, since these two files originally only used named imports for specific types:

from google.genai.types import HttpOptions, Part

When we write types.GenerateContentConfig(...) in the code, the system throws a NameError: name 'types' is not defined error.

Solution: We need to correct the import statement and directly import GenerateContentConfig:

# Before
from google.genai.types import HttpOptions, Part

# After
from google.genai.types import HttpOptions, Part, GenerateContentConfig

And use it directly in the call without the types. prefix:

config=GenerateContentConfig(labels={"client_id": "info_helper"})

Summary and Next Steps

This modification successfully injected the client_id=info_helper label into all Vertex AI API calls within the LINE Bot project.

Billing Delay: Please note that after we start including labels, GCP billing data usually has a 24 to 48-hour delay before taking effect.
Configure in GCP Billing: After two days, you can go to the GCP Console -> Billing -> Reports. In the "Group by" section on the right, select Labels and enter our key client_id.
Mission Accomplished: At this point, the report will draw info_helper as a separate billing row, perfectly solving the problem of separating project costs for reimbursement and statistics!

[AI in Action] Refining a macOS Meeting Translation App with Claude Code: Auto-reconnect, Floating Captions, and Meeting Minutes Export Evolution

Evan Lin — Sun, 05 Jul 2026 11:00:46 +0000

Foreword: Round Two, Switching to a Sharper Tool

In the previous article, we used AGY CLI (Antigravity) to build a macOS real-time meeting translation App from scratch: capturing audio from Zoom / Google Meet via ScreenCaptureKit, sending it to the Gemini Live API for real-time translation, and displaying Traditional Chinese bilingual subtitles in the App window.

After the App went live, the developer discovered several issues during actual meetings that raised eyebrows, while also coming up with more functional ideas. This time, we switched to Anthropic's Claude Code to start the second round of deep polishing in the terminal.

The following records the complete dialogue flow of this collaboration, restoring every key decision point.

Phase 1: Uncovering Hidden Dangers — The Mystery of the WebSocket Stopping After 10 Minutes

The App seemed to work perfectly, but the developer returned with a question after a real, long meeting:

User: Check this code. Why does the real-time translation stop after about ten minutes? Help me look into possible reasons.

Claude Code read all five Swift source files and, combined with the built-in Gemini Live API documentation, precisely pointed out the root of the problem:

The Gemini Live API's WebSocket connection has a Session limit of about 10 minutes. Once the time is up, the server actively closes the connection and sends a GoAway signal. However, the original code was completely unprepared for this scenario, causing a three-layer failure:

No reconnection after disconnection: The didCloseWith callback only updated the status text on the UI without any reconnection logic.
Silent audio discarding: After isConnected = false, every audio chunk from the microphone was silently consumed by guard isConnected else { return }, without the user noticing.
Incorrect isRunning state: The "Translating" icon in the main window was still flashing, but in reality, nothing was happening.

Claude Code immediately refactored GeminiLiveConnection.swift, introducing three lines of defense without changing any external interfaces:

Proactive GoAway Signal Detection: Parse the goAway JSON message sent by the server in parseServerResponse(), and actively reconnect before the connection actually drops to achieve seamless translation continuity.
Exponential Backoff Auto-Reconnection: After disconnection, retry at increasing intervals of 2s → 4s → 6s, up to 10 times; the count resets after a successful reconnection, ensuring the next disconnection also has a full set of retry opportunities. The entire process is silent and transparent to the user.
30-second Ping Keep-Alive: Start a Timer after a successful connection to send a sendPing to the server every 30 seconds, preventing intermediate network equipment from cutting the connection due to inactivity.

After the modifications were completed, the developer executed bash build_app.sh and pushed to GitHub:

Claude Code: Build successful, commit completed, pushed. Real-time translation should now be able to operate continuously for a long time, silently reconnecting automatically when the Session expires without the user noticing an interruption.

Phase 2: Feature Interview — Five Questions to Lock Down Two Priority Development Directions

With stability resolved, the developer began thinking about how the App could go further:

User: Help me research carefully and give me three new feature recommendations.

After carefully reading the code, Claude Code first proposed a directional question instead of directly giving a list:

Which category of these three new feature directions do you prefer?

A. Utility / UX — Making daily use more convenient (transcript export, hotkeys, meeting notes)

B. Translation Quality — More accurate and natural (multi-language detection, speaker identification)

C. System Integration — Connecting with other tools (Notion, Zoom overlay)

User: A

Three UX features were immediately produced: Floating Subtitle Window, Automatic Meeting Transcript Export, and Global Hotkeys. The developer's response was direct:

User: I want both 1 and 2.

Next was a brief requirement interview, where Claude Code asked only one most critical question at a time:

How many lines should the floating window display? → Double lines (small text for original + large text for translation)
Background style? → Vibrancy effect (frosted glass)
Export method? → Automatically save to the desktop, no dialog box

After five questions, the design direction was completely clear. Claude Code proposed a complete design plan and wrote the specification document. After the developer confirmed "no problem," it entered the implementation phase.

Phase 3: Plan-Driven Development — Subagent Closed-Loop Delivery, Review Catches Critical Bugs

With clear specifications, Claude Code entered its most proficient work mode: Write a plan first, then use multiple independent Subagents to execute tasks, with each Task immediately reviewed by a Reviewer Subagent upon completion.

The entire process was divided into three Tasks; the two most critical ones are recorded below:

Task 1: Automatic Meeting Transcript Export

The Implementer Subagent quickly completed three things: removed the original 25-line history limit, added the exportTranscript() method, and automatically saved the complete bilingual comparison record in Markdown format to the Desktop when translation stopped.

However, the Reviewer Subagent immediately raised a flag:

Critical Issue found: status = "Stopped" in stop() is executed immediately after exportTranscript(), instantly overwriting the save path message. The user will only ever see "Stopped" and will never know where the file was saved.

This was a logic bug just one line away, which would have been very easy to overlook without a Reviewer. The Fix Subagent then intervened, changing exportTranscript() to return a Bool: when export is successful, stop() no longer overwrites the status; "Stopped" is only displayed when there are no records to export. After the modification, the Reviewer confirmed again, and all passed.

Task 2: Floating Subtitle Window

Added FloatingSubtitleWindow.swift, with a core structure of three layers:

NSPanel (level = .floating): Always on top, does not steal focus (.nonactivatingPanel), and can be displayed across full-screen Apps.
NSVisualEffectView (material = .hudWindow): Native macOS vibrancy effect.
NSHostingView embedding SwiftUI's FloatingSubtitleView: Directly bound to TranslatorViewModel.currentLine, updating in real-time.

At the same time, ownership of TranslatorViewModel was moved up from ContentView to TranslatorApp, allowing the main window and the floating window to share the same data source, avoiding data duplication or synchronization issues. The window position is saved to UserDefaults after dragging and automatically restored after a restart.

The Task Reviewer checked all 11 specifications one by one; all passed without any need for correction.

The entire "Implementation → Review → Correction → Re-review" closed loop was completed automatically by subagents. The developer only needed to confirm that the final bash build_app.sh passed cleanly:

Claude Code: Build successful, commit completed, pushed.

Phase 4: App Brand Upgrade — Real-time Generation of Professional Icons with Python

With features complete, the developer turned their attention to the appearance:

User: The app icon doesn't look good, help me generate a professional one.

Claude Code first confirmed that Pillow (Python image library) was in the environment, then directly wrote a complete Icon generation script with the following design description:

Background: Deep sea blue gradient (#0D1B4E → #1565C0), standard macOS 22% rounded corners, echoing the macOS Design Language.
Core Pattern: Two overlapping speech bubbles. The upper bubble (semi-transparent white) contains "A" representing the original English audio, and the lower bubble (pure white) contains "中" representing the translated output. They are connected by a bidirectional arrow in the center, making the "real-time translation" product positioning clear at a glance.
Fonts: Avenir Next for English and Apple SD Gothic Neo for Chinese, both of which are built-in macOS fonts, requiring no external resources.

The script output 10 sizes at once (16px → 1024px), converted them into an .icns file using the system's iconutil command, and automatically updated build_app.sh to copy the icon into the App Bundle, adding the CFBundleIconFile declaration to Info.plist. The entire process did not require opening Xcode or using any image design tools.

Phase 5: Code Quality Refinement — Clearing All Compilation Warnings

When the developer executed bash build_app.sh for acceptance, they noticed a few lines of yellow warnings in the output:

User: There are some warnings when running build_app.sh, help me check them.

Claude Code carefully executed the Build and categorized three types of warnings, treating them accordingly:

Warning Type	Root Cause	Fix
`onChange(of:perform:)` deprecated × 2	`swiftc` did not specify a deployment target, defaulting to the latest SDK rules	Added `-target arm64-apple-macos13.0` to `build_app.sh` to let the compiler know we are targeting macOS 13, where the old API is the correct choice
`SCRunningApplication` non-Sendable × 2	Types in the ScreenCaptureKit framework are not marked `Sendable`	Changed `import ScreenCaptureKit` to `@preconcurrency import ScreenCaptureKit`
`TranslatorViewModel` non-Sendable captured	ViewModel was captured in a `@Sendable` closure	Added `@MainActor` to `TranslatorViewModel` (modern standard practice for SwiftUI ViewModels) and added `@preconcurrency` to delegate conformance to suppress derived warnings

The final Build output was as clean as new, without any Warnings:

🛠 Starting compilation of Swift files (target: arm64-apple-macos13.0)...
🎨 Copying App Icon...
📝 Generating Info.plist...
✅ Packaging complete!

All modifications were committed and pushed to GitHub together.

Phase 6: Real-World Pitfalls — The ScreenCaptureKit Permission Labyrinth

The App's functionality seemed complete until the developer actually turned it on to start using it:

User: Is it a permission issue? I open the app and it keeps failing to scan the "Target App" list. Help me check the relevant code.

The App list was always empty. "Screen Recording" in System Settings was indeed checked. This is a typical "permissions are granted, but it just doesn't work" dead-end problem.

First Cut: Handling Silent Failures

After reading AudioCaptureManager.swift, Claude Code immediately spotted the problem: when the fetchShareableApps() call to SCShareableContent.current failed, it only printed to the console. The UI showed an empty list without any prompt. The developer had no idea what was happening.

The first wave of modifications did three things:

Added NSScreenCaptureUsageDescription to Info.plist: Without this key, the macOS authorization dialog will never pop up.
Added an ad-hoc signing step: codesign --sign - --force --deep — ScreenCaptureKit requires the App to have a code identity to appear in the "System Settings > Screen Recording" list.
Surfaced errors to the UI: Changed fetchShareableApps() to return (apps, errorMessage?). Any failure would be displayed in the App's status bar, allowing the developer to see immediately what happened.

Build completed, tested again—still the same error message.

Second Cut: Overly Aggressive Error Classification Logic

Looking closely at the error judgment code:

let isPermissionDenied = nsError.domain == "..." && nsError.code == 1
    || error.localizedDescription.lowercased().contains("permission")
    || error.localizedDescription.lowercased().contains("denied")

The contains("permission") line was too aggressive. As long as any word containing "permission" appeared in the error description, it would be incorrectly judged as "Permission Denied," displaying "Please go to System Settings to enable authorization." In reality, it could be a completely different error.

Claude Code corrected the judgment logic—only the exact ScreenCaptureKit userDeclined error code (-3801) is treated as a permission issue. All other errors display the actual domain, code, and description for easier diagnosis:

let isPermissionDenied = nsError.code == -3801
let message = isPermissionDenied
    ? "Screen recording permission required: Please go to System Settings to enable authorization"
    : "Unable to get App list (code \(nsError.code)): \(error.localizedDescription)"

Third Cut: Finding the Root Cause — TCC Identity Mismatch

After correcting the error classification, Claude Code ran the App and captured the logs, finding that the status bar displayed a new message with a code number, not -3801. This confirmed: The problem wasn't that the user hadn't given permission, but that macOS didn't recognize the App at all.

The root cause:

Every time build_app.sh is executed and re-signed ad-hoc, the hash of the binary changes, and the macOS TCC database treats it as a completely new App. The old screen recording authorization was given to the previous binary; the new binary did not inherit it. System Settings shows it as checked, but that's authorization for the old identity, which is invalid for the new binary.

The solution is to reset TCC to let macOS re-trigger the authorization dialog:

tccutil reset ScreenCapture com.poc.MeetingTranslator

After execution, reopening the App and clicking "↻" caused macOS to immediately pop up the "MeetingTranslator wants to record the contents of this screen" dialog. Clicking "Allow" instantly listed all running applications in the App list.

Permanent Countermeasure: Writing the Reset into the Build Process

The ad-hoc signing issue persists during development—every rebuild requires re-authorization. Claude Code added tccutil reset directly as the last step of build_app.sh:

tccutil reset ScreenCapture com.poc.MeetingTranslator 2>/dev/null && \
  echo "✅ Reset complete. The system will ask for authorization again after opening the App" || true

From then on, after every bash build_app.sh, simply open MeetingTranslator.app, and the system will ask for authorization again. The entire development cycle will never again get stuck in the "permissions are granted but it doesn't work" loop.

Conclusion: The True Value of the "Plan → Subagent Implementation → AI Review" Closed Loop

This collaboration with Claude Code made me feel a completely different way of working compared to the first AGY CLI development:

Proactively asking questions instead of just starting: Faced with "give me three new feature recommendations," Claude Code's first step was to ask for a direction; faced with the "floating window," it confirmed style and details one by one. This rhythm of "align first, then implement" is much more reliable than directly guessing requirements.
The plan is a moat for quality: Writing specification documents and implementation plans before implementation gives each Subagent clear boundaries and acceptance criteria. This seemingly "redundant" step directly discovered a state-overwriting bug in the Task 1 review that human developers could easily overlook.
AI Reviewing AI is a different layer of protection: The Reviewer Subagent and Implementer Subagent are started completely independently; they share no context. Because of this, the Reviewer can discover the Implementer's blind spots from a fresh perspective—this is the extra protection brought by "AI double-checking," not a replacement for human Code Review, but a completely new level of quality.
Tool boundaries are functional boundaries: App Icon generation, Warning fixes, Git commit/push—Claude Code moves freely throughout the development environment. The developer doesn't need to switch tools; all actions are completed within the dialogue.
Real-world use is the best test: The ScreenCaptureKit issue in Phase 6 never appeared in any build tests until the developer actually turned it on to use it. This kind of "silent failure + system-level identity mismatch" problem only surfaces in real-world scenarios. Claude Code's diagnostic method—from correcting error classification to letting the UI display real error codes, to finding the TCC root cause—is a typical "narrowing the hypothesis range and letting the problem speak" debugging approach.

If the first article was about "from zero to one," this record is about "from usable to good, and then to standing firm in real-world scenarios." Two types of AI Agents, two collaboration styles, together completed a full Native macOS App covering low-level audio, WebSocket connections, SwiftUI UI, Python image generation, and system permission diagnosis. See you next time!

[Gemini API in Action] Building MemeFinder: A Native Mac Menu Bar Widget for Finding Memes via Text Using Gemini Vision & Semantic Embeddings

Evan Lin — Mon, 22 Jun 2026 00:41:17 +0000

The Origin: Mid-Conversation, Where on Earth Is That Meme?

Anyone who chats a lot has a folder full of memes on their phone and computer, but the moment you actually need one — the conversation is rolling, you want to drop a "thanks but no thanks" or an "I'm trash" reaction — you can't find it. The filename is IMG_4821.jpg, the photo library has no categories, and search is a non-starter.

I first came across a wonderful open-source project, ShiQu1218/MemeTalk. It builds a local meme semantic-search system with Python + Streamlit + SQLite: it scans your local meme folder, indexes images with OCR and vector embeddings, then does multi-route retrieval. Feature-complete, but research-oriented and requires opening a browser to run Streamlit.

What I wanted was something closer to an "everyday handy tool":

A native Mac app, one search box. I type what I'm looking for and the relevant meme pops up. Click it and it's copied straight to the clipboard.

So MemeFinder was born. This post records its journey from zero to "menu-bar resident + global hotkey," and several representative pitfalls along the way.

System Design and Architecture

The core concept is simple: point at a local meme folder → have Gemini build an index for each image → type to do a semantic search → click to copy.

I made three key technical decisions:

Native SwiftUI app, not Electron. Copying images to the clipboard, global hotkeys, menu-bar residency — with AppKit these are all first-class citizens.
Gemini does two things: the vision model gemini-3-flash-preview reads the text in each image and generates a Traditional Chinese description plus emotion tags; gemini-embedding-2 turns that semantics into a 768-dimensional vector.
Hybrid semantic-vector + keyword search. Pure keyword recall for Chinese is too poor; only semantic vectors achieve "type a related description and find the image."

System Architecture Flow

The project is deliberately split into two Swift Package targets:

Target	Type	Contents
`MemeFinder`	library	Logic, models, services, ViewModels (all unit-tested)
`MemeFinderApp`	executable	SwiftUI views + menu-bar shell (thin layer, depends on the library)

This split isn't decorative — it directly determines whether the tests can run smoothly, as "Pitfall #2" will explain.

Core Implementation

1. Auto-tagging memes with the Gemini vision model

During indexing, each image is sent to the vision model with a request to output only JSON: the text in the image, a Traditional Chinese description, tags, and emotion. responseMimeType is set to application/json to keep the output format stable:

public static func annotateRequest(apiKey: String, imageData: Data, mimeType: String) -> URLRequest {
    let prompt = """
    你是迷因圖標註助手。請閱讀這張圖，輸出 JSON，欄位：
    ocr_text(圖中所有文字), description(用繁體中文描述畫面與梗),
    tags(3-8 個繁體中文關鍵字陣列), emotion(單一情緒詞)。只輸出 JSON。
    """
    let body: [String: Any] = [
        "contents": [[
            "parts": [
                ["text": prompt],
                ["inline_data": ["mime_type": mimeType, "data": imageData.base64EncodedString()]]
            ]
        ]],
        "generationConfig": ["responseMimeType": "application/json"]
    ]
    // ... set URL, x-goog-api-key header, POST body
}

2. Hybrid semantic + keyword ranking

After the query string is embedded into a vector, we compute cosine similarity for every image, then add weight for keywords that hit the OCR text and tags, and merge-sort:

public func search(queryEmbedding: [Float], queryText: String,
                   in images: [IndexedImage], limit: Int) -> [SearchResult] {
    let tokens = queryText.lowercased().split(whereSeparator: { $0.isWhitespace }).map(String.init)
    let results: [SearchResult] = images.compactMap { image in
        let cos = cosineSimilarity(queryEmbedding, image.embedding)
        let haystack = (image.ocrText + " " + image.tags.joined(separator: " ")).lowercased()
        let matches = tokens.filter { !$0.isEmpty && haystack.contains($0) }.count
        let boost = 0.1 * Float(min(matches, 3))   // keyword boost capped at 0.3
        let score = cos + boost
        return score > 0 ? SearchResult(image: image, score: score) : nil
    }
    return Array(results.sorted { $0.score > $1.score }.prefix(limit))
}

The whole search engine is a pure function, with Gemini hidden behind a protocol, so this logic can be fully unit-tested offline without hitting the real API.

Major Pitfalls and Solutions

The real time sink in this project was never the happy path — it was the pitfalls below.

Pitfall #1: The mysterious `GeminiError error 0` — indexing and search both fail

App packaged, key set, folder chosen, hit search — and nothing shows below, just GeminiError error 0.

Rather than guessing, I hit the embedding endpoint once with a real key and printed the response:

curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-embedding-2:embedContent" \
  -H "x-goog-api-key: $KEY" \
  -d '{"content":{"parts":[{"text":"貓"}]},"output_dimensionality":768}'

The evidence was unmistakable:

{ "embedding": { "values": [ -0.0063, -0.0200, ... ] } }

The problem: my parser was reading the plural embeddings[0].values (that's the batchEmbedContents batch-endpoint format), but the single embedContent call returns the singular embedding.values. So every embed call failed — indexing each image failed, embedding the query string failed — all throwing badResponse (shown in the UI as GeminiError error 0).

[Solution]
Fix the parser to read the singular embedding.values, keeping the plural format as a fallback; I also hardened the annotation parser (a thinking model sometimes returns a textless "thought" part first, so skip to the first part that actually has text):

public static func embedding(fromEmbedContent data: Data) throws -> [Float] {
    guard let root = try? JSONSerialization.jsonObject(with: data) as? [String: Any] else {
        throw GeminiError.badResponse("cannot parse embedContent payload")
    }
    // A single embedContent returns {"embedding":{"values":[...]}}
    if let embedding = root["embedding"] as? [String: Any],
       let values = embedding["values"] as? [Double] {
        return values.map(Float.init)
    }
    // batchEmbedContents is {"embeddings":[{"values":[...]}]} — tolerate it too
    if let embeddings = root["embeddings"] as? [[String: Any]],
       let values = embeddings.first?["values"] as? [Double] {
        return values.map(Float.init)
    }
    throw GeminiError.badResponse("cannot parse embedContent payload")
}

Lesson: trust the actual API response over your memory or secondhand docs. A single line of curl saved countless guesses.

Pitfall #2: SwiftPM's `main` entry-point conflict and the SwiftUICore linking error

I initially made the whole project a single executableTarget with the tests depending on it directly. The result: tests failed to link no matter what. An executable target needs a main entry point, but that entry point only exists at the UI step's @main App; and casually adding a placeholder main.swift then conflicts with @main (Swift doesn't allow two entry points in one target). Worse, SwiftUI in an executable target spews SwiftUICore.tbd ... not an allowed client linker warnings.

[Root cause analysis and solution]
This is actually an architecture problem, not a compilation problem. The right approach is to split the project into two layers:

MemeFinder (library target): all logic, models, services, ViewModels — the tests depend only on this layer, it has no entry point, and it links cleanly as a library. ViewModels import Combine (not SwiftUI) to get ObservableObject.
MemeFinderApp (executable target): only SwiftUI views and @main, with import MemeFinder to use the public types above.

After the split, the library and tests don't touch SwiftUI at all, the linker warnings disappear, and the @main conflict no longer exists. "What the tests need to depend on" often forces out clean module boundaries.

Pitfall #3: Parallel indexing's rate limit and "I want to stop indexing halfway"

The first version indexed one image at a time, serially calling Gemini (annotate then embed). For hundreds of images this was painfully slow. So I switched to bounded parallelism with withTaskGroup (at most 4 at once), which brought three new problems:

The Gemini free tier has a rate limit — too much concurrency triggers 429.
The user wants to cancel halfway through a large folder.
Parallel completion order is chaotic, but the results need stable sorting.

[Solution]
Handle the three problems separately, all converging in the same buildIndex:

429 backoff retry: retry only GeminiError.rateLimited with exponential backoff (max 3 attempts); other errors are recorded without retry.
Cooperative cancellation: honor Task.isCancelled; on cancel, stop scheduling new work and keep the completed portion. Even the backoff Task.sleep lets CancellationError propagate normally instead of swallowing it and firing one more API call.
Stable sorting: collect results into a [path: image] dictionary, then reassemble the output in the order of the pre-sorted file list, decoupled from completion order.

// Seed maxConcurrent tasks first, then refill one per completion — strictly cap concurrency
for _ in 0..<maxConcurrent { if !scheduleNext() { break } }
while let res = await group.next() {
    if let img = res.image { resultsByPath[res.path] = img }
    if let err = res.error { errors.append(err) }
    done += 1
    progress(done, total)
    _ = scheduleNext()
}

Incidentally, the HTTP status code was also extracted into a pure function mapResponse(data:statusCode:): 429 → rateLimited, other non-2xx → httpError(code), 2xx → return the data. The retry logic then has a basis, and this part is easy to test too.

Pitfall #4: Evolving from a "windowed app" into "menu-bar resident + global hotkey"

Whether a tool is pleasant to use comes down to "how many steps to summon it." I wanted to hit ⌃⌘M mid-conversation to bring up the search popover, with the app tucked into the menu bar, not occupying the Dock. This step hit two classic macOS pitfalls:

(a) Does a global hotkey need accessibility permission? No. Use Carbon's RegisterEventHotKey to register a fixed hotkey, which doesn't need Accessibility permission (unlike monitoring the whole keyboard). But under Swift 6 strict concurrency, the C event callback has to dispatch through a static id → instance registry, requiring nonisolated(unsafe) and relying on the invariant that "Carbon events are delivered on the main thread" for safety. If ⌃⌘M is already taken, RegisterEventHotKey returns failure — in which case we silently degrade, log a line, and the menu-bar icon still works.

(b) The timing race in the menu-bar right-click menu. The initial approach was "set statusItem.menu → performClick → immediately clear menu," but clearing synchronously fights AppKit's menu-tracking loop, and the menu flashes and disappears.

[Solution]
Pop the menu up directly, fully bypassing the assign-and-clear of statusItem.menu:

@objc private func statusButtonClicked() {
    guard let event = NSApp.currentEvent else { togglePopover(); return }
    if event.type == .rightMouseUp {
        // Pop up directly; don't assign then synchronously clear statusItem.menu
        // (it races AppKit's menu-tracking loop)
        if let button = statusItem?.button {
            NSMenu.popUpContextMenu(makeMenu(), with: event, for: button)
        }
    } else {
        togglePopover()
    }
}

Finally, adding LSUIElement = true to the Info.plist produced by build-app.sh makes the Dock icon disappear, and MemeFinder officially becomes a pure menu-bar tool.

Pitfall #5: The settings form is blank — one symptom, three layers of cause

After moving to the menu-bar version, a user reported "the settings window is completely blank." This seemingly simple bug, peeled apart, actually had three layers, each highly representative.

Layer 1: a Form collapses to zero height inside a hand-rolled NSWindow.
Originally the settings screen lived in SwiftUI's native Settings { } scene, which sizes it sensibly. After the refactor it was hosted in a hand-rolled NSWindow(contentViewController: NSHostingController(rootView: SettingsView())), and SettingsView ended with only .frame(width: 460) — width only, no height. NSWindow(contentViewController:) sizes the window from the content's natural size, but a SwiftUI Form is vertically greedy; with no constraint, its natural height resolves to nearly 0, so the window opens as a 460-wide, near-zero-height blank strip. The fix is just to add a height:

.padding(20)
// When hosted in a hand-rolled NSWindow (not a SwiftUI Settings scene), a Form
// with no height constraint collapses to ~0, turning the window into a blank strip.
.frame(width: 460, height: 320)

Layer 2: ⌘, and the menu-bar "Settings…" go down two different paths.
After adding the height, the user said "still blank." On follow-up I found out he was summoning settings with ⌘,, while the menu-bar right-click "Settings…" went down a different path. The reason: ⌘, in a SwiftUI app triggers the Settings { } scene, and to dodge a state-sharing problem during the refactor, I had set that to Settings { EmptyView() }:

// During the refactor, the Settings scene was left empty to avoid state-sharing
// — so ⌘, opens a blank window
var body: some Scene {
    Settings { EmptyView() }
}

In other words, settings had two entry points pointing at different things: ⌘, pointed at the empty scene, the menu-bar "Settings…" pointed at the real window. The fix unifies the two paths — let the Settings scene host the real SettingsView (so ⌘, works directly), and make the menu-bar "Settings…" open the same native settings window too:

Settings {
    SettingsView(vm: appDelegate.settings, indexing: appDelegate.indexing,
                 onReindex: { appDelegate.reindexNow() },
                 onCancel: { appDelegate.cancelReindex() })
}

// The menu-bar "Settings…" now opens the same Settings scene
@objc private func openSettings() {
    NSApp.activate(ignoringOtherApps: true)
    NSApp.sendAction(Selector(("showSettingsWindow:")), to: nil, from: nil)
}

This also leverages the fact that a SwiftUI App body is @MainActor-isolated — so reading the @MainActor appDelegate.settings directly from the body is legal, with no extra bridging needed.

Layer 3 (the most insidious): open doesn't reload a menu-bar app at all.
The biggest time-waster in the process was that after recompiling, I'd ask the user to open MemeFinder.app, yet he kept seeing the old behavior. Because MemeFinder is an LSUIElement menu-bar-resident app — when an instance is already running, open only wakes the existing old process instead of relaunching with the new binary. So we were actually testing the same old build the whole time. The correct dev loop is to truly kill it first, then run from source:

killall MemeFinderApp 2>/dev/null; swift run MemeFinderApp

This layer reminds me: when debugging, first confirm "what you're testing really is the version you changed" — otherwise all your reasoning is built on faulty observations.

On the "Development Process" Itself

This project was driven almost entirely by an AI agent workflow of spec → plan → subagent task-by-task implementation → two-stage review: each feature started with a design spec, was broken into independently testable small tasks, every task wrote a failing test first (TDD) before implementing, and after completion an independent review agent checked spec compliance and code quality, followed by one final whole-branch review.

Several of the pitfalls — GeminiError error 0, the library/executable split, swallowing CancellationError during backoff, the menu timing race — were in fact caught half the time during the review stage, not written correctly on the first pass. This echoes that old principle: having tests as armor, and someone (or an agent) seriously reading the diff, matters far more than writing fast. The final project maintains 47 unit tests and a zero-warning release build.

Results and Benefits

Type to find, click to paste: type a Chinese description in the menu-bar popover, semantic search instantly lists relevant memes, click one to copy it to the clipboard and paste straight into LINE / Slack / Messages.
Privacy-friendly, searchable offline: images and the index live locally (~/Library/Application Support/MemeFinder/index.json); only the "build the index" step calls Gemini.
A truly handy tool: ⌃⌘M is available anytime, menu-bar resident, no Dock footprint; incremental indexing only processes new/changed images, and indexing can show progress and be canceled.
A clean, maintainable architecture: a two-layer library/executable design, Gemini hidden behind a protocol, pure logic fully covered by tests.

All the development code for this project is open-sourced on GitHub: kkdai/meme-finder-app. Feel free to clone it, point it at your own meme-collection folder, and experience the joy of "type to find your meme"!

[Gemini API] Gemini Batch API and Webhook API practical usage on restaurant survey

Evan Lin — Mon, 15 Jun 2026 04:09:16 +0000

A Powerful Tool for Asynchronous Processing: Gemini Batch API & Webhooks

When developing LLM-based applications, we often need to handle a large number of data analysis tasks—for example, analyzing reviews from dozens of restaurants at once, classifying a large volume of articles, or batch generating translations. If we use traditional synchronous APIs (real-time calls), we would not only face severe Rate Limit blockages but also fail due to network connection timeouts and extremely high computing costs.

To overcome this limitation, Google has launched the Gemini Batch API and Webhook API:

Gemini Batch API: Allows developers to package a large number of requests into a JSONL file and upload them all at once. Gemini performs asynchronous scheduled computations in the background, without consuming your daily real-time API quotas (Rate Limits), and its computing cost is usually half that of real-time APIs, making it a perfect choice for non-urgent big data processing.
Webhook API: Traditional Batch tasks require us to constantly write polling logic locally to check the status. With Webhooks, when Gemini completes a Batch computation, it actively sends an HTTP POST callback to your specified URL, instantly notifying you that the task is complete, making the system architecture more elegant and energy-efficient.

This article will document how we integrated these two powerful APIs into our LINE Bot Restaurant Analysis Assistant to achieve one-click deep review and signature dish big data analysis for specific restaurants on mobile devices.

System Design and Optimized Architecture

Originally, the restaurant analysis function worked by having the Bot list nearby restaurants when a user sent their location, and then providing a generic "Deep Review Analysis (Batch)" button. Clicking it would send all nearby restaurants for analysis at once. However, this led to a poor UX: analyzing all restaurants took too long, and users often only wanted to delve into one specific restaurant they were interested in.

Therefore, we optimized the function into dynamic Quick Reply buttons:

The user sends their location, and the Bot searches for nearby restaurants via Google Maps Grounding.
After the client receives a plain text list of restaurants, the Bot automatically uses Gemini to extract the top 3 highest-rated restaurant names.
Three customized Quick Reply buttons are generated (e.g., 🍴 Analyze Din Tai Fung).
After the user clicks a specific restaurant button, the Bot immediately replies "Processing" to avoid LINE timeouts, and submits the Batch task for that single restaurant in the background. Once Gemini completes the computation, it proactively pushes a dedicated big data report.

System Architecture Flow

graph TD
    A[User Sends Location] -->|Location Message| B[Google Maps Grounding Search]
    B -->|Plain Text Restaurant List| C[Gemini-2.5-flash Extracts Top 3 Restaurants]
    C -->|Dynamically Generates Quick Reply| D[LINE Bot Replies with 3 Customized Analysis Buttons]
    D -->|User Clicks Specific Analysis| E[FastAPI Background Task]
    E -->|Immediate Reply ACK| F[LINE Chat Message]
    E -->|Package JSONL and Upload| G[Gemini Batch API Submission]
    G -->|Computation Complete Webhook/Polling Callback| H[Proactively Pushes Deep Report to User]

Core Implementation

1. Precisely Extracting Restaurant Names from Grounding Text using Gemini

In tools/maps_tool.py, the map search returns a plain text string rich in formatting and descriptions. We use Gemini-2.5-flash's structured output concept to precisely extract restaurant names in JSON format:

        # Extract top three restaurant names for Quick Reply
        names = []
        if place_type == "restaurant":
            try:
                extract_prompt = f"Please extract all restaurant names from the following text and return them in a JSON array format (e.g., [\"Restaurant A\", \"Restaurant B\"]). Please output the JSON array directly, without any markdown tags (like ```
{% endraw %}
json) or explanatory text.\n\n{result}"
                extract_res = client.models.generate_content(
                    model="gemini-2.5-flash",
                    contents=extract_prompt
                )
                extract_text = extract_res.text.strip() if extract_res.text else ""

                try:
                    names = json.loads(extract_text)
                except Exception:
                    import re
                    array_match = re.search(r"\[(.*?)\]", extract_text, re.DOTALL)
                    if array_match:
                        import ast
                        names = ast.literal_eval(f"[{array_match.group(1)}]")

                names = [str(n).strip() for n in names if n]
                logger.info(f"Extracted restaurant names for Quick Reply: {names}")
            except Exception as e_extract:
                logger.error(f"Failed to extract restaurant names: {e_extract}")
{% raw %}

2. Dynamically Generating LINE Quick Reply Buttons

In main.py, after obtaining the restaurant list, we dynamically generate QuickReplyButton. We need to pay special attention to LINE API's length limit for button label:


python
        quick_reply = None
        if place_type == "restaurant" and result.get("status") == "success":
            restaurant_names = result.get("restaurant_names", [])
            if restaurant_names:
                buttons = []
                for name in restaurant_names[:3]:
                    clean_label = name
                    # LINE label limit is 20 characters
                    if len(clean_label) > 10:
                        clean_label = clean_label[:9] + "…"
                    buttons.append(
                        QuickReplyButton(
                            action=PostbackAction(
                                label=f"🍴 分析 {clean_label}",
                                data=json.dumps({
                                    "action": "specific_foodie_deep_analysis",
                                    "restaurant_name": name
                                }),
                                display_text=f"🔍 進行「{name}」深度評論與招牌菜色分析"
                            )
                        )
                    )
                quick_reply = QuickReply(items=buttons)

Major Pitfalls and Solutions

During the process of connecting this dynamic Quick Reply to the Batch API, we encountered several critical UX and API limitation issues:

Pitfall One: LINE 20-character Limit Causing API Sending Errors

Initially, when implementing, we directly used the full restaurant name in the button's Label, for example: 🍴 Analyze Love Hot Pot Ultimate Hot Pot. As a result, the LINE API immediately returned a 400 error, and the message could not be sent at all:


plaintext
LineBotApiError: status_code=400, error_message=The property 'label' must be less than 20 characters.

[Cause Analysis and Solution] LINE's official label limit for Quick Reply is extremely strict; including emojis and spaces, it can have a maximum of 20 characters. To address this, we added a character count check and dynamic truncation mechanism in our code:

First, the original restaurant name (clean_label) is truncated: if its length exceeds 10 characters, it is forcibly cut to the first 9 characters and appended with "…" (occupying 10 characters).
Adding the prefix 🍴 Analyze (a total of 5 characters), the maximum total length becomes 15 characters, safely staying within the 20-character limit, thus eliminating the error!

Pitfall Two: Batch API Asynchronous Delay and LINE Webhook's "Three-Second Timeout Survival Battle"

When a user clicks the "Analyze Restaurant" button, the Bot must first call Google Search Grounding to collect online reviews for that restaurant, then package the JSONL file and upload it to Gemini to submit the Batch task. This entire sequence usually takes 3 to 8 seconds. However, the LINE Webhook server requires the Bot to return an HTTP 200 OK response within 3 seconds, otherwise it will be deemed a connection failure and re-send the request, leading to severe server congestion.

[Cause Analysis and Solution] We completely asynchronous the processing architecture:

Fast Response: When the Bot intercepts a specific_foodie_deep_analysis Postback action, it does not execute the analysis directly within the Request flow. Instead, it immediately calls LINE's reply_message to respond to the user: 🔍 Received! Performing deep analysis for you... This will take about 1-2 minutes..., and then instantly returns HTTP 200 to end that Webhook request.
Background Task Dispatch: Use Python asyncio.create_task to dispatch heavy network search, upload, and submission tasks to FastAPI's background Worker for execution.
Big Data Push: When the background Polling listener or Gemini Webhook receives a task completion notification, it then uses LINE's push_message to proactively send the analysis report to the specific user.

Pitfall Three: Gemini Batch API's Queuing and Pending Status

During testing, users sometimes got confused, "Why hasn't there been a reply after three minutes? Is the Bot down?". After checking the system logs, we found that our JSONL file had been successfully uploaded, but the task status on the Gemini server side was stuck at JobState.JOB_STATE_PENDING.

[Solution] This is a characteristic of the Batch API; tasks need to be queued, waiting for Google's server resources. We adopted two major optimizations:

Minimize Workload: Reduce the number of restaurants for batch analysis to 1, shrinking the number of request lines in the JSONL to the extreme, to speed up Gemini's scheduling and processing.
UX Optimization and Deduplication Mechanism: When a user clicks to analyze, we first check if that user already has a Batch Job running. If so, we reply: ⏳ Your deep analysis task is currently running, please wait patiently, preventing users from submitting multiple duplicate Batch Jobs due to anxious repeated clicks, which would consume unnecessary resources.

Results and Benefits

This optimization of Quick Reply and Gemini Batch API for the LINE Bot Restaurant Assistant has achieved excellent practical value:

Highly Customized Mobile Experience: After locating, users don't need to type; they can directly click on a restaurant of interest with one tap to precisely get a summary of its signature dishes and review pain points.
Robust Backend Architecture: By leveraging asynchronous background tasks and LINE's character limit safety valve, the risks of Webhook timeouts and LINE API errors have been completely resolved.
Cost Advantage for Big Data Processing: Through the Batch API's half-price advantage and Webhook's proactive callback, while ensuring user experience, it also saves significant computing resources and API costs for the server.

Through this architecture, the LINE Bot truly achieves a low-latency, highly stable big data deep analysis experience on mobile!

All development code for this project has been open-sourced on GitHub: kkdai/linebot-helper-python. Everyone is welcome to deploy and personally test this one-click analysis function, which we believe can bring a higher level of intelligent experience to your LINE Bot projects!

[I/O Extended Taipei] Building with Gemini APIs: From Calls to Autonomous Systems

Evan Lin — Sun, 14 Jun 2026 07:12:11 +0000

(Activity: Google I/O Extended 2026 Taipei / Presentation: SpeakerDeck)

Context: The Gemini API is no longer just "adding one more prompt"

If your impression of the Gemini API is still limited to "select a model, send a prompt, get back a piece of text," then when you see this round of updates in 2026, you'll likely suddenly realize something:

The Gemini API has evolved from a simple API interface into a complete platform that can be used to build applications, agents, and asynchronous workflows.

This content is compiled from my talk "Building Applications in the Gemini API Family" at Google I/O Extended 2026 Taipei. Evan Lin, Technical Director of LINE Taiwan Developer Relations, repeatedly emphasized a core observation at the event: what developers truly need to consider now is no longer just "Should I use Pro or Flash?", but rather "How do I string together models, retrieval, agents, callbacks, and cost control into a cohesive system?".

In other words, the focus is shifting from calling APIs to designing systems.

First, let's look at the big picture: What's new in the 2026 Gemini API family?

If we view the 2026 Gemini API as a capability map, it can broadly be divided into three layers.

Layer 1: Core Models

Gemini 3.5 Pro: Strongest reasoning capability, suitable for complex planning, advanced analysis, and multi-step tasks.
Gemini 3.5 Flash: Main model, best balance of speed, cost, and capability, suitable for most product traffic.
Flash-Lite: Intent classifier and pre-classifier for high-frequency, low-cost scenarios.
Gemini Embedding 2: Supports not only text but also multi-modal vectorization needs.

Layer 2: Key Capability Modules

Retrieval: File Search, Google Search Grounding, URL Context.
Agent / Async: Agents API, Webhook, Deep Research agent.
Infrastructure: Context caching, Batch API, Live API.

Layer 3: System Design Approach

This layer is arguably the most important. Because once the above capabilities are offered as platform services, many "intermediate layers" that previously had to be built manually suddenly disappear:

No longer necessarily need to build your own RAG pipeline.
No longer necessarily need to maintain your own agent loop.
No longer necessarily need to block the main server with polling while waiting for results.

Core Observation: The Gemini API upgrade is not just about "stronger models"; it's about Google absorbing the complexities that were originally at the application layer into the platform layer. This will directly change how we design AI systems.

Architectural Turning Point: Three Tools, Three Paradigm Shifts

What's most worth repeatedly digesting from this talk are the architectural changes represented by these three tools.

1. File Search: Shifting from Hand-Coded RAG to Managed RAG

Previously, when discussing enterprise knowledge Q&A, the immediate thought was:

Chunking.
Creating embeddings.
Storing in a vector DB.
Writing retrieval code.
Then manually adding citation and permission control.

Now, with the advent of File Search, developers can focus more on "how documents are governed, how permissions are allocated, and how answers are presented," rather than repeatedly writing that foundational infrastructure.

More importantly, it doesn't just search text.

Why is this File Search particularly noteworthy?

Images and text in the same space: Screenshots, charts, and mixed text-image layouts in PDFs are no longer just attachments, but content understandable by the model.
Metadata filtering: Can filter by department, system, and document type, which is crucial for internal enterprise knowledge retrieval.
Precise citation: Can refer back to specific page numbers and grounding metadata, making answers more trustworthy.

This represents a very practical shift: much of the time enterprises previously spent on LangChain, vector databases, and chunking strategies can now largely be redirected towards permission design, UX, and content governance.

2. Agents API: Shifting from Client-Side Loop to Server-Side Managed Agent

In the past, to build an agent, the common approach was to maintain your own ReAct or tool loop:

Model decides the next step.
Calls a tool.
Receives results.
Feeds back to the model.
Repeats until completion.

The problem is that this is full of engineering details: state preservation, timeouts, retries, background execution, long-task monitoring. Ultimately, you'd find yourself spending most of your time maintaining an "agent runtime."

What the Agents API changes is that you can POST a task to Gemini, allowing it to complete the long process on the server side, even handling complex tasks that take up to 20 minutes.

The significance behind this is not just "more convenient"; it means developers can finally refocus on:

How are tasks defined?
Which tools can be used?
What are the success criteria?
How should the product integrate the results when they return?

3. Webhook: Shifting from Polling to Event-Driven

Once tasks might run for several minutes, or even more than ten minutes, traditional synchronous requests become unreasonable.

Therefore, the role of Webhook is actually crucial: it's not a minor feature, but a prerequisite for the entire agent workflow to truly enter production. When Gemini completes a task and actively POSTs the result back to your server, your system can become event-driven:

The frontend first responds to the user with "Task received."
The Agents API executes in the background.
Upon completion, the result is pushed back via webhook.
Your service then notifies the user, updates the database, or triggers the next step.

This is particularly important for high-concurrency products, as you finally don't need to hold a bunch of server connections idly waiting.

From the Perspective of a LINE Bot, How Should a Gemini Application Be Designed?

A very practical suggestion Evan gave in his talk is to place a router layer before the LLM.

This design sounds simple, but it largely determines your cost, latency, and predictability.

A Very Pragmatic Routing Approach

First, use the inexpensive Flash-Lite for intent routing:

Quick Q&A: Directly handed over to Flash or Flash-Lite for generation.
Query company documents: Enters File Search.
Complex long tasks: Enters Agents API.

Doing this has three benefits:

Cost control first: Not every query directly hits the most expensive, heaviest model.
Latency control first: Simple requests should not mistakenly enter long processes.
System behavior control first: Makes the overall process more stable than "throwing everything at a large model for improvisation."

If you're building a LINE Bot, customer service assistant, internal knowledge assistant, or workflow agent, this router should almost certainly be the default configuration, rather than an afterthought.

Infrastructure is Not Unimportant, But You Don't Have to Rebuild it Yourself Every Time

Another strong message from this talk is that developers' time should be reallocated.

Previously, much of the man-hours in many AI projects were actually consumed by these tasks:

Vector database operations and maintenance
Chunking and retrieval parameter tuning
Long-task scheduling
Websocket / polling / callback processes
Token cost optimization

Now, with File Search, Agents API, Webhook, Context caching, and Batch API, the areas where we should spend more time have shifted to:

Business rules and tool boundaries
Document permissions and data governance
User interaction experience
Task decomposition and routing strategies
Failure recovery and result interpretability

This is also why I strongly agree with Evan's underlying message: What's truly valuable is not whether you can build your own vector database, but whether you can redirect 80% of your energy back to the product's core.

Three Most Valuable Practical Takeaways

1. Place a routing layer before the LLM

Don't send all problems directly to the same model. First classify, then decide whether to generate, retrieve, or enter an agent task.

2. Embrace asynchronous operations; don't force long tasks into synchronous APIs

If a task might take more than a few seconds, you should seriously consider Agents API + Webhook. This is not an optimization; it's an architectural correctness issue.

3. Redirect RAG engineering time to permissions and experience

When File Search can handle a large amount of foundational work, developers should be more concerned with: can data be securely queried, can answers be verified, and can citations be trusted by users.

Why is this talk worth revisiting repeatedly?

Because it highlights a turning point that many teams are currently facing:

We are no longer just writing prompts for LLMs; we are designing operating systems for AI applications.

Models are certainly still at the core, but what truly differentiates products is increasingly not "which model you choose," but:

How you decide when to use which capability.
How you make the system run reliably for extended periods.
How you make answers traceable, verifiable, and maintainable.

If you still understand generative AI using the 2024 approach of "a single chat endpoint for everything," then you'll easily underestimate the 2026 Gemini API family.

Postscript: From API User to AI System Designer

The most valuable aspect of this "Building Applications in the Gemini API Family" talk is not teaching you another new parameter or SDK, but reminding everyone of a more fundamental shift:

The competitiveness of the next phase will not be about who is better at calling models, but who is better at assembling models, retrieval, agents, and event flows into a truly functional system.

If you are working on a LINE Bot, enterprise knowledge base, internal assistant, customer service process, or any product requiring multi-step AI collaboration, this architectural perspective is well worth using to redraw your current system diagram.

Often, what truly needs refactoring is not the prompt, but the entire pipeline.

[Hands-on Gemini 3.5 Live

Evan Lin — Fri, 12 Jun 2026 06:09:59 +0000

Brand New API Unveiled: Gemini 3.5 Live Translate

On June 9, 2026, Google officially released its brand new real-time voice translation model — Gemini 3.5 Live Translate. This marks another significant breakthrough for Google in AI voice translation technology. It is currently available for public preview to developers in Google AI Studio and Gemini Live API, and has been simultaneously integrated into services like Google Translate and Google Meet.

Key features of Gemini 3.5 Live Translate include:

Fluent and Natural Bidirectional Voice Translation: Supports over 70 languages, automatically detecting the input voice language without manual configuration.
Continuous Stream Generation (Instead of Single-Sentence Turn-Taking): Unlike previous turn-by-turn systems that required the speaker to finish speaking before translation, Gemini 3.5 Live Translate generates translations in real-time while listening. It strikes a balance between contextual understanding and immediacy, with translations lagging only a few seconds behind the speaker, completely avoiding awkward pauses.
Preservation of Intonation and Rhythm: The generated voice is not only smooth but also retains the original speaker's tone, intonation, and speaking rhythm.
Robust Noise Cancellation Capability: Accurately captures and recognizes speech even in noisy or unstable environments.

This article will document how we developed a native macOS application, MeetingTranslator, using Swift, to integrate with this powerful new API and achieve real-time translation of specific app audio into Traditional Chinese voice and subtitles.

System Design and Architecture

Our goal is to develop a Native SwiftUI application that does not require installing virtual sound cards like BlackHole. Instead, it utilizes Apple's official ScreenCaptureKit framework to directly capture the audio stream from a selected application (such as YouTube in Google Chrome or an online meeting) and, through the Gemini Live WebSocket API, achieve ultra-low-latency conversational voice translation.

System Architecture Flow

graph TD
    A[ScreenCaptureKit <br>Capture Application Audio] -->|48kHz Stereo Float32| B[AVAudioConverter <br>Resampling and Channel Conversion]
    B -->|16kHz Mono Int16 PCM| C[Gemini Live API <br>WebSocket Connection]
    C -->|Real-time Subtitle Recognition| D[SwiftUI Subtitle HUD <br>Traditional Chinese Bilingual Subtitles]
    C -->|24kHz Mono Int16 PCM Translated Audio| E[AudioPlaybackManager <br>AVAudioEngine Player]

Core Implementation One: ScreenCaptureKit Capture and Resampling

ScreenCaptureKit, introduced in macOS 13, frees developers from the pain of relying on kernel audio virtual devices, allowing precise filtering and recording of specific application screens and audio.

1. Filter and Select Target App

We use SCShareableContent to get currently running applications on the system and filter out background services without names and system-自带 services:

func fetchShareableApps() async -> [SCRunningApplication] {
    do {
        let content = try await SCShareableContent.current
        return content.applications.filter { app in
            let name = app.applicationName
            guard !name.isEmpty else { return false }
            let bundleId = app.bundleIdentifier
            return !bundleId.hasPrefix("com.apple.system") && bundleId != Bundle.main.bundleIdentifier
        }.sorted { $0.applicationName < $1.applicationName }
    } catch {
        print("無法獲取可共享內容: \(error)")
        return []
    }
}

2. Start Audio Capture Stream

After filtering out the target App (e.g., Google Chrome), we create an SCContentFilter for it and apply it to SCStream:

let appFilter = SCContentFilter(display: content.displays.first!, including: [targetApp], exceptingWindows: [])
let config = SCStreamConfiguration()
config.capturesAudio = true
config.width = 32 // When only capturing audio, set video frame to minimal to save performance
config.height = 32

stream = SCStream(filter: appFilter, configuration: config, delegate: nil)
try stream?.addStreamOutput(self, type: .audio, sampleHandlerQueue: DispatchQueue(label: "com.translator.audioQueue"))
try await stream?.startCapture()

Core Implementation Two: Gemini Live WebSocket Bidirectional Connection

The core of the Gemini Live API lies in using a wss:// connection to transmit microphone/application audio in real-time through a single channel, and simultaneously receive model-generated translated text and translated audio.

In GeminiLiveConnection.swift, we maintain this bidirectional pipeline via URLSessionWebSocketTask. After connecting, a setup control message must be sent immediately to initialize the model configuration.

Major Pitfalls and Solutions

During the process of integrating the system, we encountered three blocking difficulties. Below is our troubleshooting process and solutions:

Pitfall One: Gemini Live Exclusive Model Restrictions

Initially, we tried to use standard REST API model names (e.g., gemini-3.5-flash) in the WebSocket connection, but the server immediately disconnected:

❌ WebSocket 被 Gemini 伺服器關閉 (CloseCode: 1008, 原因: models/gemini-3.5-flash is not found for API version v1beta, or is not supported for bidiGenerateContent.)

【Solution】 Gemini's bidirectional Live API currently only supports specific optimized real-time models. We must restrict the model field to:

gemini-2.0-flash-exp (standard bidirectional conversation)
gemini-3.5-live-translate-preview (preview model optimized for real-time translation)

Pitfall Two: Incorrect JSON Payload Field Structure (Hidden Differences Between Documentation and API Versions)

When configuring real-time interpretation, we referred to Google's official documentation and placed the inputAudioTranscription (input speech-to-text) and outputAudioTranscription (output speech-to-text) fields within generationConfig, which resulted in a 1007 error:

❌ WebSocket 被 Gemini 伺服器關閉 (CloseCode: 1007, 原因: Invalid JSON payload received. Unknown name "inputAudioTranscription" at 'setup.generation_config': Cannot find field.)

【Cause Analysis and Solution】 In the official documentation, for v1alpha and client SDKs (e.g., JavaScript / Python SDK), these two fields are wrapped within generationConfig. However, in the current v1beta WebSocket native endpoint: /ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent

These two fields should be located at the root level of the setup object, while the translation-specific translationConfig must be placed under generationConfig. The correct JSON Payload structure is as follows:

setupMessage = [
    "setup": [
        "model": "models/\(modelName)",
        "inputAudioTranscription": [:], // Enable real-time input subtitles, placed at the setup root
        "outputAudioTranscription": [:], // Enable real-time output subtitles, placed at the setup root
        "generationConfig": [
            "responseModalities": ["AUDIO"],
            "translationConfig": [
                "targetLanguageCode": "zh-TW", // Set target translation language to Traditional Chinese
                "echoTargetLanguage": true
            ]
        ]
    ]
]

After this modification, the WebSocket setup finally successfully handshaked and no longer crashed!

Pitfall Three: "Zero-Byte Silence" Caused by Multi-Channel Stereo Capture

After successfully establishing the WebSocket pipeline and starting to push resampled audio, we found that Gemini still had no translation response. Observing the log output, we discovered that the content of the sent audio blocks was all 0 (Silence):

📊 [WebSocket] 已發送 500 個音訊區塊 | 大小: 640 bytes | 是否為靜音(全0): true

【Cause Analysis】 When the captured object (e.g., Google Chrome playing a YouTube video) outputs stereo (2 Channels) or multi-channel audio, our original method for converting CMSampleBuffer to AVAudioPCMBuffer:

// Old method: Directly assumes a single Channel pointer and copies
var audioBufferList = AudioBufferList()
var blockBuffer: CMBlockBuffer?
CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(..., &audioBufferList, ...)

In a multi-channel environment, this would lead to insufficient memory allocation, causing copy interruption or fill failure, resulting in all subsequent audio resampler (AVAudioConverter) inputs being null values (silence).

【Solution】 It is necessary to use the Double-Call technique to dynamically allocate memory space for AudioBufferList:

First Call: Pass nil as the buffer output, used only to precisely query the required physical memory size (bufferListSizeNeededOut) for that sampleBuffer.
Memory Allocation: Use UnsafeMutablePointer<AudioBufferList>.allocate to dynamically allocate space based on the queried size.
Second Call: Pass the allocated pointer to safely fill in multi-channel audio data.
Channel Reassembly: Based on the multi-channel format (Interleaved/Non-Interleaved), precisely use memcpy to copy the corresponding data segments into a temporary buffer, then send it to the converter for noise reduction and downsampling.

Core code correction:

private func audioBufferFromSampleBuffer(_ sampleBuffer: CMSampleBuffer, asbd: AudioStreamBasicDescription) -> AVAudioPCMBuffer? {
    guard let sourceFormat = sourceFormat else { return nil }

    // 1. Dynamically get the required AudioBufferList memory size
    var bufferListSize = 0
    var status = CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(
        sampleBuffer,
        bufferListSizeNeededOut: &bufferListSize,
        bufferListOut: nil,
        bufferListSize: 0,
        blockBufferAllocator: nil,
        blockBufferMemoryAllocator: nil,
        flags: 0,
        blockBufferOut: nil
    )

    guard status == noErr else { return nil }

    // 2. Allocate a pointer with sufficient space and fill it
    let bufferListPointer = UnsafeMutablePointer<AudioBufferList>.allocate(capacity: bufferListSize)
    defer { bufferListPointer.deallocate() }

    var blockBuffer: CMBlockBuffer?
    status = CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(
        sampleBuffer,
        bufferListSizeNeededOut: nil,
        bufferListOut: bufferListPointer,
        bufferListSize: bufferListSize,
        blockBufferAllocator: nil,
        blockBufferMemoryAllocator: nil,
        flags: 0,
        blockBufferOut: &blockBuffer
    )

    guard status == noErr else { return nil }

    // 3. Create an AVAudioPCMBuffer conforming to the source format and safely copy...
    let frameCount = AVAudioFrameCount(CMSampleBufferGetNumSamples(sampleBuffer))
    guard let pcmBuffer = AVAudioPCMBuffer(pcmFormat: sourceFormat, frameCapacity: frameCount) else { return nil }
    pcmBuffer.frameLength = frameCount

    let audioBuffers = UnsafeMutableAudioBufferListPointer(bufferListPointer)
    for (index, audioBuffer) in audioBuffers.enumerated() {
        guard let mData = audioBuffer.mData, index < Int(sourceFormat.channelCount) else { continue }
        // Differentiate between non-interleaved and interleaved formats for copying
        let isNonInterleaved = asbd.mFormatFlags & kAudioFormatFlagIsNonInterleaved != 0
        if isNonInterleaved {
            if let dst = pcmBuffer.int16ChannelData?[index] {
                memcpy(dst, mData, Int(audioBuffer.mDataByteSize))
            }
        } else {
            if let dst = pcmBuffer.int16ChannelData?[0] {
                let offset = index * Int(frameCount)
                memcpy(dst.advanced(by: offset), mData, Int(audioBuffer.mDataByteSize))
            }
        }
    }
    return pcmBuffer
}

After applying this refactoring, when we played a test video on Chrome's YouTube again, the console finally printed: 是否為靜音(全0): false, and we successfully received Gemini's real-time voice feedback!

Results and Benefits

Full development repo: https://github.com/kkdai/gemini-live-translate-macos

Through this architectural upgrade and bug fixes, MeetingTranslator has demonstrated excellent practical value:

Zero External Device Dependency: No need to set up complex routing like BlackHole or Loopback; it works out of the box.
Accurate and Real-time Subtitles: The Gemini Live API can complete English to Traditional Chinese translation within hundreds of milliseconds, smoothly displaying the results in a HUD floating window.
Synchronized Voice Translation Broadcast: Through AudioPlaybackManager, users can listen to the original meeting while simultaneously hearing high-quality 24kHz Traditional Chinese interpretation in their headphones.

We hope this record of pitfalls encountered with macOS Core Audio / ScreenCaptureKit and the Gemini WebSocket API can provide valuable reference for developers also exploring AI real-time voice applications!

[AI Practice] Building blazing-Fast AI Mac OS App with Antigravity CLI

Evan Lin — Fri, 12 Jun 2026 06:09:40 +0000

Foreword: A Developer's New Collaboration Model

Imagine this scenario: you are developing a real-time meeting translation App that combines macOS low-level audio (CoreAudio/ScreenCaptureKit) with Gemini Live API WebSocket. During the testing phase, the program suddenly crashed with an error, and the audio stream produced a complete silence of all zeros.

In the past, your troubleshooting process might have been:

Open the terminal and retrieve the log file.
Copy the entire error message and relevant code.
Switch to the browser, open an AI chat window, paste it, and ask for the reason.
After receiving modification suggestions, copy them back to the editor and test manually.
Repeat the above steps until fixed, then manually write README.md, write a blog post, create a GitHub repository, commit the code, and push it.

In this development cycle, we adopted the AGY CLI (Antigravity-CLI) agent designed by Google DeepMind. We were surprised to find that all the tedious context switching mentioned above could be fully automated through conversations with the intelligent agent within the terminal. This article will reconstruct the actual Prompt dialogue flow and share how we collaborated with AGY CLI to build a macOS meeting translation App from scratch.

Phase One: Idea Generation and Architecture Design

Everything originated from a development idea and a newly released Google API document. The developer pasted a URL into the terminal for AGY CLI and posed the first core question:

User : Following this example, is it possible to create a Mac OS App that can capture audio from the computer and translate it into other languages in real-time? Start helping me think about how to do this. My main goal is to be able to use it for Zoom / Google Meet meetings to directly translate other countries' languages into Traditional Chinese.

Here are other documents for your reference: https://ai.google.dev/gemini-api/docs/live-api/live-translate?hl=zh-tw

Upon receiving the instruction, AGY CLI acted as an architect, quickly analyzing and providing a set of native macOS solutions:

No virtual sound card installation required: It recommends using Apple's ScreenCaptureKit framework, introduced in macOS 13+, to directly capture the clean audio track of selected applications (such as Zoom or Chrome) in Sandbox secure mode.
Real-time audio resampling: Utilize AVAudioConverter to resample multi-channel/stereo 48kHz audio in real-time to the 16kHz mono PCM format supported by Gemini Live.
Bidirectional WebSocket communication: Implement a bidirectional long connection using Swift's URLSessionWebSocketTask to send PCM audio chunks while receiving Traditional Chinese translation results and broadcast audio.

Phase Two: Environment Configuration and Compilation Anxiety Elimination

Before starting to write Swift code, the developer raised questions about the macOS compilation environment:

User : Do I absolutely need Xcode to compile?

User : Help me run xcode-select –install

Addressing the developer's environment anxiety, AGY CLI provided the answers:

Although it's best to use Xcode to create standard macOS SwiftUI GUI application bundles, we don't necessarily need to manually open the Xcode IDE.
AGY CLI helped write an automated build script build_app.sh, which directly calls the terminal's swiftc command-line tool to compile all .swift source files and automatically packages them into a complete .app directory structure.
At the same time, in response to the developer's request to install Command Line Tools, AGY CLI proactively invoked the permission tool and directly ran xcode-select --install locally, automatically configuring the Swift compilation environment.

Phase Three: Connection Troubleshooting and Audio Bug Fixes

After the code was initially completed, the developer ran the App from the command line, but the connection status showed abnormalities, and no characters were translated:

User : Didn't see any error messages~ but the connection status is disconnected

This was the moment for AGY CLI to demonstrate its "autonomous troubleshooting" power. Upon receiving the prompt, it automatically located the debug.log file, called tail to analyze the runtime logs, and identified two critical issues:

Incompatible model name: The original program used the standard REST model models/gemini-3.5-flash, whereas the Live WebSocket API only accepts gemini-3.5-live-translate-preview.
Incorrect JSON configuration level: The API documentation used the v1alpha version SDK, which wrapped inputAudioTranscription within generationConfig; however, the native WebSocket's v1beta endpoint required these two fields to be placed directly under the setup root directory. This was the culprit behind the CloseCode 1007 crash.
Multi-channel stereo silence Bug: The multi-channel audio track captured by ScreenCaptureKit was truncated to complete silence (all zeros) during copying in the old code due to insufficient AudioBufferList memory allocation.

AGY CLI immediately proactively modified AudioCaptureManager.swift, introducing the "Double-Call" register allocation pointer technique, and refactored the Payload structure of GeminiLiveConnection.swift.

After the modifications were completed, the application ran smoothly, the console log finally printed 是否為靜音(全0): false (Is it silent (all 0s): false), and both real-time bilingual subtitles and real-time broadcast audio functioned correctly!

Phase Four: Automated DevOps and GitHub Delivery

Once the developer confirmed that the program was working correctly, the final step was to open-source and share the code:

User : I want to check in the swift-demo folder to my own GitHub repo. Give me a suggested repo name and write a README.md under swift-demo.

User : Help me commit all relevant changes in that folder to git@github.com:kkdai/gemini-live-translate-macos.git

AGY CLI immediately took over the final DevOps tasks:

It recommended using gemini-live-translate-macos as the Repo name and wrote the project's English GitHub description and topics tags.
It automatically completed the full environment preparation, Xcode Sandbox Capabilities settings, command-line script execution steps, and API troubleshooting tips in README.md.
After obtaining the user's repository URL, AGY CLI proactively ran git init in the background, wrote .gitignore, committed all the code, and successfully pushed it to the remote GitHub repository!

Conclusion: Development Transformation and Insights

Through this collaborative development with AGY CLI, we experienced an unprecedentedly rapid development process:

Reduced cognitive load: Developers only need to express their intentions in natural language (e.g., "help me run the installation," "help me troubleshoot why the connection is broken"), and the AI Agent will autonomously translate them into corresponding system commands and code modifications.
Native system-level control: AI can directly read and execute commands, synchronizing with the development environment in real-time, greatly reducing the hallucinations and environment version mismatches that often occurred with traditional Web AI Chat.
One-stop delivery: From the first phrase "think about how to do it" to the final "Push to GitHub repository" with a single click, AGY CLI seamlessly integrated the entire software engineering lifecycle.

This practical experience proves that in the era of Agentic AI, a single developer, paired with a powerful CLI agent, can deliver a high-quality Native application involving system-level foundations and the latest APIs in an extremely short amount of time. See you next time!

[GCP Practical] LINE Business Card Bot

Evan Lin — Sun, 07 Jun 2026 15:26:28 +0000

Upgrade Preamble

After refactoring the agent based on Vertex AI ADK, our LINE Name Card Assistant Bot (linebot-namecard-python) entered the production environment for testing. However, in real-world usage scenarios, we quickly identified three core pain points affecting user experience and security:

Unstable OCR JSON Parsing: Using the standard JSON Mode with a Prompt, Gemini occasionally still outputs Markdown tags or misses fields, causing parser errors.
Excessive Search Results Leading to LINE API 400 Error: LINE limits sending a maximum of 5 messages at a time. When search results include 5 cards plus the Agent's text reply, totaling 6, LINE directly rejects it and doesn't reply.
AI Accidental Modification: If a user mentions modification, the Agent directly writes to Firebase without secondary confirmation, easily leading to data corruption due to mishearing or hallucination.

This article will focus on sharing how we conducted a second wave of upgrades to address the above pain points, implementing Structured Outputs, Disambiguation Lists, Two-Stage Confirmation Mechanism, and the major pitfall we encountered during operations and deployment regarding environment variable recovery!

Optimization One: Embracing Gemini Structured Outputs

Previously, when calling gemini-3-flash-preview for name card image parsing, we commanded it via Prompt and manually parsed JSON. To ensure 100% format guarantee, we introduced the native Structured Outputs feature of the Vertex AI API.

1. Defining the Name Card Schema

In app/gemini_utils.py, we defined the constraint Schema for the name card object, forcing Gemini to strictly adhere to this format for output:

NAMECARD_SCHEMA = {
    "type": "OBJECT",
    "properties": {
        "name": {
            "type": "STRING",
            "description": "聯絡人姓名，如果看不出來，請填寫 N/A"
        },
        "title": {
            "type": "STRING",
            "description": "職稱或頭銜，如果看不出來，請填寫 N/A"
        },
        "company": {
            "type": "STRING",
            "description": "公司名稱，如果看不出來，請填寫 N/A"
        },
        "address": {
            "type": "STRING",
            "description": "公司或聯絡地址，如果看不出來，請填寫 N/A"
        },
        "phone": {
            "type": "STRING",
            "description": (
                "電話號碼，格式為 #886-0123-456-789,1234。"
                "沒有分機就忽略 ,1234。如果看不出來，請填寫 N/A"
            )
        },
        "email": {
            "type": "STRING",
            "description": "電子郵件信箱，如果看不出來，請填寫 N/A"
        }
    },
    "required": ["name", "title", "company", "address", "phone", "email"]
}

2. Applying to Generation Config

We only need to specify response_schema in generation_config when instantiating GenerativeModel:

def generate_json_from_image(img: PIL.Image.Image, prompt: str) -> object:
    model = GenerativeModel(
        "gemini-3-flash-preview",
        generation_config={
            "response_mime_type": "application/json",
            "response_schema": NAMECARD_SCHEMA
        },
    )
    img_part = Part.from_data(data=pil_to_bytes(img), mime_type="image/jpeg")
    response = model.generate_content([prompt, img_part], stream=False)
    return response

After application, the JSON error rate of the returned response dropped directly to 0%, eliminating complex string cleaning and parser error-prevention logic.

Optimization Two: Solving LINE Message Limit with 'Disambiguation List'

LINE Webhook has an iron rule: the number of message bubbles sent in a single reply_message must be between 1 and 5. If the search results happen to be 5 or more, and a text reply is added, the total will exceed 5, triggering a LINE API 400 error.

💡 Solution: Disambiguation List

We modified the search reply judgment in app/line_handlers.py:

When search results are 1 to 4 items: Directly display Carousel detailed name cards (conforming to LINE's 5-item limit).
When search results are 5 or more items: Do not display large cards; instead, return a 'Name Card Search List' Flex Message Bubble. The list itemizes names and companies, with a 'View ❯' Postback button on the right. Clicking it loads and displays that specific name card.

This design not only maintains a clean layout but also completely avoids the pitfall of exceeding the message limit!

        elif found_card_ids:
            if len(found_card_ids) <= 4:
                # If the quantity is less than or equal to 4, directly display Carousel detailed name cards
                for card_id in found_card_ids:
                    card_data = firebase_utils.get_card_by_id(user_id, card_id)
                    if card_data:
                        reply_msgs.append(
                            flex_messages.get_namecard_flex_msg(card_data, card_id)
                        )
            else:
                # If the quantity is greater than 4, display as a list Flex Message for disambiguation
                cards_list = []
                for card_id in found_card_ids:
                    card_data = firebase_utils.get_card_by_id(user_id, card_id)
                    if card_data:
                        cards_list.append({
                            "card_id": card_id,
                            "name": card_data.get("name", "N/A"),
                            "company": card_data.get("company", "N/A"),
                            "title": card_data.get("title", "N/A")
                        })
                if cards_list:
                    list_msg = flex_messages.get_namecard_list_flex_msg(
                        cards=cards_list,
                        title_text="🔍 Found multiple matching name cards"
                    )
                    reply_msgs.append(list_msg)

Optimization Three: Contact Modification Safety Lock — Two-Stage Confirmation Mechanism

Under the ADK agent architecture, users can update data through natural conversation (e.g., "Add 'Meeting next Monday' to Evan's memo"). However, if the LLM misinterprets the instruction, Firebase data can be directly overwritten.

To address this, we implemented a Two-Stage Confirmation mechanism:

Delayed Write: When the ADK Tool (update_namecard_field and update_namecard_memo) is invoked by the model, the system does not directly rewrite Firebase. Instead, it temporarily stores the content to be modified in user_states in memory and returns True to allow the Agent to continue generating dialogue.
Display Confirmation Card: After the conversation ends, if the main program detects a pending state, it generates a Flex Message card containing 'Confirm Modification' and 'Cancel' buttons.
Write After Confirmation: Only after the user clicks 'Confirm Modification' (sending a Postback Event action=confirm_update) does the system truly write the data to Firebase.

This not only perfectly prevents AI from accidentally triggering tools but also gives users absolute control when modifying data!

    # Handle confirmation of modification in handle_postback_event
    elif action == 'confirm_update':
        state = user_states.get(user_id, {})
        if state.get('action') == 'pending_update':
            update_type = state.get('update_type')
            card_id = state.get('card_id')
            # Read data from temporary storage based on update_type, and truly write to Firebase...
            if success:
                # Reply with successful modification, and automatically display the updated Flex Card for user verification

Ops Pitfall Record: Manual Deployment - The Mysterious Disappearance of Environment Variables

In addition to code refactoring, we also encountered a significant operational pitfall during deployment.

The Pitfall

When we attempted to upload a local folder to Cloud Run using the MCP deployment tool locally, because the command did not include environment variable declaration parameters, the previously working LINE Token and Firebase URL on Cloud Run were all cleared and overwritten. Upon restart, the Container crashed directly with an error:

Specify ChannelSecret as environment variable.

The online service instantly became paralyzed.

Recovery Process

Fortunately, Cloud Run fully retains the configuration settings of older versions. We can use the gcloud command to view previous Revisions and restore the lost variables:

Retrieve the detailed configuration of the last successfully running Revision:

gcloud run revisions describe linebot-namecard-python-00096-d89 --project=line-vertex --region=asia-east1

This will output the environment variable values bound to that version.

Re-inject environment variables into the service:

gcloud run services update linebot-namecard-python --project=line-vertex --region=asia-east1 --set-env-vars="ChannelAccessToken=...,ChannelSecret=..."

By restoring the variables, we seamlessly recovered the service within minutes. This also reminds us: when manually deploying to Cloud Run, always pay extra attention to the inheritance or declaration of environment variables to avoid accidentally clearing the official cloud configuration.

Summary and Benefits

This optimization brought excellent production-level transformations to our LINE Name Card Bot:

100% Format Security: Through API native Schema enforcement, the name card recognition format error rate dropped to 0%.
Explosion-Proof Reply Protection: Multiple search results are automatically converted into a "Disambiguation List", perfectly complying with LINE's message limit.
Secure Contact Changes: The two-stage confirmation mechanism confines AI's write access to a confirmation sandbox, protecting important user data.
Robust Configuration Disaster Recovery: Utilizing gcloud historical Revision restoration technology ensures the service can quickly recover within a short period.

The complete and linter-optimized code has been pushed to GitHub. We hope this practical experience helps everyone avoid detours when building production-grade AI Agents! See you next time!

DEV Community: Evan Lin

[Book Sharing] Tsaisang’s Tales of the Strange: Japanese Mythology, Ghost Stories, and Sometimes Taiwan

Recommended links to buy the book:

Foreword:

Synopsis

Summary of the Three Core Sections

I. Japanese Mythology: A First Family More Dramatic Than a Soap Opera

II. Yokai and Urban Legends: Collective Anxiety of Modern People

III. And Sometimes Taiwan: The Mysterious Connection Between Taiwanese and Japanese Gods and Ghosts

Reflections

[Book Sharing] Taiwan's AI Future: Analyzing Trends, Local Landscape, Corporate Strategy, and Personal Development

Recommended links to buy the book:

Preface:

Outline

Overview of the Book's Four Core Frameworks

I. Latest AI Trends: The 1:99 "Superhuman" Challenge

II. Taiwan's Situation: From "Island Involution" to the "Age of Discovery"

III. Corporate Layout: Hardware-Software Integration, Letting "Old-ventures + New-ventures" Dance Together

IV. Personal Development: Refuse "Brain Outsourcing," Be a High-Level "Questioner"

Personal Thoughts

[GCP in Action] LINE Business Card Bot Evolution: Dual-Side Recognition and Merging with Gemini

Pain Point: One Chinese Business Card is Actually Two Business Cards

Solution: First Ask, "Is there a back side?"

Core: Let Gemini See Two Images at Once and Merge Them

Two Easily Overlooked Pitfalls

Pitfall 1: Timing of Duplicate Checks

Pitfall 2: Don't Clear All States Indiscriminately

Incidental Resource Cleanup: Don't Let Back-Side Images Linger in Memory

Summary and Benefits

[Digital Certificate Wallet] Advanced: Building "Visitor-Endorsed Issuance" – A Full-Chain DID Application as Both Verifier and

Premise:

Scenario: Employee Access Control + Visitor Endorsement and Issuance

Architecture Decisions: Why not just modify the previous project?

Pitfall Record 1: Presentation successful, but the screen is stuck

Pitfall Record 2: Visitor card stuck at "Pending Issuance"

Pitfall Record 3: Collection QR too small + Mobile layout

The Truth About "Automatic Expiration"

Deployment: Directly to Cloud Run from Source Code

Summary and Future Outlook

[GCP Billing & Vertex AI] Solving Gemini Cost Allocation in a Single Project: Vertex AI Dynamic Billing Labels in Action

Pain Point: How to Accurately Allocate Gemini API Costs Within the Same Project?

A Workaround: Vertex AI "Request Labels" to the Rescue

Principle and Workflow

Project Implementation: Full Adoption of the Labels Mechanism

1. Project API Call Audit

2. Implementation Method

Scenario A: Already contains GenerateContentConfig

Scenario B: No Config parameter

3. List of Modified Files

Pitfall Guide: Watch Out for SDK Module Import Issues

Summary and Next Steps

[AI in Action] Refining a macOS Meeting Translation App with Claude Code: Auto-reconnect, Floating Captions, and Meeting Minutes Export Evolution

Foreword: Round Two, Switching to a Sharper Tool

Phase 1: Uncovering Hidden Dangers — The Mystery of the WebSocket Stopping After 10 Minutes

Phase 2: Feature Interview — Five Questions to Lock Down Two Priority Development Directions

Phase 3: Plan-Driven Development — Subagent Closed-Loop Delivery, Review Catches Critical Bugs

Task 1: Automatic Meeting Transcript Export

Task 2: Floating Subtitle Window

Phase 4: App Brand Upgrade — Real-time Generation of Professional Icons with Python

Phase 5: Code Quality Refinement — Clearing All Compilation Warnings

Phase 6: Real-World Pitfalls — The ScreenCaptureKit Permission Labyrinth

First Cut: Handling Silent Failures

Second Cut: Overly Aggressive Error Classification Logic

Third Cut: Finding the Root Cause — TCC Identity Mismatch

Permanent Countermeasure: Writing the Reset into the Build Process

Conclusion: The True Value of the "Plan → Subagent Implementation → AI Review" Closed Loop

[Gemini API in Action] Building MemeFinder: A Native Mac Menu Bar Widget for Finding Memes via Text Using Gemini Vision & Semantic Embeddings

The Origin: Mid-Conversation, Where on Earth Is That Meme?

System Design and Architecture

System Architecture Flow

Core Implementation

1. Auto-tagging memes with the Gemini vision model

2. Hybrid semantic + keyword ranking

Major Pitfalls and Solutions

Pitfall #1: The mysterious GeminiError error 0 — indexing and search both fail

Pitfall #2: SwiftPM's main entry-point conflict and the SwiftUICore linking error

Pitfall #3: Parallel indexing's rate limit and "I want to stop indexing halfway"

Pitfall #4: Evolving from a "windowed app" into "menu-bar resident + global hotkey"

Pitfall #5: The settings form is blank — one symptom, three layers of cause

On the "Development Process" Itself

Scenario A: Already contains `GenerateContentConfig`

Pitfall #1: The mysterious `GeminiError error 0` — indexing and search both fail

Pitfall #2: SwiftPM's `main` entry-point conflict and the SwiftUICore linking error