DEV Community: Denis Valášek

The Power of Gemini inside Trello: Building an LLM Assistant with Firebase Genkit

Denis Valášek — Sun, 21 Dec 2025 16:51:06 +0000

In the world of project management, Trello has long been a favorite for its simplicity and visual organization. But what if your Trello cards could talk back? What if a bot could analyze your card's details, look at your attachments, and provide intelligent updates or answers to your team's comments?

In this article, we'll explore how to integrate Google's Gemini 3.0 Flash model into Trello using Firebase Genkit.

Prerequisites

Before we dive in, make sure you have:

Trello accounts (One bot account, one to issue requests from)
A Google AI API Key
Node.js installed
Firebase/GCP account (for telemetry and deployment)

Setting up your Trello Power-Up

To interact with Trello's API, you'll need to create a Power-Up. This gives you the API Key and Token required for authentication.

Go to the Trello Power-Up Admin.
Create a new Power-Up for your workspace.
Generate your API Key and Secret.

Generate a Token to allow your app to act on behalf of your user.

Keep these credentials safe! You'll need them for your .env file.

Crafting the AI Logic with Firebase Genkit

Firebase Genkit simplifies building LLM-powered applications. We'll define a Flow that takes a Trello card ID and a comment, fetches all relevant data, and uses Gemini to generate a response.

Defining the Schema

First, we define a structured schema for our card data. This ensures Gemini receives consistent information, including checklists, comments, and even image attachments. It's possible to easily extend this schema for other details, like links to other projects, more cards etc.

export const TrelloRequestRequestInputSchema = z.object({
  instructions: z.string().describe("User's instructions"),
  attachments: z.array(z.object({
    url: z.string(),
    caption: z.string().optional(),
  })),
  details: z.string(),
  comments: z.array(z.object({
    person: z.string(),
    text: z.string(),
    timestamp: z.string(),
  })),
  checklists: z.array(z.object({
    name: z.string(),
    items: z.object({
      itemName: z.string(),
      completed: z.boolean(),
      dueDate: z.string(),
    }).array(),
  })),
  today: z.string(),
});

The Trello Flow

The core of our application is the trelloFlow. It fetches the card details using a helper function and then calls our prompt.

const trelloFlow = ai.defineFlow(
  {
    name: "trelloFlow",
    inputSchema: z.object({
      trelloCardId: z.string(),
      trelloComment: z.string(),
    }),
    // ...
  },
  async (input) => {
    const formattedCard = await getFormattedTrelloCard(
      input.trelloCardId,
      input.trelloComment
    );
    const output = await trelloPrompt({ inputData: formattedCard   });

    await trelloClient.cards.addCardComment({
      id: input.trelloCardId,
      text: output.output.responseText,
    });

    return { success: true, message: "Comment added successfully" };
  }
);

Multi-modal Capabilities: Seeing Attachments

One of Gemini's greatest strengths is its ability to understand images. Our bot can "see" attachments on a Trello card. In trello_helper.ts, we fetch attachments and convert them to base64 data URLs, which Genkit can pass directly to the LLM.

const formattedAttachments = await Promise.all(
  attachments.map(async (attachment) => {
    const response = await fetch(attachment.url, {
      headers: {
        Authorization: `OAuth oauth_consumer_key="${env.TRELLO_API_KEY}", oauth_token="${env.TRELLO_API_TOKEN}"`,
      },
    });
    const base64 = await response.arrayBuffer();
    return {
      url: `data:${attachment.mimeType};base64,${Buffer.from(base64).toString("base64")}`,
      caption: attachment.name,
    };
  })
);

Designing the Prompt with Dot-Prompt

Genkit introduces the .prompt file format, which is incredibly powerful for keeping your AI logic clean and separate from your application code. In prompts/trello.prompt, the Gemini 3.0 Flash model was selected for its speed and native multi-modal support.

The prompt uses Handlebars-style templates to inject our structured card data. We loop over the different parts of the Trello card, but this can be used to inject other context as well.

Connecting Trello via Webhooks

To make the bot reactive, we use Trello Webhooks. Whenever a comment is added, Trello sends a POST request to our application.

Different Express middleware is used to intercept, verify and format the requests, before they reach Genkit.

Security

We implement a middleware to verify that incoming requests actually come from Trello using HMAC-SHA1 signatures.

export const trelloSignatureMiddleware = (req: RequestWithAuth, res: Response, next: NextFunction) => {
  const signature = req.headers["x-trello-webhook"];
  const secret = env.TRELLO_WEBHOOK_SECRET;
  const callbackURL = env.TRELLO_CALLBACK_URL;

  const body = req.rawBody ? req.rawBody.toString("utf8") : "";
  const baseString = body + callbackURL;

  const computedSignature = crypto
    .createHmac("sha1", secret)
    .update(baseString)
    .digest("base64");

  if (signature === computedSignature) {
    next();
  } else {
    res.status(401).send("Unauthorized");
  }
};

Registering the Webhook

Once your app is deployed (e.g., to Cloud Run), you need to tell Trello where to send its updates. Make sure to fill out your deployment URL into the .env file. I've included a handy script for this:

npx tsx src/register_webhook.ts

Output:

69457XXXXXXXXXXXXXXX
Registering webhook for https://genkit-trello-{PROJECT_ID}.europe-west1.run.app/trelloFlow with board ID: 6945713XXXXXXXXXXXXXX

Seeing it in Action

Once everything is set up, you can tag your bot in a Trello comment (e.g., @GeminiBot what's the status of this task?). The bot will analyze the entire card context and reply.

It can even track progress based on checklists and card descriptions!

Monitoring

If we deploy to GCP and have the Firebase project setup, in the Genkit part of the dashboard, we get important debug and overview information right away.

Conclusion

By combining Trello's structured project data with Gemini's reasoning capabilities and Firebase Genkit's orchestration, you can create a helpful Trello assistant. Whether it's summarizing long comment threads, analyzing attached designs, or providing quick status updates, there are many ways for this to be useful. The project is also just a MVP. Feel free to fork it and extend it to your use-cases.

Check out the full source code at GitHub!

Set up RAG with Genkit and Firebase in 15 minutes

Denis Valášek — Mon, 03 Nov 2025 14:39:17 +0000

Intro

Retrieval augmented generation (RAG) is like a knowledge base to your LLM model that helps it to add your context-specific information into the prompt and answer. In this article, we will go through how to set up an endpoint that will provide an up-to-date answer about Genkit documentation thanks to RAG.

Prerequisites

Firebase project with billing enabled (you won't be charged if you follow along, but you need to have billing enabled to access Firestore and Firebase Cloud Functions)
- If you have never used Firebase, check out how to get started on the Firebase Blog
Genkit installed
- Genkit is an open-source library that abstracts much of the boring stuff when building AI-powered apps and adds great observability and dev tools on top.
Node.js/local dev environment with gcloud CLI

Getting the data & Chunking

Genkit uses Flows for organization, you can check out my other article to better understand the concept. Let's start with getting the data of the Genkit documentation.

The Firebase team provides the Genkit documentation via the llms.txt standard, which means the whole documentation is segmented into Markdown documents.

For the demo, we will be only interested in the JavaScript part of the documentation, so we can use that domain-specific URL for getting the data. Since the data is provided as text, we don't need any complicated APIs or processing. The LLM will understand Markdown just fine. But to provide specific context, we need to split the large Markdown document into smaller parts - chunks.
There are many different chunking strategies, we will use a fairly simple setup, split by sentences, with minimum chunk length of 1000 characters and max. 2000. We will also have 100 character overlap, to keep context between the chunks.

Here are two config objects that we will reuse through this demo:

const chunkingConfig = {
  minLength: 1000,
  maxLength: 2000,
  splitter: "sentence",
  overlap: 100,
  delimiters: "",
} as any;

const indexConfig = {
  // Firestore collection name
  collection: "genkitDocs",
  // Firestore field name for the content
  contentField: "text",
  // Firestore field name for the vector
  vectorField: "embedding",
  // Embedder to use for the vectorization
  embedder: googleAI.embedder("gemini-embedding-001", {
    // Dimension of the vector, it's important to use 2048 when using Firestore as the Vector DB
    outputDimensionality: 2048,
  }),
};

Embedding

The LLMs understand the text differently than we do, that's why we can make it easier for them to understand using embeddings.
In reality it means, that we will convert the text into vectors that can be used by the model to find relevant pieces of text more efficiently.

Once again, Genkit makes it easy for us to use the Google gemini-embedding-001 model to parse the chunked text into embeddings.

async function indexToFirestore(textChunks: string[]) {
  for (const text of textChunks) {
    const embedding = (
      await ai.embed({
        embedder: indexConfig.embedder,
        content: text,
        options: {
          outputDimensionality: 2048,
        },
      })
    )[0].embedding;
    await firestore.collection(indexConfig.collection).add({
      [indexConfig.vectorField]: FieldValue.vector(embedding),
      [indexConfig.contentField]: text,
    });
  }
}

This function will take the created chunks, embed them into vectors and save the results into Firebase Firestore database. It's important to also save the original text, so we can show it back to the user if necessary or reference it in other ways. It's easier to think about it if you were embedding an FAQ document, you can point the user to the specific question/answer in the document alongside the LLM generated answer by the model.

Indexing flow

Here is the full flow (missing the required file imports that can be found at the end of the article alongside the full code sample) that we can use to get, chunk, embed and save the Genkit JS documentation, or any other markdown file URL for that matter.

export const indexGenkit = ai.defineFlow(
  {
    name: "indexGenkit",
    inputSchema: z.object({
      urlLlmsTxt: z
        .string()
        .describe("URL for the .llms docs")
        .default("https://genkit.dev/llms-js.txt"),
    }),
    outputSchema: z.object({
      success: z.boolean(),
      error: z.string().optional(),
      documentsIndexed: z.number().optional(),
    }),
  },
  async ({ urlLlmsTxt }) => {
    try {
      // Download the Genkit documentation as Markdown text
      const docTxt = await ai.run("extract-text", async () => {
        const response = await fetch(urlLlmsTxt);
        const text = await response.text();
        return text;
      });

      // Chunk the documentation into smaller pieces with the defined chunking config
      const chunks = await ai.run("chunk-it", async () =>
        chunk(docTxt, chunkingConfig)
      );

      // Delete all existing documents in the collection to avoid duplicates when reindexing
      await ai.run("delete-existing-documents", async () => {
        await deleteAllDocumentsInCollection(indexConfig.collection);
      });

      // Create Genkit documents from the chunks
      const documents = chunks.map((text) => {
        return Document.fromText(text, { urlLlmsTxt });
      });

      // Save the created documents to Firestore
      await ai.run("index-to-firestore", async () => {
        await indexToFirestore(chunks);
      });

      return {
        success: true,
        documentsIndexed: documents.length,
      };
    } catch (err) {
      return {
        success: false,
        documentsIndexed: 0,
        error: err instanceof Error ? err.message : String(err),
      };
    }
  }
);

We can use the gcloud CLI to create the required index for Firestore to use embeddings, just make sure to use your project ID and change any fields values if you are not using the config provided above.

gcloud firestore indexes composite create --project={{FIREBASE_PROJECT_SLUG_ID}} --collection-group=genkitDocs --query-scope=COLLECTION --field-config=vector-config='{"dimension":"2048","flat": "{}"}',field-path=embedding

Retrieving data

In RAG, we have to create Retrievers to get the embedded or other data. Genkit provides a pre-created Firestore retriever, but you can learn how to create your own retriever in the Genkit documentation or finish this demo and ask your helpful RAG-powered assistant :)

const firestoreRetriever = defineFirestoreRetriever(ai, {
  name: "firestoreRetriever",
  // Firestore instance
  firestore,
  // Firestore collection name
  collection: indexConfig.collection,
  // Firestore field name for the content
  contentField: indexConfig.contentField,
  // Firestore field name for the vector
  vectorField: indexConfig.vectorField,
  // Embedder to use for the vectorization
  embedder: indexConfig.embedder,
  // Distance measure to use for the similarity search
  distanceMeasure: "COSINE",
});

The finished Flow

Putting it all together, we have the genkitQAFlow, that takes in the question, uses the Firestore Retriever to get the relevant documents and sends them to the LLM model (gemini-2.5-flash in this case) to generate the answer.

export const genkitQAFlow = ai.defineFlow(
  {
    name: "genkitQA",
    inputSchema: z.object({
      query: z.string().max(500).default("What is Genkit?"),
    }),
    outputSchema: z.object({ answer: z.string() }),
  },
  async ({ query }) => {
    // Retrieve relevant documents
    const docs = await ai.retrieve({
      // Retriever to use for the similarity search using Firestore as the Vector DB
      retriever: firestoreRetriever,
      query: query,
      options: {
        k: 3, // Number of top documents to retrieve
      },
    });

    // Generate a response
    const { text } = await ai.generate({
      model: googleAI.model("gemini-2.5-flash"),
      prompt: `
You are acting as a helpful AI assistant that can answer
questions about the Genkit documentation.

Use only the context provided to answer the question.

Question: ${query}`,
      docs,
    });

    return { answer: text };
  }
);

Trying it out

You can find the full code sample on GitHub. The sample contains Firebase setup, env and other parts, that are not necessary for the article, but are required to run the code.

When first running the retrieval Genkit QA flow, you will get an error, asking you to create a relevant Firestore index. Just follow the URL provided in the error to create the index.

Now you can run the following command, to start the Genkit dev UI.

genkit start -- tsx --watch src/genkit-sample.ts

Now you can visit http://localhost:4000 to see the Genkit Dev UI in the browser. Select Flows and the indexGenkit flow.

After pressing run, it will take about 2 minutes to get, chunk, index and save the required documents. For larger datasets, it would be better to optimize this process.

After successful run, you can see the spans along with the details in the right output panel.

After the indexing, we can switch to the genkitQA flow.

We can ask question in the input field and wait for the answer. On the right side we can once again see what is actually happening in the flow - the retriever, embedding and finally putting it all together in the call to the Gemini API.

Conclusion

We have explored the basics of RAG and Genkit, you should be able to take any long-form text and create custom embeddings to answer your domain-specific questions. Using Firebase, we have explored Firestore as a quick and cheap (free if you are within limits) vector database.

Thanks to Dominik Šimoník as a co-author of this article that is based on our talk at DevFest.cz 2025

Gemini in your Slack workspace using Firebase & Genkit

Denis Valášek — Fri, 27 Jun 2025 10:16:38 +0000

Ever wished you could just tag your custom AI model in Slack? With connection to your internal systems (and your safeguards), to get actually relevant data along with helpful monitoring of the usage and access?

Let's take a look how you can do that today using Slack Bolt API, Google Gemini and Firebase Genkit.

Slack setup

We need to create a Slack app that will handle all the Slack permissions, events, and interactions. It's fairly easy to do this for your local workspace if you are an admin, you might need additional permissions if you are not, or just create a new workspace for testing.

Head on to Slack Apps to create a new Slack app from scratch.

Make sure to pick some reasonable name, as it will be used to tag the bot later on and install it into the appropriate workspace.

After creating, you are presented with a bunch of secrets and values. You will need Signing Secret later on, but it's always available on this screen.

Go to OAuth & Permissions section in the left panel and pick 2 Bot Token Scopes:

app_mentions:read to get events when the bot was mentioned
chat:write to be able to reply to the mentions

Proceed to install the app into your workspace in OAuth Tokens, this will give you a bot token, with the permissions you set above.

Backend setup

For our backend we will be using Cloud Functions for Firebase and Firebase Genkit.

Cloud Functions will provide us with serverless webhook URL, that we can use for our Slack app and Genkit will make it easy for us to iterate on prompt design, switch models and get step-by-step logs of our AI requests.

Follow the README in the sample Gemini in Slack repository for this article to have the project setup.

Genkit

In the genkit.ts file, you can see several Genkit concepts being used, let's highlight a few:

enableFirebaseTelemetry();

This single line gives us a complete overview & traces for each API call we make using Genkit, making it very easy to keep track of our spend, usage and how well the app works. You can find all the live calls in the Firebase dashboard in the Genkit section.

export const slackFlow = ai.defineFlow(
 {
   name: "slackFlow",
   inputSchema: z.string().describe("Slack message text"),
   outputSchema: z.string(),
 },
 async (subject) => {
   // Edit the prompt to fit your use case, make sure to experiment with the models and tools to get the best results.
   const prompt = `You are a helpful assistant that can answer questions about the number of customers and revenue for a given date. You can use the tools provided to you to get the information you need. You might also answer general questions about the workspace.

   Here is the user's message: ${subject}`;


   logger.info("slackFlow", { prompt });


   const response = await ai.generate({
     model: gemini20Flash,
     prompt: prompt,
     config: {
       temperature: 1,
     },
     tools: [getCustomersForDate, getRevenueForDate, getDate],
   });


   return response.text;
 }
);

This code snippet represents a Genkit flow, which is like a Genkit function wrapper, which handles many internal tasks for one goal. Inside one flow, you might do several AI calls, call internal APIs or access the file system, but it makes it incredibly easy to access the function from the Genkit Dev Tools and to categorize it in Genkit Monitoring.

For simplicity, prompt is also defined right in the flow, but we could use Genkit Dotprompt if our prompt grows or we want to test multiple versions efficiently.

const getCustomersForDate = ai.defineTool(
 {
   name: "getCustomersForDate",
   description: "Gets the number of customers for a given date",
   inputSchema: z.object({
     date: z.string().describe("The date to get the number of customers for"),
   }),
   outputSchema: z.object({
     output: z
       .number()
       .describe("The number of customers for the given date")
       .optional(),
     error: z.string().describe("The error message text").optional(),
   }),
 },
 async (input) => {
   try {
     logger.info("getCustomersForDate", { input });
     checkIfFutureDate(input.date);
     // Here, we would typically make an API call or database query. For this
     // example, we just return a random value.
     return {
       output: Math.floor(Math.random() * 100),
     };
   } catch (error) {
     logger.error("getCustomersForDate", { error, input });
     return {
       error: "I cannot provide data for the future dates.",
     };
   }
 }
);

This part showcases how tool calling works in Genkit. We define tools and their parameters, along with descriptions to be later on passed to the model itself. Recently, Genkit also started supporting dynamic tool creation at runtime, which might be interesting for some use cases.

In the sample code tool calling is used as an example to call your internal APIs to provide answers to various questions, like:

How many customers did we have yesterday?
What is our revenue today?

I also provided some examples on how to handle errors in tool calling and some potential logical errors as well, like asking for revenue in the future.

Local testing

Once you go through the codebase, set your own secrets in the .env file, you might want to play with the prompt itself and the tool calling, to better fit your use-case.

You can run:

npm run genkit:start

To start the Genkit Dev UI, you will get a localhost URL as the output. In the left panel, select Flows and the slackFlow to start testing. You can treat the input field as whatever the Slack message would say and on the right side you can see step-by-step calls made inside the flows for easy debugging.

You can see that one of the tools handles transformation of dates like today or yesterday into date string

And result of the date tool gets passed into the getCustomersForDate tool, which simulates an internal API call to return a random number of customers.

And final output. Genkit debugging is super useful when dealing with complicated flows and tool calls, make sure to try it out.

Deployment

Now you can run $$npm run deploy$$ to deploy the function and get the URL.

Slack events setup

From the previous step, you should have a URL looking like this:

https://gemini-in-slack-{random-id}-uc.a.run.app

Head to your newly created Slack app into the Event Subscriptions section and enable events. This will give you an option to paste your Firebase Function URL there, make sure to add /events to the end of the URL after pasting it, so the complete URL looks like this:

https://gemini-in-slack-{random-id}-uc.a.run.app/events

You should get a green checkmark once Slack verifies your endpoint. Below the URL, subscribe to the app_mention event, to start receiving event notifications.

Testing

Now it all should be ready to test! Head on to your Slack workspace and choose a channel where you won't disturb anyone, making sure to tag the bot with your query. You should get an in-thread answer. If you didn't change the prompt or tool calls, here are some queries that should work just fine:

How many customers did we have today?
What was our revenue yesterday?

And to try the error handling, try asking: "What will be our revenue tomorrow?"

Genkit Monitoring

After few conversations, head on to Firebase Genkit in the Firebase Console to see all the logs and traces for your conversations, where you can later debug new calls and see if the tool replies as it should.
You also have an overview of tokens consumed and if something isn't taking too long.

Troubleshooting

If you do not receive an answer within a few minutes, make sure to check the Firebase Cloud Function logs to get an idea what went wrong. You might be missing an API key or left out one of the permissions.

Conclusion

We went over the basics of how to set up a Slack bot, explained some Genkit concepts and put it all together. I hope you can modify the sample code to something actually useful for your organization and thanks for reading!

Google Cloud credits are provided for this project. #AISprint

Understanding Genkit flows with Czech language tricks

Denis Valášek — Sat, 28 Sep 2024 11:48:18 +0000

Intro

Developing with LLMs still carries some risks: they're hard to debug, it's difficult to observe what's happening, and when you're using multiple prompts for a single task, it can quickly become a mess. Let's explore Firebase Genkit to see how it can make things a bit easier. I'll use a practical example of converting text with difficult characters (equations, dates, physics symbols) in Czech to their spoken variants, which will be better handled by Text-to-Speech systems.

What is Genkit?

Genkit is an open source framework from Firebase that provides libraries and developer tools to get the most of the various AI and LLM APIs, not just from Google. While it's made by the Firebase team, there are many additional plugins available that enable support for other providers like Anthropic, OpenAI and Ollama simply by switching a line of code.

A great part of Genkit are also developer tools that add additional observability and debugging options to developing LLM-powered apps, giving you a bit of reassurance in the non-deterministic characteristics that come with building on LLMs.

Getting started with Firebase Genkit

Use official docs to set up a new Genkit project. We will be using TypeScript, but Genkit also provides Go support. Make sure to say yes to creating a sample flow.

Flows

Flows are wrapper functions that enable us to tie various tasks together into a single callable function, while giving us observability and easier testing.

When creating new Genkit project, in the index.ts, you will have sample flow generated:

// Define a simple flow that prompts an LLM to generate menu suggestions.
export const menuSuggestionFlow = defineFlow(
  {
    name: 'menuSuggestionFlow',
    inputSchema: z.string(),
    outputSchema: z.string(),
  },
  async (subject) => {
        // Construct a request and send it to the model API.
    const llmResponse = await generate({
      prompt: `Suggest an item for the menu of a ${subject} themed restaurant`,
      model: gemini15Flash,
      config: {
        temperature: 1,
      },
    });

        // Handle the response from the model API. In this sample, we just convert
    // it to a string, but more complicated flows might coerce the response into
    // structured output or chain the response into another LLM call, etc.
    return llmResponse.text();
  }
);

Run genkit start in your project folder to build the project and start the developer UI at http://localhost:4000

Here you can see the flow being ready in the left menu under Flows:

Let's try it out! In the code, we can see that this flow expects just a string input (inputSchema: z.string()), so providing "sushi" as a restaurant theme should be enough.

We can also click on the "View trace" button, to see exactly what's happening inside our flow with defined spans. Currently, we only have the model call.

Preparing our prompts

Genkit supports the Dotprompt prompt format via the Dotprompt plugin, let's install it by following the docs. Don't forget to add Dotprompt as a plugin when calling the configureGenkit(...) function.

Dotprompt enables you to save your prompts as separate files that you can then directly use in your code. This separation helps you keep things clean and manageable as you scale, upgrade prompts etc.

Opening the Gemini 1.5 Flash model (or other models) within the Genkit UI will give us access to a prompt testing & creation UI.

Feel free to play with the UI and create some sample prompts. When you are done, click "Export prompt" and save the result in the prompts folder in your project.

We can modify the .prompt file to accept one or more parameters by adding the input header with a schema. The parameters can then be used inside our prompt text using the handlebars syntax.

Note: If you don't speak Czech, you might be wondering what's inside the prompts. The first one gives a generic instruction about the role the model is to assume - in this case, a student doing their physics finals. The 2nd one gives various examples about how specific things should be rewritten. For example, 1998 should be written as nine thousand ninety eight, etc. This helps the TTS engine pronounce words correctly in Czech.

Here is my physics.prompt including parameters:

---
model: vertexai/gemini-1.5-flash
config:
  temperature: 0.3
  maxOutputTokens: 8192
  topK: 32
  topP: 0.95
tools: []
input:
    schema:
        question: string
---

{{role "system"}}
Jsi kluk, jmenuješ se AI maturant a právě maturuješ na osmiletém gymnáziu, vytahuješ si téma z fyziky. Tvůj výklad by neměl zabrat déle než jednu minutu, poté budou následovat otázky od komise, které budou navazovat na dané téma a předmět. Na začátku uvítej komisi a pokračuj ve svém výkladu. Pokud dostaneš příklad k vypočítání, buď co nejpřesnější, dávej si pozor na jednotky, nezaokrouhluj na vysoké čísla, je velmi důležité, aby výsledky byly správně. V úvodu zahrň vypočítaný příklad včetně rovnice.

{{role "user"}}
Tématem je "{{question}}". Co mi o tomto tématu řekneš prosím?

And spoken_text.prompt:

---
model: vertexai/gemini-1.5-flash
config:
  temperature: 0.4
  topK: 32
  maxOutputTokens: 8192
  topP: 0.95
tools: []
input:
    schema:
        writtenText: string
---

{{role "user"}}
Prosím převěď následující text tak, aby se dal snadno vyslovit. Mimo jiné, následuj tyto pravidla:
===PRAVIDLA:===
# Vzorečky piš tak, aby se daly přečíst a vyslovit.
===Příklad:===
F = m * a
===Výsledek:===
Vektor síly se rovná hmotnosti tělesa vynásobenou vektorem zrychlení.
===Příklad:=== 
g = G * M / r^2
===Výsledek:===
Intezita gravitačního pole se rovná gravitační konstata krát hmotnost tělesa děleno vzdáleností tělesa na druhou
# Používej psané číslovky:
===Příklad:===
20. století
===Výsledek:===
dvacáté století
===Příklad:===
0,00027 N/kg
===Výsledek:===
nula celá nula nula nula dvacet sedm newtonů na kilogram

# Místo znaků jako =, / nebo * používej slovní spojení jako "rovná se", "děleno" nebo "krát".
===Příklad:===
2 * 3 = 6
===Výsledek:===
Dva krát tři se rovná šest

# Místo 1., 2., 3., piš "první", "druhý", "třetí" nebo Za prvé, za druhé, za třetí a podobně, v závislosti na kontextu.
===Příklad:===
1. Místo
===Výsledek:===
první místo

===Příklad:=== 
1) Věda a výzkum
===Výsledek:===
Za prvé, věda a výzkum

===Příklad:=== 
Alexandr I.
===Výsledek:===
Alexandr první

=== Konec pravidel ===
Následující text: 
{{writtenText}}

Převedený text:

As a quick check, this is how your folder structure should look like now:

Connecting the prompts inside a flow

Let's modify the pregenerated example flow, changing the name and input/output schemas. Genkit uses zod to define strongly typed schemas enforced even at runtime.

We also need to use the Dotprompt plugin to import the prompts into our code by calling promptRef with our prompt file name as a parameter.

Now we make 2 sepaerate calls to the model with our prompts, chaining them together with the results of the previous one, with the option to also not generate the spoken form at all.

const physicsPrompt = promptRef("physics");
const spokenTextPrompt = promptRef("spoken_text");

export const studentAnswerFlow = defineFlow(
  {
    name: "studentAnswerFlow",
    inputSchema: z.object({
      question: z.string(),
      convertToSpokenText: z.boolean(),
    }),
    outputSchema: z.object({
      answer: z.string(),
      spokenAnswer: z.string().optional(),
    }),
  },
  async (inputs) => {
    const answerResponse = await physicsPrompt.generate({
      input: {
        question: inputs.question,
      },
    });

    // If we don't need spoken answer, return only the written answer
    if (!inputs?.convertToSpokenText) {
      return {
        answer: answerResponse.text(),
      };
    }

    const spokenAnswerResponse = await spokenTextPrompt.generate({
      input: {
        writtenText: answerResponse.text(),
      },
    });

    return {
      answer: answerResponse.text(),
      spokenAnswer: spokenAnswerResponse.text(),
    };
  }
);

Now, when we load the Genkit UI and open the updated flow, we can already see the newly defined parameters as JSON. Let's populate the question parameter with "Mechanika kapalin a plynů" and set convertToSpokenText to true.

Running the flow, we can see the outputs as we have defined them in the code. We can notice that the outputs are slightly different, because the second prompt is transforming the text to be better understood by the TTS engine.

Opening the traces, we can see both calls being defined. Drilling deeper, we can inspect the individual runs for every prompt, along with timings, to see what's taking up the most time.

Deploying flows

Genkit is highly flexible with deployment options - you can deploy to any Node.js environment. But you get the most out of the framework by deploying to Google Cloud Run or Cloud Functions for Firebase . The details for deployment are well described in the docs. I would recommend Cloud Run as it uses a single Cloud Run service for all the flows, keeping the costs low in case you need to deploy multiple flows and keep minInstances: 1 to avoid longer cold starts.

You can view the complete sample project on GitHub.

Thanks for reading and special thanks to Peter Friese from the Firebase team for helping me out!

Google Cloud credits are provided for this project #AISprint

Learning from my YouTube History: Summarization via Gemini

Denis Valášek — Mon, 26 Feb 2024 15:08:44 +0000

Intro

We have already probably seen 1000+ videos on YouTube and hopefully gained some knowledge out of them. In this series I would like to explore the possiblities of extracting data from my YouTube history and ending up with personal YT query engine, where through Gemini and embeddings I could converse with all the videos I have watched.

Getting data

You can export your YouTube history either using the Google Data Portability API, if you want to download your data more often and convintiently or do it manually using Google Takeout. Formatting should be the same in both cases, if you select JSON.

Subtitles

Since downloading all the videos and getting the transcription from better models (As far as I know, YouTube doesn't recreate the subtitles with updated STT models) would be probably against ToS, we have to use the subtitles that YouTube provides as a content of the video. YouTube has official API that we can use to download captions for our use-case.

Our data export contains YouTube video IDs, then we can use the list endpoint to get all subtitles avaliable for that video. Sometimes, manually uploaded subtitles are avaliable, which is better option than the STT machine transcription, as they better separate sentences and speakers, which would yield improved results.

Example request:

curl \
  'https://youtube.googleapis.com/youtube/v3/captions?part=id%2Csnippet&videoId=yWBzsBaU-Os&key=[YOUR_API_KEY]' \
  --header 'Authorization: Bearer [YOUR_ACCESS_TOKEN]' \
  --header 'Accept: application/json' \
  --compressed

Example response:

{
  "kind": "youtube#captionListResponse",
  "etag": "09LrB4i7CoUwNzOLNJE6CCffKPU",
  "items": [
    {
      "kind": "youtube#caption",
      "etag": "oO2GJpAhG7JcHcqx3d2xYnMumA8",
      "id": "AUieDabcbEXa0PfMheqVCKM2A_H-JMA8hrRDlOWDlhTfZnBcA7k",
      "snippet": {
        "videoId": "yWBzsBaU-Os",
        "lastUpdated": "2024-02-25T01:43:01.37359Z",
        "trackKind": "asr",
        "language": "en",
        "name": "",
        "audioTrackType": "unknown",
        "isCC": false,
        "isLarge": false,
        "isEasyReader": false,
        "isDraft": false,
        "isAutoSynced": false,
        "status": "serving"
      }
    }
  ]
}

Once we get the captions ID, we can use the Download endpoint to get the subtitle file.

curl \  'https://youtube.googleapis.com/youtube/v3/captions/AUieDabcbEXa0PfMheqVCKM2A_H-JMA8hrRDlOWDlhTfZnBcA7k?key=[YOUR_API_KEY]' \
  --header 'Authorization: Bearer [YOUR_ACCESS_TOKEN]' \
  --header 'Accept: application/json' \
  --compressed

We repeat the process for all the videos and store the results.

Summarization

We can use Gemini to create a summary from the subtitles and title that we can store alongside the embeddings and subtitles. Let's do it!

Google Gemini via Vertex AI

Gemini can run in a few different enviroments, today, we will use Gemini via Google Vertex AI, as it's easily accessible even in Europe. If you are interested in learning the differences between the versions of Gemini, check out this article made by fellow GDE Allen Firstenberg.

Prompt that we will use:

I will give you a video transcription, please return a 2-4 sentence summary of the text and few tags that represent the video well in JSON format with fields summary and tags as array of strings. 

Title: ${VIDEO_TITLE}
Transcription: ${TRANSCRIPTION}

Getting started guide for Gemini describes the details pretty well, building of it, let's build example request:

Note: The curl command requires GCP project and gcloud installed. Follow the starter guide if you need to do that.

curl --request POST \
  --url https://us-central1-aiplatform.googleapis.com/v1/projects/{YOUR_GOOGLE_PROJECT_ID}/locations/us-central1/publishers/google/models/gemini-1.0-pro:streamGenerateContent \
  --header 'Authorization: Bearer $(gcloud auth print-access-token)' \
  --header 'Content-Type: application/json; charset=utf-8' \
  --data '{
  "contents": {
    "role": "user",
    "parts": {
        "text": "I will give you a video transcription, please return a 2-4 sentence summary of the text and few tags that represent the video well in JSON format with fields summary and tags as array of strings. \nTitle: Turning Old Sawmill Blades into Knives | How It’s Made | Science Channel\nTranscription: a sawmill blade lasts 5 to 10 years but when its Jagged edges wear thin it can'\''t cut logs anymore luckily the carbon steel can be salvaged a computerized high-press water tool cuts into a stack of three Sawmill blades in doing so it carves out numerous knife blade blanks the computerized tool also Cuts holes for the handles and a tab for attaching the knife sheath next the blades are transferred to a vibratory tumbler the tumbler is filled with triangular ceramic stones and a soapy solution for several hours the vibrating Stones smooth and clean the blades once complete a technician secures the blades in a fixture with screws the screws hold the blades down flat and lock them in position for the next step"
    }
  },
  "generation_config": {
    "temperature": 0.2,
    "topP": 0.8,
    "topK": 40
  }
}'

After merging array of responses, we get a result like this:

{
  "summary": "Old sawmill blades are repurposed into knives by cutting them into blanks using a high-press water tool. The blanks are then tumbled in a vibratory tumbler with ceramic stones to smooth and clean them. Finally, the blades are secured in a fixture and screws are used to hold them in place for the next step in the knife-making process.",
  "tags": [
    "Sawmill blades",
    "Knives",
    "High-press water tool",
    "Vibratory tumbler",
    "Ceramic stones",
    "Fixture",
    "Screws"
  ]
}

Here we can see that Gemini managed to parse the raw transcription quite well, even though it didn't even have separated senteces. We can save the summary and tags for further processing.

Watch out when running this on a large number of videos or longer videos (15 min+), as it can consume tokens very quickly. I have used this YT shorts video as it can fit nicely into the article.

Thank you for reading and next time we will look at how we can use Google Gemini to create embeddings from the subtitles.

Disclaimer: Google Cloud credits are provided for this project #GeminiSprint

Sorting NFT Art with MakerSuite and PALM API

Denis Valášek — Sat, 30 Sep 2023 16:11:18 +0000

At rc.xyz we sort out hundereds of thousands of NFT Artworks so the artists have nice and clean portfolios of their work. However, sometimes it's hard to convert the logic into the code, so let's use the help of Google MakerSuite and PALM API!

We won't dive deep into what MakerSuite or PALM is, if you are interested, please watch the linked introduction videos!

Getting started

MakerSuite offers several ways for interacting with the API, for our use-case, we will use the Data prompt, since it gives us a UI to create a prompt with few examples more easily.

First version of our prompt:

You are a category classifier, your job is to decide whether the items match together and if so, generate suitable category name. You output JSON with value valid: boolean and optionally name: string if valid is true.

We are using few tricks to make our prompt perform better:

Giving the LLM instructions about what it is supposed to do
Describing desired output structure (JSON with types like string and boolean)
Describing what choices it should made

Examples:

Tip: You can import/export examples right from the MakerSuite, under the Actions button

Testing

MakerSuite enables us to test our prompt with multiple test fields at once, so we don't have to call the API again and again, let's try it out with some cases we have not used in the testing data.

And after running the prompt:

Which is in line with what we expect, great!

Implementation

MakerSuite offers you to export the code snippet right from the web UI, which includes the full formatted prompt along will the other model settings.

While the JS snippet suggest installing helper packages, it's still fully possible to use just fetch with the API key, since there isn't any complicated auth process as with most of the Google APIs. Thanks to minimal dependencies, it can run anywhere in your backend, like in Firebase Cloud Functions.

As with any Generative AI stuff, we highly recommend adding human in the loop to verify that the results are valid and revert them if not.

You can view the full prompt thanks to MakerSuite sharing here.

Solving missing System Entities in Actions Builder: Community Types!

Denis Valášek — Fri, 24 Jul 2020 09:44:40 +0000

If you have tried the new tool for building Actions on Google, the Actions Builder, you may have noticed a few things missing (feature comparison of Dialogflow/Actions Builder). One of them being a large collection of System Entities - predefined data sets that enable you to extract specific data from user queries.

With Actions Builder, you have the following Types (entities) available:

For comparison, Dialogflow contains 30+ system entities in different languages.

But one of the cool features of Actions Builder is the ability to relatively quickly import new Types/Intents using gactions CLI.

So I decided to use that to create open collection of types, created by the community, for the community! You will find the list of types along with instructions on how to include them in your new Action.

Contributions are highly appreciated!

Find Actions Builder Community Types on Github!

Enjoy #AoGDevs!

Actions Builder vs. Dialogflow: Feature availability

Denis Valášek — Sat, 11 Jul 2020 16:46:00 +0000

Google had recently released new tool for building new Actions - Actions Builder. It replaces Dialogflow and is a default tool to create new Actions from now on, it also comes with a new SDK library.

I recommend you try it out with new Actions on Google Codelabs!

However, since it was released only a few weeks ago, many features we are used to from Dialogflow are missing. This could be a breakpoint for you if you are deciding what to use for your next Action.

I created comparison spreadsheet with a list of features and their availability. It will be updated when the feature availability changes or when there are new features for each platform.

If the feature is marked as unavailable, that doesn't mean it will stay that way. Actions Builder is still in active development and the table reflects the current state. It should help you to decide if you are starting a new project today, but features could change when you create the next one, so I recommend bookmarking the spreadsheet for future use :)

Actions on Google platform comparison spreadsheet

Thanks @afirstenberg for helping me with this!
Enjoy #AoGDevs!

Track time you spent on your projects using your #Voice

Denis Valášek — Sat, 27 Jun 2020 17:19:43 +0000

Why?

If you are a freelance developer, you probably have to track your time for the projects you do - or perhaps you just want to know, how long did you work on your side project, that you never finished. But keeping some app open all the time and keeping the entries in sync is a bit of a hassle and distracting. I was facing the same problem, so I decided to solve it with my favorite interface - Voice!

Solution

I have created Action for Google Assistant called Tracker for Toggl. Why Toggl? Because while voice is great, voice app itself is not enough for timetracking, you probably want web access, fancy reports and mobile app too. Toggl is a time-tracking app, which is available for free, so I picked it as a "backend" for the Action.

How does it work? Google Assistant offers Actions on Google platform, which you can use to build your own conversational experiences, where you take full control over what the user says and how you respond.
That means you have to understand, what the user says. I use Dialogflow to do the NLU and Firebase Cloud Functions for fulfillment and web hosting.

Showcase

There are several ways, how you can try the Action out! If you have a Google Assistant enabled device - like Google Nest Mini or Home Hub, you can say "Hey Google, talk to Tracker for Toggl", which will get you talking with the Action! Or if you are on the computer, use the Assistant Directory (Google Play version for voice apps) to send it easily to your phone or smart speaker.

At the launch, you are welcomed with initial introductions and an opportunity to sign in with your Toggl API Token, which is used for communication between Action and Toggl.

Now, if you are using a smart speaker or display, you will be redirected using notification to perform the sign in using web popup. You enter the token, it will verify that it is actually token for your account and hurray, back to the Action! Don't worry, you have to sign in only once.

Now you can access all features that the Action has to offer + your previous entries, projects, and workspaces are also available!

The simplest thing you can do is to start tracking, lets try it out by saying "Start tracking Writing an article for project Toggl Action".

Several things have to happen here. First, the NLU has to recognize the intent itself - to start new time tracking. After that, there are 2 Slots to extract. First is the name of the task "Writing an article" and second is "Toggl Action", which will get matched to users projects.

Now that tracking is running, we can confirm in Toggl web app that it's working! So you can combine webapp and the Action, whenever you need it.

And we can easily stop it, by saying "Hey Google, ask Tracker for Toggl to stop tracking" anytime.

You can also get reports, so for example, if you want to know how long did you work today, you can just ask.

Of course, you can choose different time intervals or even filter by specific projects. Let's follow up by saying "And past month?".

Here we have a wider variety of tasks, so we get a nice sum up. You can filter this more with projects and workspaces.

Few other things you can do:

Hey Google, ask Tracker for Toggl to start tracking Testing (go right to the task)
Continue last tracking
Get name and project for current tracking
Set description/project/tags after the tracking has started
Get help any time, to get you back on track
and some easter eggs.. :)

You can check out the full feature list at the Tracker for Toggl web!

The project is still in the early phase and I would appreciate any feedback, thanks for reading!

Actions on Google VS Code Extension

Denis Valášek — Wed, 27 Feb 2019 19:36:48 +0000

You know it, you got a new idea, so you hurry up to get that idea to life, but now you have to do all that boring (or interesting :)) stuff again. Create a new project, install node_modules, remember what was the basic project structure and then finally start writing the code. Which, some of the time, can be repetitive too.

Unfortunately, I can’t help you with most of the work I talked about, but I hope to help you with some of it :) I got tired of writing the same stuff over and over, so I have looked into the VS Code snippets. Since I wanted to ease my Actions on Google development, I looked if some snippets are available in VS Code Marketplace. To my surprise, there weren’t any! So I decided to create them myself and share with the community!

A faster way to start a new project

Table of current snippets is below and more are coming soon. If you already know what’s missing or what you want to be added, feel free to let me know or send apull request on Github :) Currently supporting JavaScript and TypeScript.

You can find it here or search for the Actions on Google snippets in VS Marketplace! Happy coding!

New intent