More adventures in building a first flow with Kestra.io

Oops, this accidentally turned into a blog series. You may want to read this blog first.

If you could see the revisions and comment blocks and all the other associated chaos that went into writing this blog... you would know I've been through it! But we are learning and iterating and I can nearly guarantee it will be way easier for you when I'm done. Plus, I'm having a lot of fun.

Let's get into it.

We left off on our adventure a bit stuck in the mud with the Google Calendar API + plugin. While nobody likes meetings, it was incorrectly returning an empty array with an otherwise successful looking API call. I fixed that!

If you aren't into rants, scroll on down to "Now what?"

Your calendarID is probably not primary

I incorrectly thought that if I used the default primary value I wouldn't need to change anything in my Google Calendar Plugin. My personal and work emails don't have other calendars that I own. I have other calendars that have been shared with me and whatever special thing is going on with the "Birthdays" calendar, but I just assumed if I never made another calendar the first calendar ever made in my account would be primary and that's incorrect.

Your calendarID is often, but not always, your email address. Sometimes it will contain additional characters, and other times if it's part of a group, it may end in @group.calendar.google.com.

But by far the most irritating part of this was the practically silent failure. Those are the worst kinds of failures in software development and I spent DAYS revisiting this issue here and there to see if I could figure out what would work or not.

But honestly, if you got this to work with primary as your calendarID let me know what are you doing to make it work. For now, we've updated the docs.

You (probably) don't need to enable domain-wide authority to the service account

I did stumble upon some guidance that said you needed to enable domain-wide authority to the service account, which for this PoC and knowing my coworkers have run this plugin before, had me really skeptical. Maybe a reader can share more insight here, but I did not need to enable this. Maybe primary is a valid default if you have this enabled?

So long as you've invited the service account to your calendar and you've correctly set up your API key, you are good. Just make sure your calendarID is correct.

Now what?

So I have the plugin working, but even with the service account it can only see busy blocks and a small amount of metadata. This has to do with how our Google Workspace is configured, so your experience might be different, just like mine is different on my personal account.

I used this as an opportunity to learn the ForEach plugin, iterate through my meetings, and ultimately decide I probably wouldn't do anything beyond count the number of meetings I had. I do have access to the calendar link, but when the page loads it defaults to the current time slot, not the time slot for the actual meeting. Oh well.

ForEach

I used the ForEach plugin (remember, everything is a plugin), mostly as an excuse to test it out. I'm iterating over the list of events I get back in the previous task.

  - id: for_each
    type: io.kestra.plugin.core.flow.ForEach
    values: "{{ outputs.get_calendar_events['metadataList'] }}"
    tasks: 
      - id: get_individual_event
        type: io.kestra.plugin.core.debug.Return
        format: "{{ json(taskrun.value).htmlLink }} }}"
      - id: set_meetings
        type: io.kestra.plugin.core.kv.Set
        key: meetings_today
        value: "{{ taskrun.iteration + 1}}"
        kvType: NUMBER
        ttl: PT5M

You can see I actually use the metadataList because that's where the most interesting info I can "see" sits, but like I mentioned previously, the service account can really only see busy blocks. I did decide to keep the htmlLink I pulled from the individual events because it was a good example of working with JSON.

KV Store

Similarly to what I did with grabbing the htmlLink, you can see I set a key value (KV) and use the taskrun context somewhat trivially. It was really just an excuse to test these features in a flow.

Now my flow sets a key-value pair and I can "see" it in the KV Store. Although I have to move fast because I only set it for 5 min (ttl: PT5M).

So while I'm not going to achieve my ultimate vision of "seeing" what meetings I was in, I will add the count to my recap notes because who doesn't like numbers.

Notion

The Notion plugin was interesting to investigate. The "OOTB" documented functionality was to delete all existing content on a page and add new fresh content. Not exactly what I was expecting. I was hoping for more of an "append to the bottom of the existing content" type experience.

So I have some choices. I can create a feature request, I can create a new page for every weekly update, I can get the content off the page, put it somewhere, then put it back with the update (basically append it myself), or I could do some mix of the above.

Since this is a PoC to really learn more about the product, I'll create a feature request and then take what I think will be the easiest route - create a new page for each weekly update. (This is where she realized she needed to eventually adjust the trigger and meeting range to weekly.)

I could also just run it and see what happens.

So you'll never guess what happened when I actually ran the Notion Update plugin - it appended the content to the bottom of the page. Just like I wanted!!! So scratch what I said about created a separate page for each update, but I'm back to making an issue anyway because I cannot read the page... the response didn't include the actual page content.

Pivot again? Confluence?

Because I can't read the content from my Notion page via the plugin, I'm exploring the Confluence plugin which appears to do some of the same things Notion will do.

The biggest problem I ran into was the Confluence API wants you to pass back the version number, incremented exactly 1 over the current one. I used a KV Store value of type NUMBER for this, but I can't see the current version and I'm incrementing this value based on successful executions. Realistically, Confluence users will probably also update or modify the page in the browser, which will increment the version number and make the API unhappy. So another feature request!

GitHub

Oh GitHub. I love a love hate relationship with your query experience. My goal will be to get issues and PRs I created or was assigned to during the week and closed.

As a reminder, or maybe a heads up, GitHub considers everything an issue, which is why you'll see me call the issues endpoint, but I'll get both issues and PRs.

- id: get_issues
    type: io.kestra.plugin.core.http.Request
    uri:  "https://api.github.com/orgs/kestra-io/issues?filter=assigned&state=closed&since=2026-01-07T08:00:00.000-06:00"
    method: GET
    headers:
      Authorization: "Bearer {{ secret('CLASSIC_GITHUB') }}"
      Accept: application/vnd.github+json
      X-GitHub-Api-Version: 2022-11-28

- id: log_github
  type: io.kestra.plugin.core.log.Log
  message: "{{ outputs.get_issues.body }}"

- id: for_each_issue
  type: io.kestra.plugin.core.flow.ForEach
  values: "{{ outputs.get_issues.body }}"
  tasks: 
    - id: get_individual_issues
      type: io.kestra.plugin.core.debug.Return
      format: "{{ json(taskrun.value).url }} "
    - id: set_issues
      type: io.kestra.plugin.core.kv.Set
      key: issues_today
      value: "{{ taskrun.iteration + 1 }}"
      kvType: NUMBER
      ttl: PT5M

- id: log_issue_count
    type: io.kestra.plugin.core.log.Log
    message: "{{ kv('issues_today') }}"

Please enjoy this WIP snippet of YAML using the HTTP Request plugin. You may be asking yourself, where is the GitHub plugin?! Unfortunately, it didn't work for me on 1.1.11 but resolved with an updated to 1.1.13. You'll also see me taking healthy advantage of the Log plugin because I want to see what values I'm getting as I'm building out my flow. I'll take those out eventually. Or not, who knows.

💡 Pro tip to use the HTTP Request plugin if you are having trouble with any of the plugins that are essentially API wrappers.

Will I add the GitHub plugin back into my flow? Maybe? It returns a .ion and there is a plugin that turns a .ion into a JSON. So realistically I can get this into a shape I want to work with and I think the returned response will always be quite small. I like that this gives me more opportunities to explore plugins and expression syntax, so I'll add it to the TODO list.

What's next?

I got a pointer from a coworker to use the Return plugin, which looks pretty interesting for doing some of the data massaging I want to do with my issue and meeting responses. I don't need everything in the response, in fact, I need about 2 things in both of them, so I'll play around with this a bit more.

I did use the Return plugin in a snippet above, but you'll notice I just glossed over it. I definitely need to revisit this and share more info with you. Hold me to it!

Dare I say it's coming together? I can see how many meetings I went to, how many "issues" (remember, this is PRs and issues) I was assigned and closed, and I have something working with both Notion and Confluence, even if it's not perfect.

Now it may seem biased, but as I create these bugs and feature requests, the Kestra team has been super responsive to triaging them. Of course, I'm their coworker 😈, but I also strongly believe they are passionate about making a great product experience, so the more feedback they hear, the better they can respond and make a great product... well, great!

If all goes well, I'll wrap up this blog series with one more blog to give you tips and tricks about working with plugins in your own Kestra flows.

But for now, I think I have a few more issues to create.