DEV Community: Oleg Sucharevich

Sync RSS feed to Trello

Oleg Sucharevich — Fri, 08 May 2020 13:58:07 +0000

RSS is a format for delivering regularly changing web content. Many news-related sites, weblogs, and other online publishers syndicate their content as an RSS Feed to whoever wants it.

TL;DR

rss-sync, is an application I just published that lets you to sync your favorite RSS feeds into your Trello board.
I use it to sync podcast RSS feeds into my Trello list. So each morning I see what podcast I need to listen to catch up. Using the rss-sync you can customize the card will be created, based on the information from the RSS feed.

The why

It is fun
I love automation - read my previous post about productivity
Zapier and other tools are too expensive :-)
It is fun, really.

The use-case

I use RSS feeds to subscribe to podcasts. Of course, I don't manually check the RSS. I tried so many apps over the last two years. Finally, I decided to stay with Spotify. The main reasons were that it has an integration with Google Home, there are mobile and desktop applications, I already use Spotify to listen to music. The only downside in Spotify is that they are not allowed to add manually RSS. It must be done through Spotify platform. I have a few podcasts that are password-protected, which means that to get the RSS I need to "log-in". In the end, I decided that I will not stop using Spotify. I need to get the notifications about the podcasts into one place. All my tasks are managed in one Trello board, both my personal and my professional life on the same board. I decided that once a podcast has an update, I want to get a new ticket into the board and set it in the right list, condition-based.

Example

In this example, we will sync the podcast show Stuff You Should Know and will create a new Trello card for each show that was published in the last 24 hours.

Requirements

Download the binary from Github
Trello credentials - you need to get Key and API token from Trello. Go to Trello Developer API Keys. You can find immediately the Key, the token should be generated, follow the steps to generate it.
Trello board data
- Board ID - The board ID is part of the URL when you logged in (https://trello.com/b/BOARD_ID/BOARD_NAME)
- List-ID - use the Trello API to get a JSON about your board, inside that you can find the List ID (https://trello.com/b/{BOARD_ID}.json).
- Labels IDS - Similar to List-ID, but harder to find :-( (not required)
RSS feed file - copy the content into feed.yaml

bindings:
- name: stuffyoushouldknow
  source: stuffyoushouldknow
  target: This Week List

targets:
  - name: This Week List
    trello:
      token: '{{ env.Getenv "TRELLO_TOKEN" }}'
      key: '{{ env.Getenv "TRELLO_KEY" }}'
      board-id: '{{ env.Getenv "TRELLO_BOARD_ID" }}'
      list-id: '{{ env.Getenv "TRELLO_LIST_ID" }}'
      card:
        title: '[{{ .source.name }}] Listen to: {{ .item.title }}'
        description: "{{ .feed.title }}\nLink: {{ .item.link }}\nDescription: {{ .item.description }}"

sources:
- name: stuffyoushouldknow
  res:
    url: https://feeds.megaphone.fm/stuffyoushouldknow
  filter:
    just-released: '{{ ((time.Now).Add (time.Hour -24)).Before (time.Parse "2006-01-02T15:04:05Z" .item.publishedParsed) }}

The tech

Using open-integration - read my previous post about it. What is the architecture, what are the usec-case a more.
The feed.yaml is composition of:
- Sources
- Targets
- Bindings
Golang template engine is used here to template the credentials and other values that cannot be published. It uses gomplate to extand the functionality of the basic tempalte library
Additional datasources that are injected into the template language are:
- {{ .item }}
- {{ .source }}
- {{ .target }}
- {{ .binding }}
- {{ .feed }}

Conclusion

It seems to me as a nice-to-have tool. Of course, it cannot replace for most of you the automation system like Zapier. I see a few reasons:

Tools like Zapier are SAAS platform, you pay the price and you don't need to manage anything.
Tools like Zapier will trigger this flow automatically without you running it.

Disclosure: I work @codefresh .
I use Codefresh, which originally is a CI/CD platform for Kubernetes. Codefresh has a free plan with 1 concurrent pipeline, some limitations on the CPU and memory spec, and the most important, unlimited executions per month. To run the rss-sync, it more than enough. I use the Cron based trigger of Codefresh to run this process once a day to catch up on all my stuff.

Continuous Automation with Open-Integration

Oleg Sucharevich — Mon, 23 Mar 2020 18:49:52 +0000

Open-Integration is a pipeline framework (aka pipeline-as-a-code), where you can create your pipeline, compile and execute it anywhere. I created it because it helps me to automate some of my daily tasks to increase productivity.

Disclosure: I work @codefresh , which is building a CI/CD platform with a strong orientation for a containerized application running in a Kubernetes environment. As Codefresh is adopting Golang, I wanted to learn and practice it. I, therefore, decided to write open-integration using Golang. My state of mind is all about automating anything that does not require my intervention, both at work and in my personal life.

A couple of weeks ago, I was listening to a podcast (in Hebrew), where the guest in the show described how he is saving an hour each day, consolidating all the tasks into one place and continuously prioritizing it.

TLDR: He is using Trello with 4 lists:

Backlog - where he holds all the stuff that should be prioritized.
Tomorrow - the plan for tomorrow.
Today - the plan for today (prioritized).
Done - what is done.

The guest in the show said that he is checking the email inboxes and other channels every few hours to add new tasks to the Trello board, updating the relevant board, prioritizing it (if needed) and focusing on the prioritized list, ignoring all the channels until the next sync. For me it sounds great, to consolidate all the tasks into one place, both for my personal life and my professional career, so I gave it a try.

I have made some changes that better worked for my workflow:

I renamed the Tomorrow to This week list.
I created labels with tags Codefresh, Personal Life, etc, to make sure I can group relevant cards.
In the “Backlog” list, each card holds a checklist with the relevant tasks, each item will be converted into a single card when I move to “This Week” with the relevant labels.

Very quickly I noticed that there were too many cards in the Done list and I wanted to remove them, but also to keep that data for my next retrospective. I wanted to automate the following flow:

Update cards in Google spreadsheet
add the card as a new row in case it is a new one
update “update-at”, “labels”, “status” and other columns for cards that moved between lists
Archive all cards in the “Done” list to keep the board clean.

So I started to look for a solution that can archive them and also store them into my Google spreadsheet. This way I can come back to it in my next annual review to check exactly what was done in my professional life but also, to do some kind of personal retrospective.

There are many tools out there that I tried: Zapier, IFTTT, automate.io, integromat, Microsoft flows. Some of them worked pretty well, some didn’t match my requirements, some were a bit expensive. I decided that a good side project to work on would be an open-source pipeline execution engine that can do the simple tasks that I wanted to automate in my life for so long.

After this quick overview, which explained what was the motivation for me, and which tools I have checked, let’s dive into the Open-Integration project. The Open-Integration project is a pipeline framework, and it allows you to create pipeline-as-code. It is written in Golang. I took that approach of pipeline-as-code because of:

We have way too many configuration files today (tons of yamls,tomls, and jsons are everywhere!)
Test and debug! I was missing this ability everywhere, as a developer, I am trying to cover my code with tests, there is no reason for me to not do the same with my automated flows.

Example

Lets create our first pipeline (requires go 1.11 and above):

First let's install oictl, the command line tool that generates basic pipelines ( for mac you can use Homebrew, otherwise, download the binary from Github )

    brew tap open-integration/oictl
    brew install oictl
    mkdir hello-world && cd hello-world
    oictl generate pipeline

Check the generated file

package main

import (
    "github.com/open-integration/core"
    "github.com/open-integration/core/pkg/state"
    "github.com/open-integration/core/pkg/task"
)

func main() {
    pipe := core.Pipeline{
        Metadata: core.PipelineMetadata{
            Name: "hello-world",
        },
        Spec: core.PipelineSpec{
            Services: []core.Service{
                core.Service{
                    As:      "exec",
                    Name:    "exec",
                    Version: "0.0.1",
                },
            },
            Reactions: []core.EventReaction{
                core.EventReaction{
                    Condition: core.ConditionEngineStarted(),
                    Reaction: func(ev state.Event, state state.State) []task.Task {
                        return []task.Task{
                            buildTaskCommand0(),
                        }
                    },
                },
            },
        },
    }
    e := core.NewEngine(&core.EngineOptions{
        Pipeline: pipe,
    })
    core.HandleEngineError(e.Run())
}

func buildTaskCommand0() task.Task {
    return task.Task{
        Metadata: task.Metadata{
            Name: "hello-world-command-0-0",
        },
        Spec: task.Spec{
            Service:  "exec",
            Endpoint: "command",
            Arguments: []task.Argument{
                task.Argument{
                    Key:   "command",
                    Value: "echo \"hello-world\"",
                },
            },
        },
    }
}

Now simply run:

    go run main.go

See the output:

    cat logs/tasks/*

Let’s go over it:

Concepts:

Engine (the open-integration/core pkg) - the piece to code that is executing the pipeline.
Event - Indicator that something has changed in the state of the engine, allows us to react to it (EventReaction section).
EventReation - A function that gets the current event and copy of the engine’s state, the function returns a set of tasks to be executed.
Service - A standalone binary, the engine is communicating with it over gRPC, exposing endpoints to run some logical task.
Task - execution of logical flow, a task is a request to call a service endpoint and pass the arguments.

Architecture

Flow:

Use Cases

I have created a few pipelines which I use daily

trello-google: a pipeline that will read my Trello board, update a Google spreadsheet with new, updated, done tasks, and archive the “Done” list - this way I can review the spreadsheet once a while, to understand (using the labels I have put earlier) what has been accomplished, what is staying in the “Backlog” for too long and so on.
jira-trello: I really love task management tools, Jira is the one we are using at Codefresh, but I really do not want to sync every time I am mentioned in an issue or an issue I am watching has been updated. This pipeline is executed before I arrive to the office, I request Jira information using JQL (Jira query language), and create cards on my “Today” list with proper labels, so I can start my working day quickly to understand what is the picture, ignoring all the emails Jira is sending.
open-integration/core CI - a pipeline that will use a Kubernetes service to run pods that are sharing a common PVC to clone, test, security scan, compile and release the open-integration/core repo.

Contribution

Project plan: Trello board

Open-Integration is an open-source tool, if you find it useful for you, please let me know. If you have any feedback, on the architecture, code, design, this blog post or any other stuff, I would love to talk to you! If you would like to contribute, please feel free to open a PR, the Open-Integration Github organization is made of the following repositories:

core: the pipeline engine, exposing structs and functions to build the pipeline, services, task, etc.
oictl: a command-line tool to generate code for pipeline and services
service-catalog: All the available services