DEV Community: Dylan J. Sather

Build your own chat bot with OpenAI and Pipedream

Dylan J. Sather — Thu, 09 Nov 2023 16:54:23 +0000

Meet Pi, Pipedream's little AI helper. Pi is a support bot, trained on our docs, code, and integrated APIs. It answers questions in our Slack and Discourse communities. Like ChatGPT, it provides (mostly) human-level answers to challenging problems.

Pi, not affiliated with https://pi.ai/

It's fun to talk to, it's saving us hours a day, and it's providing support to our community that we never could without an artificial assistant.

This week, we also updated Pi to use OpenAI's new Vision API, which lets us process screenshots and other media that Pi couldn't parse before.

Pi has access to Pipedream integration data, pricing, product docs, and more

Collectively, we've spent hundreds of hours researching, refining, and deploying Pi. We've made amazing progress, but since this tech is so new and moving fast, we still find issues and improve it incrementally every day.

We're sharing everything we've learned so you can build your own.

Terms

This article assumes you know how to code — or want to learn — and are curious about building AI bots. But we don't assume knowledge of language models like GPT-4. Here's a few terms you'll see used in this post and more generally:

LLM : Large language model. GPT-4 is an LLM, as is Meta's LLaMA. LLMs are trained on websites, papers, and other vast amounts of text, often from the Internet. They learn how to predict the next most-likely token to appear after the text the user provides. This is why you often hear token prediction referred to as "completion". We use OpenAI's GPT-4 model for Pi. As of 2023, GPT-4 performs better than other models on most problems.
Token : A piece of a word, a unit of text that models like GPT-4 deal with.
Prompt : The text you provide to the LLM. It can be a question, a set of instructions — any text.
Completion : Since LLMs are predicting the next most-likely token that appears after a user's prompt, it "completes" that text. This is why you'll see this verb used when describing LLMs. See e.g. OpenAI's chat completion API docs.
Embeddings : Embeddings are vectors that store the semantic meaning of a piece of text. (see OpenAI reference). If you want to compare the semantic meaning of two strings, compare their embeddings using a measure like cosine distance. Embeddings that are "close" in distance are more related. We'll discuss this all in more detail below.

Designing your bot

This is the high-level design of Pi. Your bot will probably look very different!

You should think through five key things when designing your bot:

What docs / content does it need to answer questions?
How are users going to interact with it? On Slack, email, UI?
Given a question, how do you process it and retrieve the most relevant docs? In our case, users are often asking about app integrations, so we try to identify those apps in the question and retrieve relevant app metadata.
What rules should it follow when communicating with users? How should it return its answers and links to docs? How should it respond when it doesn't know an answer? How should you include the docs from step #3 in the prompt to give the LLM more knowledge? This is where you'll develop the core instructions the LLM will use to answer questions.
How do you evaluate and improve the accuracy of answers? You'll constantly find issues with your docs and your prompt. You need a way to evaluate and improve your bot's performance for all of your key use cases. You don't want to ship a subtle change to your prompt and introduce a major regression in its behavior in another area, as we've done many times.

We'll detail how we approached each of these below.

1. Your bot needs docs and knowledge

Step 1 — Collect, process, and save your docs somewhere

If you're building a bot to answer questions about your business, it's likely GPT-4 doesn't know everything about you. That was the case with Pipedream. Since we launched in 2019, it knows we exist. But we add new features, docs, and integrations daily, we don't know all the public docs the model is trained on, and GPT-4 has a knowledge cutoff — a date where its knowledge ends. So today, it can't answer deep questions about the Pipedream product or API.

If ChatGPT returns reliable answers for your use case, and you don't care about events or updates after a certain date, you may be able to skip this part. Everyone else should read on.

Retrieval augmented generation

So how do we solve this problem? Enter Retrieval Augmented Generation ("RAG"):

Retrieval : Given the user's question, retrieve relevant docs / content
Generation : Pass the docs, along with the original question, to the LLM

A base model like GPT-4 is generally very good at summarizing text, classifying input, writing code and more. But it lacks the specific information it needs to answer your users' questions. When you provide that context with the question, the LLM provides better answers.

At this stage, you need to figure out what docs you need, how to process them, and where you're going to store them.

Choosing the right docs

Only use trusted docs. For example, your pricing page, terms of service, and your product's docs can be considered trusted sources. But your community Q&A, blog posts from external authors, and other user-generated content are not necessarily vetted.

We originally included community Q&A, but it reduced the accuracy of answers. Some posts matched the user's question, but were unanswered or had wrong answers. Others referred to years' old features that had since been improved or deprecated. After a few attempts to separate good posts from bad, we removed them entirely and saw our accuracy improve.

Today, we use our docs, pricing page, terms and privacy page, our blog, and our integrations data: information on the app, all of the sources and actions you can use on Pipedream, and the OpenAPI specs and docs of our partners.

We run this Jupyter notebook to fetch, process, and save docs into Supabase. We use Python because of the availability of text processing and AI client libraries. For example, we use PyTorch and scikit-learn to customize our embeddings — Python is the de-facto language for this use case.

Since the notebook processes Pipedream docs and custom Snowflake tables, you'll need to modify it for your use case. We'll walk through the core parts below so you see how this works.

Chunking

Chunking refers to the process of dividing a large document of text into smaller pieces. Typically, we need to do this for two reasons:

OpenAI and other LLM APIs have token limits, so to fit the user's question, docs, and answer, you can't pass every doc every time. These limits are rising — the newly-announced GPT 4 Turbo model supports 128K tokens per request — but since commercial LLMs charge per token, you'll want to minimize tokens to only the required information.
You want your bot to retrieve docs similar to the user's question. If you store pages of docs with a range of different concepts in a single record, the embeddings created from that will encode meaning about the document as a whole, and not any individual sections. We like to think of chunks as a section of text that maps to a unique concept.

To separate our docs into chunks, we used two utilities from LangChain:

For Markdown docs, MarkdownHeaderTextSplitter
For HTML docs, HTMLHeaderTextSplitter

When saving chunks, there should be some overlap between sections. For example, chunk 2 should contain the end of the text from chunk 1 and the start of the text from chunk 3. Since adjacent sections of docs are likely related, this keeps semantic context between these related chunks when they're saved as independent sections.

I haven't seen a one-size-fits-all answer for how large your chunk size or chunk overlap should be. According to Pinecone, embeddings with the latest OpenAI models work best when chunks are multiples of 256 tokens. But these should be tested and optimized for your use case.

## TEXT SPLITTERS ##
headers_to_split_on = [
    ("h1", "Header 1"),
    ("h2", "Header 2"),
]
text_splitter = HTMLHeaderTextSplitter(headers_to_split_on=headers_to_split_on)

## h2 and h3 sections of our Markdown docs separate logical sections
markdown_headers_to_split_on = [
    ("#", "Header 1"),
    ("##", "Header 2"),
    ("###", "Header 3"),
]
markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=markdown_headers_to_split_on)

Docs and data should be structured

You should convert your docs to Markdown or another structured format as a part of the processing — if they are not already — with separate content in separate sections, delimited by headers like # and ##. In practice, we've found that the inherent structure of headings, text, and other formatting communicates structure and meaning better than the same HTML. OpenAI also uses Markdown for their own prompts, so we think it's a safe bet.

Take note, OpenAI constructs prompts using markdown. I believe this is reflective of their instruction tuning datasets.

In our own work we’ve seen markdown increase model compliance far more than is otherwise reasonable. https://t.co/VniaLFbO2B

— Mike Conover (@vagabondjack) October 15, 2023

You'll see in our notebook how we retrieve data from Snowflake and convert that to Markdown text, which details how users can use our integrations in the Pipedream UI. Specifically, we convert data about our integrations to numbered instructions that tell users how to configure our triggers and actions, which Pi can use directly when talking to users.

Example Markdown instructions for Pi

Brex's prompt engineering guide and Anyscale's RAG guide highlight other techniques for passing structured data to the LLM. Experiment with your own data and keep iterating.

Convert the docs to embeddings

Once you have chunks of docs, you can convert them into embeddings.

Embeddings are vectors that store the semantic meaning of a piece of text. Embedding models accept text like "What's the temperature outside?" as input and converts it to a vector of numbers.

Embedding models convert text to numbers

Since these numbers represent semantic meaning, if you want to compare the meaning of two strings, compare their embeddings using a measure like cosine distance. Embeddings that are "close" in distance are more related. You don't have to understand the math behind this — cosine distance is implemented in many libraries.

How "close" are these two embeddings?

We use LangChain's OpenAIEmbeddings class to generate embeddings with OpenAI. You can use the OpenAI Embeddings API directly, or use an entirely different service like Cohere or your own embeddings model to generate embeddings.

from langchain.embeddings import OpenAIEmbeddings

embeddings = OpenAIEmbeddings()
embedded_docs = embeddings.embed_documents(docs)

OpenAI embeddings that use the latest text-embedding-ada-002 model are vectors of 1536 numbers. embedded_docs is a Python list of lists of embeddings. We'll store this alongside the docs content below.

Store your docs and embeddings

Now you need to save your sources and embeddings in a data store. We evaluated many different options — primarily Pinecone and PostgreSQL — and landed on Postgres for reasons I'll describe below.

We quickly realized that we'd want to return a complex set of docs based on the user's query:

Fetch all docs that matched the embeddings (via vector similarity)
Do a standard full-text search (people call #1 + #2 "hybrid search")
If we can identify apps in the user's question (e.g. "Slack", "Discord"), retrieve all integration docs tied to the app. This includes everything on https://pipedream.com/apps — the OAuth configuration for apps, test code, and information on all triggers and actions.

Docs are returned in order of descending similarity, so if we have to truncate content to fit into the 8K GPT-4 token limit, we'll (theoretically) get the best content at the top.

Postgres is mature and the pgvector extension provides the vector operations we need to compare embeddings. It was also easy extend our Postgres queries to do the full-text search and integration docs retrieval:

SELECT content
FROM docs
WHERE app = 'slack'

Retrieving all docs for the app is as simple as a WHERE clause, which is part of the reason we chose Postgres

Like with chunking, consider your data and retrieval model, and test different providers. A different embedding model may work better for your use case, and Postgres may not be the optimal place to store your content. This is an important step to get right. Your bot will provide poor answers if it's given bad docs, so it's worth the time.

2. Interface with users

We use Slack and Discourse for community support. Pipedream supports triggers and actions for these apps and hundreds more. So we've built Pipedream workflows to handle incoming questions and pass them to Pi to answer.

With Slack, Discourse, and other messaging apps, you don't need to manage your own UI. Even Midjourney — with millions of users — still uses Discord for their core app.

OpenAI's Chat Completion API also accepts chat history. These platforms store chat history for you, so you don't need to maintain that state yourself. The app-specific workflows just retrieve chat history directly from Slack or Discourse when a new question arrives.

For example, we've configured the Pi Slack app to send app_mention events to this workflow. The workflow fetches thread history, processes attachments, and sends this payload:

{
  "question": "test",
  "chat_history": [],
  "channel": "public_slack",
  "meta": {
    "channel": "C046G5HMU75",
    "thread_ts":"1699465441.545199",
    "files":[]
  }
}

meta contains channel-specific information, which is passed to the workflows that send Pi's answer back to the user.

to the core workflow that fetches docs, constructs the prompt, and sends it to GPT-4 to answer the question.

This workflow runs asynchronously, pulling user questions from a queue, keeping the concurrency of our workflow to 5 running workers at a time to avoid hitting OpenAI rate limits. This introduces latency during peak periods, but late answers are better than no answers at all, and users don't expect immediate responses via these channels. We also enable auto-retry in case of error, and run a dedicated worker to process questions as quickly as possible.

The next two sections cover how this workflow works and some of the lessons we've learned about docs retrieval and prompt engineering — two key parts of building a good chat bot.

3. Fetch the right docs

In section 1, we stored all the docs and knowledge our bot needs to answer questions. But we can't pass all of these docs to GPT-4: we're limited to 8K tokens per request and we want to minimize cost. So we need to identify the docs that best match the user's question.

We also want to identify all possible data that'll be useful to Pi. The more accurate data, the higher the accuracy of our answers. For us, this data includes text from screenshots and app integrations tied to the user's question, and we talk about those steps below. You probably have very different data that's important to your bot, but we hope the examples get you brainstorming.

Extracting text from images

With OpenAI's new Vision API — aka GPT-4V — we can process images. This is huge. Our users frequently attach screenshots of errors, API docs, and code, so we can process them and pass that context to Pi.

The get_files_and_get_content_with_openai step of the workflow downloads any files attached to the user's Slack message and asks GPT-4V to return the text, which we can include in the prompt we send to GPT-4.

const content = await axios($, {
  url,
  headers: {
      Authorization: `Bearer ${this.slack.$auth.oauth_access_token}`,
  },
  responseType: 'arraybuffer',
});

// Format the image for OpenAI
// See https://platform.openai.com/docs/guides/vision
const image_url = `data:image/jpeg;base64,${Buffer.from(content, 'binary').toString('base64')}`

const resp = await openai.chat.completions.create({
  model: "gpt-4-vision-preview",
  messages: [
    {
      role: "user",
      content: [
        { type: "text", text: "Can you return all of the text present in these images? Please put each section of text on its own line." },
        { type: "image_url", image_url },
      ],
    },
  ],
});

Extracting apps

API integrations are fundamental to our business — it is what we do. Over half of all questions ask about one or more apps, e.g.

How do I connect Google Sheets and Trello?
How do I send an email with AWS?

The extract_apps step of the workflow does a fuzzy match between the text in the question and the list of apps we support, retrieved from a Pipedream data store in the list_records step above. If we can identify apps in the question, we can retrieve useful metadata about the app — its configuration, OAuth scopes, etc. — that a keyword or embeddings search may not directly yield in its top results.

One cool thing to note: the vision API step used Node.js and this step uses Python, all in the same workflow — just use the library or language that works best to solve each problem.

Create embeddings from question, retrieve docs, hit GPT-4

We saved embeddings for all of our docs in Supabase. So when we search for docs that match the user's question, we also have to generate the embeddings for the question using the same text-embedding-ada-002 model as before. The create_embeddings step does that.

Now we can retrieve the docs that best match the original question. Recall that when we chose Postgres, we wanted to run a search that combined the best results of three queries:

Docs matching the embeddings (via vector similarity)
Docs matching a standard full-text search
If we identified any apps, the data on those app integrations.

The fetch_vectors step handles that hybrid search. We implemented a match_documents function from this Supabase post on pgvector, and the kw_match_documents function from this LangChain doc on hybrid search.

4. Write your prompt

In 2022, everyone said, "learn to code". Now we're telling people, "learn to write".

The hottest new programming language is English

— Andrej Karpathy (@karpathy) January 24, 2023

Your #1 goal when developing your prompt is to map everything you want your bot to know to structured English (Markdown). We'll pass in a clear set of rules for how the bot should answer specific questions, information retrieved from our docs, a strict output format for our answers, and more. Our final prompt looks something like this:

## System instructions

...

## General rules

...

## Rules for writing Node.js code

...

## How you should answer questions

Given the rules above, and the docs below, answer the user's question.

## Docs

...

### App metadata / API docs

...

## User question

Question: ${userMessage}

## Attachments

The user attached images in the message, which had the following text: ${steps.get_files_and_get_content_with_openai.$return_value.join('\n\n')}

---

Answer:

"Prompt engineering" has reached its hype stage, but it's core to building a good bot. I'd recommend trying any prompting tips you find when you're learning. I've picked up random hacks or links to research papers that have proved useful for our domain, like Chain of Thought or the Reason and Act (ReAct) model.

But most prompt engineering just involves writing clear, structured English. You'll see how we prompt Pi in the call_gpt step of the workflow.

System Instructions

You're a helpful Pipedream support bot and autoregressive language model that has been fine-tuned with instruction-tuning and RLHF. You carefully provide accurate, factual, thoughtful, nuanced answers and code, and are brilliant at reasoning.

You help users answer question about Pipedream. You're an expert at writing Pipedream component code. 

Try to use the information in the sources in user messages to answer the question, if possible. If you don't know the answer, you MUST follow up with questions to try to solicit more information from the user to arrive at an answer. You should be very, very skeptical you know an answer if you don't see it in your sources or the information you've been trained on. You also must not keep responding with the same answer over and over again — try to vary your responses and keep them short if the user is asking the same question.

This is important for my career. You better be sure of your answers.

We pass these system instructions on every request, and have modified it heavily over time. It's a been formed from a combination of observations, results of research papers, and our own, Pipedream-specific instructions. For example, we added the section on skepticism because we've noticed Pi was overconfident with answers, even when it wasn't passed any sources. Afterwards, it was more cautious.

User asked about "tokens", which referred to an OpenAI error message

General rules

Then we define other, basic rules for how Pi should operate.

# Rules

Below you'll find a list of rules for how you should answer questions and produce output. Please read them carefully. They are all critical. Consider them your constitution — any output that violates them is against the law.

## Rules for how you should answer questions

These rules are listed in no particular order.

1. DO NOT attempt to answer greeting with with "Hello, world!" examples. Greetings like "Hello" or "Hi" are not requests for Hello, World examples.

2. If you sense frustration or negative sentiment from the user, you should respond apologetically. Direct them to visit https://pipedream.com/support for more support options.

2b. If the user asks to talk to a human / person / someone on Support, direct them to visit https://pipedream.com/support for more support options.

...

Pricing questions

Math is notoriously difficult for LLMs, since they have not been trained for that purpose. But we try to get Pi to answer questions as best it can, directing users to our pricing page if it can't.

a. Any pricing questions will probably yield HTML documents that define pricing plans, FAQs, and other information on pricing. Parse the structure / content of these files as data and use it to answer pricing questions.

5b. If the user asks a question about pricing, plans (Free, Basic, Advanced, Business, Enterprise), costs, credits, or other pricing-related questions, think step-by-step. For example,

Question: How many credits are on the Business plan?
Thought: This requires using my sources at https://pipedream.com/pricing
Action: Lookup information in the https://pipedream.com/pricing SOURCE_URL below
Action Input: "How many credits are on the Business plan?"
Observation: I see a Business plan that includes 5,000 credits a month to start, and a cost per credit after.
Final Answer: The Business plan includes 5,000 credits a month to start, and charges a cost for additional credits after the 5,000 base. Please see https://pipedream.com/pricing for more details.

This prompt uses a technique from the Reason and Act (ReAct) paper, which shows performance improvements on reasoning tasks when you instruct the machine to observe, reason, and answer step-by-step.

Rules for writing code

We provide a large list of instructions to help Pi write code on Pipedream. The Pipedream runtime exposes a special API not available in Node.js or Python, and GPT-4 can't write perfect Pipedream code, so we provide that knowledge directly in the prompt.

You can read the code for the full prompt, but you'll see we clearly define the API, the output format, instructions on how to use our axios client, and more.

The first and foremost rule: when you're asked to write code, you MUST return Pipedream component code.

I need to teach you what a Pipedream component is. All Pipedream components are Node.js modules that have a default export: \`defineComponent\`. \`defineComponent\` is provided to the environment as a global — you do not need to import \`defineComponent\`. \`defineComponent\` is a function that takes an object — a Pipedream component — as its single argument. 

// many rules follow...

The key takeaway for you : identify what the LLM doesn't know about your use case and teach it explicitly in the prompt, with plenty of examples.

Managing token budgets

Recall that we're limited to 8K tokens in the GPT-4 model, across the question and answer. We've implemented some code to manage this budget. When we have long chat history or a lot of docs content, we truncate it to make sure we fit within the limit.

// There is no magic in these numbers. Just best-guesses at the proportion of context each section should provide.
const OPEN_AI_TOKEN_LIMIT = 8192 - systemInstructionsTokens.length;

// Allocate 20% of budget to docs, or lower if the docs are shorter
let DOCS_SOURCES_TOKEN_LIMIT = Math.min(Math.round(.20 * OPEN_AI_TOKEN_LIMIT), docsTokens.length);

// Allocate 10% of budget to chat history
const CHAT_HISTORY_LIMIT = 1000
const chatHistoryTokens = encode(JSON.stringify(chatHistory))
const CHAT_HISTORY_TOKEN_BUDGET = Math.min(chatHistoryTokens.length, CHAT_HISTORY_LIMIT)

// etc.

Weighting our sources

Our queries against Supabase returned the full URL to the doc and a similarity score between the doc and the user's question, ordered by that score.

const sources = steps.fetch_vectors.$return_value
// Sort sources by similarity in case we have to truncate
const sortedSources = sources.sort((a, b) => b.similarity - a.similarity)
const docs = sortedSources.map(s => `SOURCE URL: ${s.url}\n\CONTENT: ${s.body}\n\nSIMILARITY: ${s.similarity}`).join('\n\n---------------------\n\n')

We tell GPT-4 to weight the knowledge in these sources in its answers:

10. A source's SIMILARITY defines its cosine distance to the provided question and answer. Higher similarity scores are better — use that weighting to guide your answer.

Processing input, returning output

Remember that these are "completion" models. They're designed to predict the next-most-likely token to appear given the text in the original message. This is why we end the prompt with:

Question: ${User question}

---

Answer:

Send the answer

After it receives a response from GPT-4, the workflow sends the final answer and sources to to the intended destination. For example, the core workflow triggers this notification workflow for Slack, which creates a formatted message and sends it as a reply on the original thread. The same happens for Discourse.

5. Evaluate and improve accuracy

Your bot is likely to perform well on common questions where you have clear docs. In other cases, it'll provide incorrect but confident answers that will waste your users' time. Be skeptical of your bot's performance and carefully interrogate its answers for flaws.

Getting your bot to answer 100% of questions accurately is the hardest part of this process. As your product evolves, you'll constantly find new issues that need to be addressed with better docs or better prompts. You need to monitor its accuracy and develop a system to identify and fix bugs. Here's a few things we do.

Have a test suite

When we ship a new change, we use this workflow to send a flurry of test messages to Slack.

Each message is a test case: we're asking pricing, integrations, billing, and other random questions that have previously elicited poor responses. Like any test suite, we need to make sure Pi doesn't regress on core questions.

Whatever test suite you develop, use real questions from your users. Add short prompts with limited information to simulate users who will ask things like "help" or "pricing".

Many better tools exist for LLM unit testing and we're excited to test them as we improve Pi.

Review bad answers and iterate

Develop a system to log and review bad answers. We log every question, answer, and sources to a Google Sheet for later evaluation. At this stage, we're still constantly monitoring Pi's ability to answer real questions. And we can generally classify bad answers into two categories:

We passed bad docs to Pi, e.g. they were out-of-date or had a bug in the code.
Pi's response was poor given good information. In this case, we're providing good docs, but Pi didn't interpret them correctly.

We try to monitor every interaction with Pi. If it returns a bad answer because of bad (or no) docs, we add the 📚 emoji, which triggers this workflow. It pulls the thread history from Slack, summarizes it with OpenAI, and creates a new GitHub issue asking us to add better docs:

Pipedream automatically creates the GitHub issue for us

Adding the ⚙️ emoji triggers the same workflow, but adds a different title and labels:

const { reaction } = steps.trigger.event
const titlePrefix = reaction === "books" ? "New docs" : "Pi improvement"
const labels = reaction === "books" ? ['docs', 'pi', 'tracked internally'] : ['pi', 'tracked internally']

return await octokit.rest.issues.create({
  owner: "PipedreamHQ",
  repo: "pipedream",
  title: `${titlePrefix}: ${title_summary}`,
  body: `${longer_summary}\n\nSee the original Slack thread [here](${permalink}).`,
  labels,
});

Advanced: Rate your sources, use that to fine-tune your embeddings

If you're building a bot to answer questions on well-researched concepts like history and politics, you may not need this step. If you're building a bot for a product like Pipedream, you might want to fine-tune your embeddings.

Read through this amazing Twitter thread, which references this Jupyter notebook from OpenAI.

Boost your Semantic Search with Customized Embeddings 📈

1️⃣ Gather similar/dissimilar pairs

2️⃣ Train a matrix to highlight relevant aspects of your embedding

3️⃣ (Original Embedding)×(Custom Matrix)=(Custom Embedding) pic.twitter.com/1HT0XmfPAO

— Glavin Wiechert👨‍💻 (@GlavinW) February 20, 2023

Here's the core idea:

Run your embeddings similarity search against common questions (e.g. if you're using Supabase, run the match_documents function from this blog post). Return the top 5 sources.
For each source, record a 1 if it's relevant to the user's question and a -1 if it's not. We wrote a Pipedream workflow let us ask questions and retrieve docs directly in Slack. We rated each source with a 👍 or 👎 and saved the question, doc, and rating in a Google sheet.
Create a CSV of records with with the fields text_1, text_2, and label. Put the question in text_1, the content of the docs in text_2, and the 1 / -1 in label.
Load the data into this Jupyter notebook and run through the code, adjusting it as necessary for your embedding model and your data. It uses PyTorch to train a model that learns from in the feedback you provided.

This produces a numpy array that you can save locally and upload as a workflow attachment. The main workflow loads this array and takes the dot product of the array and the original embeddings from the user's question:

import numpy as np

def handler(pd: "pipedream"):
    custom_embeddings_matrix = np.load(pd.steps["trigger"]["context"]["attachments"]["custom_embeddings_matrix.npy"])
    modified_embeddings = np.dot(pd.steps["create_embeddings"]["$return_value"]["data"][0]["embedding"], custom_embeddings_matrix)
    return modified_embeddings.tolist()

This took us a long time to fully understand, but once we worked through the notebook and applied it to our embeddings, we started returning better docs and getting better answers from GPT.

Tend the garden, keep iterating, and ask questions

These are not set-it-and-forget-it projects. They require maintenance and tuning. The tech and APIs are new. We're still writing new docs, editing the prompt, and testing new techniques. But what we've shipped is working well, and it's providing a needed technical support function for our community that we couldn't provide otherwise.

Please join us the Pipedream Slack and Discourse communities if you have any questions about Pi, want to chat with it yourself, or just want to talk about AI and automation.

Build workflows faster with AI

Dylan J. Sather — Thu, 27 Jul 2023 16:54:07 +0000

The CEO of Stability AI, Emad Mostaque, recently predicted that "there will be no programmers in five years." It's an intriguing idea, but Emad is wrong.

Since the release of Copilot and ChatGPT, we've seen Pipedreamers write more code than ever. We expect that trend to continue. Thomas Frank, one of our most prolific users (see his voice notes to Notion workflow), only learned to code a year ago with the help of AI:

// Detect dark theme var iframe = document.getElementById('tweet-1655315536404787200-837'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1655315536404787200&theme=dark" }

We've always wanted to build AI into Pipedream itself, and we knew we'd start with code. Today, we launched a way to generate code from English (or any language ChatGPT supports). You select an app, tell us what you want to do, and we'll generate the Node.js code that does it.

It's the quickest way to build integrations with any API, and gives you full code-level control over your app.

How to use it

To manage load as we scale this out, we've released this to all Advanced and Business customers. Since you're getting a dedicated AI engineer, we think it's worth the upgrade!

Add a new step to your workflow, select an app, and choose the Use AI option.

This opens a modal where you can tell us the code you want us to write. Be verbose. Write something like "Send a Slack message to the #general channel in the following format: Hello, ${steps.user.name}" vs. just "Send a Slack message".

You can also edit existing code. Click the Edit with AI button at the top-right of any Node.js code step. You'll see the code gen window appear with the original code from your step. Enter a prompt to suggest an edit, and we'll give you the modified code.

The code gen service understands the Pipedream component API and references the API docs of integrated apps (opens new window). For example, you can tell it to include specific props (input) or async options, and reference specific API endpoints you want to use for the selected app. The more information you provide, the better the code will be.

What's next

First, we're adding Python support to the code gen service. Soon you'll be able to generate either Node.js or Python code with AI, all in the same workflow.

And we're testing the interface for generating full workflows , with triggers, actions, custom code, and anything you need to build your app.

We've also been working on our Q&A bot, Pi, for a few months.

Pi, Pipedream's little helper. Not affiliated with https://pi.ai/

Like ChatGPT, he provides (mostly) human-level answers on a range of questions about Pipedream, APIs, and code. He's fun to talk to, he's saving us hours of time a day, and he's providing support to our community that we never could without an artificial assistant.

Pi has access to Pipedream integration data, pricing, product docs, and more

You can chat with him in our Slack and Discourse communities, and we're planning to embed him directly in https://pipedream.com so you can ask him questions as you build workflows.

Please let us know if you see any bugs, want to suggest a feature, or just want to chat about AI integrations!

Archiving Slack threads to Discourse with GPT-3

Dylan J. Sather — Sun, 16 Oct 2022 22:36:14 +0000

We operate a public Slack community at Pipedream. Our users love the quick, synchronous chat, and the communication happens in public, so it's easy for others to find answers to common questions.

But these messages are private to Slack. You can't search Google for answers — you've got to sign up for our workspace. Plus, Slack deletes messages after 90 days on the free plan, so we lose threads quickly. But upgrading to a paid plan is costly for communities with thousands of users.

So what do we do?

Discourse as an archive

Discourse provides an open-source discussion forum. We already run our own Discourse forum for people who prefer forums to chat. All posts are public and indexed by search engines.

Google knows about 858 pages on https://pipedream.com/community

Discourse provides an API that will let us programmatically create new topics (the Discourse equivalent of a thread). All we need to do is convert the Slack messages to format Discourse accepts, and we'll have an archive.

For that, we use Pipedream.

Discourse Bot

When someone asks a question in Slack, we start a thread to organize communication on a specific issue. We created a Slack Bot that lets us easily archive these threads.

When we've solved a thread, we mention @Discourse Bot with the title we want for the archived Discourse post. For example, for this thread:

We post:

@Discourse Bot Deprecating an Action

The "at-mention" runs this Pipedream workflow, which creates a Discourse topic with the contents of the Slack thread, adding the title we sent in the message:

The workflow handles a ton of useful logic:

Fetches all messages in the thread where Discourse Bot was called
Not every Slack user has a corresponding Discourse account, so we convert Slack usernames to generic Discourse users, like user-1 and user-2 above. Not perfect, but it distinguishes messages from different Slack users on the original thread.
Converts Slack's mrkdwn format to standard Markdown, which Discourse uses for posts.
Creates a Discourse topic with posts for each message in the thread. We also create the topic in the right category, e.g. the #announcements channel archives threads to Announcements, #help is archived to Help, etc.

Challenges with this approach

Discourse Bot was working great. But it took time to manage this process:

It’s hard to keep track of threads that are complete. We'd set calendar reminders once a week, find all the threads we solved in the last week, and archive them to Discourse, but we inevitably missed some.
It took time to read through some threads and figure out an appropriate summary for the title of the Discourse topic.

We let these problems simmer in the background for a few months until we thought of something that might work.

Calling threads "done"

We can safely call threads done when 7 days have passed since the last message. Any time you need to trigger something after a period of inactivity, you can use a dead man's switch:

Set a timer, e.g. 7 days. This is your switch.
Some service pings the switch every so often, checking in. Every time the service checks in, we reset the timer to 7 days.
If the service stops checking in, and 7 days pass, the switch activates, triggering some action.

This is a common feature of alerting tools. Your server can ping a monitoring system every minute to tell it the server is up. If the server stops pinging, it might be down, so the monitoring system triggers an alert.

A dead man's switch can trigger any action. In this case, when 7 days have passed since the last message on a thread, we want to trigger a Pipedream workflow.

We built the switch in AWS, using DynamoDB TTLs. DynamoDB has a few properties that make it a perfect option:

It triggers an event when an item hits its TTL, using DynamoDB streams.
To reset the "timer" for a Slack thread, we overwrite the record for a given (channel , thread_ts ) — the key which uniquely identifies a thread — when new messages arrive on that thread. Pipedream workflow #1 handles that logic. Every public Slack message in our workspace triggers the workflow, sending them to Dynamo.
It's cheap. Assuming an average Slack thread has ten messages, it costs us roughly $0.00001538 to process one thread. That's about $15 for every one million threads, which we're unlikely to hit soon!

When threads expire, they trigger workflow #2. DynamoDB streams can't send an event to a Pipedream workflow directly, so we use a few other AWS services to get that done:

We created an AWS CDK construct that creates all the necessary resources, so you can easily make your own switch to notify a Pipedream URL when records expire:

import { Stack, StackProps } from "aws-cdk-lib";
import { Construct } from "constructs" 
import { DDBDeletedItemsToHTTPS } from "@pipedream/cdk-constructs"

export class DDBDeadMansSwitch extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    new DDBDeletedItemsToHTTPS(this, 'DDBDeletedItemsToHTTPS', {
      notificationURL: "https://your-endpoint.m.pipedream.net",
    });
  }
}

The summarization process

We had also started experimenting with GPT-3 internally. GPT-3 is a language model. You give it a "prompt" — often a question, or a problem to solve — and it produces text that looks like it was written by a human.

GPT-3 can be used for a variety of tasks. For example, we can ask it a question:

Can I use Pipedream to automate anything?

Thankfully, it replied positively 🤣:

You can ask it to classify items into categories:

You can also use it to summarize text. So we thought we should ask GPT-3 to pick titles for our Discourse topics.

Completed Slack threads from Dynamo trigger workflow #2. That workflow retrieves the thread's messages from the Slack API and asks GPT-3 to summarize it. We experimented with a few prompts. This one yielded the best results:

Pick a title for the following conversation:

But the summaries can be hit or miss, depending on the thread, so we ask GPT-3 to return its best three summaries from five attempts:

{
  "prompt": steps.format_prompt.$return_value,
  "model": "text-davinci-002",
  "max_tokens": 40,
  "n": 3,
  "best_of": 5,
  "temperature": 0.9,
}

GPT-3 model parameters

Then the workflow sends a private message to our team, asking us to choose the best one:

Choosing a summary triggers workflow #3: it at-mentions Discourse Bot on the original thread, which uses the original Discourse Bot workflow to archive the thread. Bots triggering bots!

This is what the process looks like end-to-end:

How Slack, Pipedream and AWS interact

GPT-3 prices based on the specific model you use, and the number of tokens you pass it. In our case, threads average ~500 tokens each. Since we're running five summaries per thread, that works out to $.05 to summarize a single thread. Given the SEO benefit of these posts and the time this is saving us, it's a cost we can bear.

What we learned

GPT-3 is saving us time, and we're finally archiving every solved thread. But the summaries are imperfect, so the process still requires a human in the loop. For our use case, this is acceptable. We'll keep experimenting with new prompts and other methods to improve the model's accuracy.

And while using DynamoDB as a dead man's switch is cool, it's an extra dependency. Building this workflow refined our thinking on how we might add dead man's switches directly into Pipedream.

Are you using GPT-3 or Pipedream for similar use cases? Did this spur any other ideas? Tell us in our Slack community!

Personal Support at Internet Scale

Dylan J. Sather — Thu, 14 Oct 2021 17:24:47 +0000

How Pipedream does Developer Relations

When I was 17, I worked the Guest Services desk at Target: the place you take returns or deal with anything that goes wrong.

Source

I was lucky to get the job. For a kid, it was a lot of responsibility. I made $6.75 an hour (a dollar more than my first job). I treasured it, and learned everything I could about how to work with customers.

Target did support right:

We listened and empathized deeply with customers to form a personal connection and understand their needs. I learned that listening patiently and saying, "I hear you completely" goes a long way.
We solved the problem quickly.
We tried to solve 100% of problems. If a customer was looking for an item we didn't have in stock, we'd call other stores. If the customer tried to return something that broke after our 90-day return window, we'd help them call the manufacturer to see if a warranty covered the product.
We had the training , tools and autonomy to solve most problems ourselves. Escalating to a manager wastes customers' time. If we could solve the problem in the moment (even if it meant an exception to our policy), we'd do it.

16 years later, I'm a co-founder at Pipedream, an integration platform for developers.

I run a team of Developer Relations Engineers (DevRelEng). We're customer-facing software engineers that support a community of 200,000+ developers. And we're hiring!

Since our customers are developers, every customer interaction = developer relations. This includes marketing, sales, support, solutions architecture, and more. DevRelEng is involved in it all.

When I work with our customers, I use the same principles I learned at Target. But now, we're supporting users over the Internet. It's tougher to fully empathize because we can't read their body language. Communication is asynchronous, so it takes longer to address issues. And since anyone with a computer can use Pipedream, we receive a higher volume of requests.

How do we maintain highly-personal support at Internet scale? And how do we specifically tackle this for a developer tool like Pipedream? This is how DevRelEng was born.

Three principles govern our work:

Our customers are developers. Developers can empathize best with other developers, so they should interact with customers.
DevRelEng is equipped with the skills and autonomy needed to solve nearly every customer issue. We frequently ship code to production to fix a bug or address small feature requests. Issues are solved fast, and our customers notice.
We approach developer relations like software engineers, building internal tools and automating everything we can to make us more efficient.

It works well for us, and I'm excited to tell you all about it.

What is Pipedream?

Pipedream runs an integration platform for developers. What does that mean?

Developers use Pipedream to connect different apps and APIs together using a visual programming environment. If you need to send a message to Slack or write a record to your database each time someone opens a new issue in your GitHub repo, you can develop a "workflow" without writing code.

Here, the title of the GitHub issue is the text of the Slack message

But you can run custom Node.js code anywhere you need it, extending any of the built-in actions or adding your own. This is where Pipedream shines. You can quickly transition from a no-code workflow to a Node.js serverless function as you need more complex logic.

You can use Pipedream to automate almost anything, and it's free. Watch the video below, or read our docs to learn more.

What does DevRelEng do at Pipedream?

You can break down DevRelEng into two core functions:

Supporting our customers
Building software / tools to better support our customers

Supporting our customers

It's worth reiterating our core mantra:

Since our customers are developers, every customer interaction = developer relations.

So when we say "support" in DevRelEng, we're talking about the whole customer lifecycle, from marketing to sales to technical support.

Pipedream is free for low-volume workflows. Most users chat with us in our Discourse, Slack, and GitHub communities. Many of these users are evaluating Pipedream for use on their team, and later upgrade to one of our paid plans. We aim to answer 100% of our community questions because:

It builds a public database of answers to common questions. If we answer a question once, it's indexed by Google and we can refer future users to these answers. We acquire a lot of new users from search keywords for specific APIs, so each new page we add (e.g. "How do I connect Stripe and Discord?") contributes to that acquisition strategy.
When someone gets a workflow working, they're hooked. They end up building more, and we convert a percentage of those users to a paid plan. We're doing Sales without explicitly selling anything. We just helped them use the product.

Larger teams have access to dedicated, priority support. DevRelEng handles that, too. Most of these customers use Slack, so we setup dedicated Slack Connect channels where they can reach us quickly (from the customer's perspective, nothing is better than chat support). These teams are composed of highly-technical power users. With them, we're often wearing our Solutions Architect hat, answering workflow design questions or helping them build custom sources and actions.

Like with our free customers, we try to answer questions in public. If a Teams customer asks a general product question in a private channel, we'll write a doc or a Discourse post and share it with them so that all users benefit.

We also hear a lot of product feedback as we talk to customers. Here, we act as Product Manager, asking a lot of clarifying questions to get at the core of what the user's asking for. Every new request is assigned a GitHub issue, and we ask future users to +1 those issues so we can track the most popular asks. We've developed a Pipedream workflow that collects this reaction data so we can analyze it:

An API / CLI for workflows is our #1 ask by far, and we're working on it!

This is a great example of how we're thinking like engineers when we solve support problems. It's a perfect segue to talk about the engineering side of DevRelEng.

DevRelEng software and tools

We use standard tools to manage our community (GitHub, Slack, Discourse) and infrastructure (AWS, Vercel, Snowflake, etc.). I'll tell you how we leverage these services, and how we automate everything we can using Pipedream (DevRelEng alone runs 40+ Pipedream workflows internally).

GitHub

import axios from "axios"

export default {  
  key: "http-get-request",
  name: "GET Request",
  description: "Make an HTTP `GET` request to example.com",
  type: "action",
  version: "0.0.1",
  async run() {
    return await axios({
      method: "GET",
      url: "https://example.com",
    })
  },
}

A simple Pipedream action. Learn more here.

Our GitHub repo is the center of our developer community. All Pipedream components (triggers and actions you can use in workflows) live on GitHub. Any user can contribute new components for any app, and we manage component development on this project board. Our docs are public, as is our roadmap.

We run 10+ Pipedream workflows on different events from GitHub. For example, we kick off a workflow when specific labels are added to issues. When a user requests a new integration, the app label is assigned, and it sends the app request to our integrations team. When a user requests an event source, we add the source-docs label, which sends a message to the user asking them if they'd like to develop the source themselves:

This is a good example of how we automate highly-personal support. We write a kind, helpful note once, and we develop an automation to send it to any customer with the same request.

Discourse

Discourse provides an open-source community forum. All Discourse posts are indexed by Google, which helps existing users find answers to common questions, and attracts new users through search. We self-host our Discourse forum at https://pipedream.com/community as a Docker container using the discourse-docker project.

Slack

A recent announcement. Join our Slack for updates!

We also operate a public Slack, where people can ask questions about Pipedream, chat component development, or share workflows and tips.

Some people prefer to chat on Slack, others live in Discourse. We maintain both to let users communicate over their preferred channel. The main drawbacks are:

We split our community between Slack and Discourse (something I believe we can solve)
Slack isn't indexed by Google.

To solve #2, we built Discourse Bot : a Slack bot, powered by a workflow, that converts Slack threads to public Discourse topics.

Any time an issue is solved in Slack or a good discussion wraps up, we chat @Discourse Bot with the desired title of the Discourse topic:

That kicks off the workflow, converts Slack's message format to Markdown, uploads images, and creates a Discourse topic with posts for each message in the Slack thread:

Docs

Our docs deployed on Vercel

https://pipedream.com/docs is managed with VuePress and deployed to Vercel. Docs are mostly written as Markdown, but VuePress lets us write custom Vue.js components when we need it. Anyone can edit our docs on GitHub.

We try to write comprehensive docs. Any time a user has a question, we ask, "is it documented?" If not, we'll either write a doc for it and share it with the user, or make a note to add a doc later. For this, we developed a Slack Bot called Docs Bot. We chat the Docs Bot in Slack:

That triggers this workflow, which creates a GitHub issue with the docs label:

Every couple of weeks, we search for all open docs requests and write a batch of them.

AWS

We run Pipedream on AWS, and the DevRelEng team is no different. A few example apps:

We operate a Squid proxy paid customers can use to route HTTP traffic through a fixed IP.
We run an anomaly detection app powered by Facebook's Prophet forecasting library. It tells us if metrics dip or rise in unexpected ways ("Did signups drop? Is something broken with that flow?"). We built the service because customers kept reaching out to tell us some feature broke before we noticed. Normally these issues show up in product data, so the app looks for these anomalies and tells us when they happen.
We've also setup "dead man's switches" that tell us when core integrations break. For example, we run a Pipedream workflow that sends a message to Discord. A different workflow runs on new Discord messages in the same channel. If both workflows run successfully, we were able to send and receive messages from Discord, so the integration works. We ping AWS CloudWatch to notify them that message was processed end-to-end. But if this process breaks, the dead man's switch triggers since it hasn't received data, and we raise an alarm in Slack:

Segment, Snowflake, Looker

Product data is critical for our team. When we're troubleshooting a bug, we need to understand how many users might be impacted. Or we might want to look at the number of users who use a specific integration to assess the priority of developing new actions.

For analytics, we use the SSL stack:

Segment: Usage data is recorded via Segment track calls. Segment syncs these events to Snowflake, managing the schema for us.
Snowflake: Snowflake is fast, and works well as a product analytics database.
Looker: Looker is essentially a pivot table for your database. We create a model of our Snowflake schema that maps SQL to metrics and dimensions. When you use Looker's UI, you write no SQL. You just pull in metrics and dimensions (e.g., I want to see the count of users who signed up for Pipedream week over week), and Looker runs SQL behind the scenes. Looker is critical for answering complex questions quickly.

Challenges ahead

Pipedream is three years old. The product itself and the nature of how we interact with our developers will change drastically over the coming years. As we grow the business, DevRelEng is thinking about a few core things:

Content / SEO : Search traffic is critical for our user acquisition. If a developer asks Google how to integrate two apps, we want them to land on Pipedream. The whole company thinks about this problem, and DevRelEng tackles it from a community perspective. How do we develop more useful content for Pipedream and the integrations ecosystem at large? How do we encourage the community to generate content?
Maintaining high-quality support : How do we continue to respond to 100% of requests in a personal, timely manner as we scale? This will involve a thoughtful mix of automation, technical writing, and cooperation with our community. We support 500+ apps, but we're not experts in them all. How do we leverage our community of experts, or the app providers themselves, to assist with bespoke integration questions?
Scaling contributions : Building event sources and actions requires expertise in the target API. We are experts at building integrations, but again, we lack deep expertise in any one specific API. Our team is focused on improving the core Pipedream component API and reducing the barrier to contributions so that it's easy for the experts (app providers, community developers) to create components.
Automating everything : We have a backlog of 30+ workflows we'd like to develop to solve things we do frequently (and manually) today.

Join the team

We're a small team with a lot of experience growing companies, but we need help solving these problems. We're looking for software engineers who like DevRel, DevRel who like writing code, teachers, solutions architects, and more. If you know how to write code and you've supported people in a technical role, you'll be a great fit. Apply here.

We're also hiring a UI Engineer and a Devops / Site Reliability Engineer.

If you have any questions about DevRelEng, Pipedream, or just want to say hi, I'm:

Dylan Sather (Pipedream) on our public Slack
dylburger on Discourse
dylan at pipedream dot com

Avoiding parking tickets with Pipedream

Dylan J. Sather — Sun, 01 Nov 2020 19:44:27 +0000

I grew up in Oklahoma City. In OKC, you drive everywhere. Public transportation is lacking and it's a huge city — over 100 square miles bigger than Los Angeles.

Ten years ago, I moved to San Francisco and drove nowhere. My car sat on the street as I biked to work. But the Public Works department cleans most streets on a schedule, and if you don't move your car before cleaning begins, they give you a ticket.

The schedule varies from street to street — one block might be Mondays, another Wednesdays — and I would always forget to move my car at the right time. Since I rarely drove it anyway, I ended up selling it.

Fast forward to the present. My wife and I have a child:

And we bought a new car. The memories of those tickets rushed back. I needed something to remind me to move my car at the right time, no matter where I parked. I needed a way to turn this:

into a Google Calendar reminder:

I used iOS Shortcuts and Pipedream to automate this, with help from a handful of APIs. You might not ever park your car in San Francisco, but I hope what I learned about solving a real-world problem will be helpful for any automation you do.

What is Pipedream?

Pipedream is an integration platform for developers. You can use it to run workflows - a set of steps that compose an automation. You can run workflows on HTTP requests, emails, on a schedule, or in response to app-based events like new tweets, new files uploaded to Google Drive, and more.

Within a workflow, you can use pre-built actions that perform common operations across 300+ apps, like creating a new calendar event, or sending an email. You can also write any Node.js code when you need to solve a custom use case where code works best.

You can think of Pipedream as a mix of integration tools like Zapier / IFTTT and serverless platforms like AWS Lambda / Google Cloud Functions.

First attempts - extracting text from the signs

I'll spare y'all the details of my many failed attempts. In short, the real world produces messy data.

Since the street signs include the cleaning schedule, I thought I could take a picture of the sign and use the text to create the reminders. I extracted that text using Google Cloud's Vision API. Then, I parsed that text to get the day and time of the cleaning, creating a Google Calendar reminder. This worked great for a few signs around my neighborhood.

Then I found this sign:

and this one:

Clearly, I couldn't rely on reading the text of every sign. I needed a different approach.

DataSF to the rescue

The City of San Francisco operates an "open data" portal called DataSF. The DataSF team helps source data from various city departments and aggregates it into a single platform.

Anyone can download COVID-19 stats, 311 calls, a map of every tree maintained by the city, and hundreds of other public datasets. All the data are accessible via API.

I'd used DataSF for past projects, so I figured I'd search their portal for "street cleaning". And what do you know - they provide the street cleaning schedule for all city streets.

Every record in this data set had the following structure:

A Centerline Network Number - the city's unique ID for the street segment (roughly a few city blocks)
The side of the street this cleaning schedule applies to (different sides have different schedules)
The day and time of cleaning for that side of the street, e.g. Wednesdays from 12pm to 2pm
A Well-known text (WKT) representation of the geographic line representing the street segment

That last value is a LINESTRING: pairs of (longitude, latitude) values that compose a line.

LINESTRING (
  -122.485667402435 37.778117247851,
  -122.485532329198 37.776252458259
)

You can use a WKT viewer like Wicket to visualize these lines on a map:

I wanted to find the street segment closest to where I was parked so I could get its cleaning schedule. I hadn't worked with geo data much, so it wasn't obvious how I'd solve that problem.

DataSF is powered by a platform called Socrata. When you make API requests to DataSF, you're using Socrata's API. They provide a list of functions that can be used in API requests. I scanned this list and found the intersects() function, which "allows you to compare two geospatial types to see if they intersect or overlap each other".

Instead of finding the closest street segments to my car, I could reframe the problem: I want to find the street segment that overlaps with my current location.

I could use my phone to get my current location. But my location is a point, and the street segment is a line in the center of the street. Those geometries don't intersect.

Instead, we need to draw a box around our current location a few meters in each direction. In WKT terms, this box is a POLYGON, and we can ask DataSF whether that polygon intersects with any streets. This also helps address the imperfect accuracy of GPS by broadening the area where we search for close streets.

This worked great, but presented a new problem. Every street segment has two cleaning records: one for the left side of the street, another for the right. The WKT LINESTRING for both sides ran down the center of the street, so there's no way to tell whether you're on the right side or the left side by comparing the distance from your current location. And since GPS isn't perfectly accurate, we wouldn't want a program to pick — it might pick the wrong side.

There's a non-obvious but simple solution to this problem: just let me choose the right time from a list.

Glueing it all together with iOS Shortcuts

Even in the early versions of this project, when I was taking pictures of street signs, I used iOS Shortcuts to kick off the automation from my iPhone.

Shortcuts provides a "no code" programming environment for your iPhone. You can use it to retrieve your current location, take pictures, make HTTP requests, trigger actions in other iOS apps, and more. They even allow for basic flow control (if / then statements, for loops, and more). And they provide a way to prompt a user to select an option from a list. So I used the DataSF API to find the closest streets and their cleaning schedules, then let the user choose the right one.

(Android users should check out Tasker, which offers similar functionality).

End-to-end, here's how the automation works:

Shortcuts : Get current location
Shortcuts : Send that location to Pipedream (HTTP Request)
Pipedream : Get street cleaning schedules for a given location
Shortcuts : Present schedules in list, let user choose
Shortcuts : Send chosen schedule and location to Pipedream (HTTP Request)
Pipedream : Delete old reminders, create Google Calendar reminder with current location

So far, no tickets!

Creating the calendar reminder

This last workflow is neat, and I want to show you two things you might be able to use in other projects.

SFData gave me the day and time of cleanings, but busy streets are cleaned multiple days: Tuesday and Thursday, for example. I had to find the closest calendar date in the future given that schedule. If I parked on that Tue / Thu street on a Wednesday, I'd need to create a reminder to move my car by the next day (Thursday). If I parked there on Friday, the reminder would need to be for next Tuesday.

Enter Chrono, "a natural language date parser in Javascript". Chrono parses "Today", "this Tuesday", and other human-readable dates into a JavaScript Date object. They provide a function that did exactly what I needed:

// Give me the Date object of the first 12PM Wed in the future
// https://github.com/wanasit/chrono#parsing-options
chrono.parseDate("12PM Wed", new Date(), { forwardDate: true });
// Wed Nov 4 2020 12:00:00 GMT+0000 (Coordinated Universal Time)

Once I had JavaScript Dates for streets with multiple cleanings, I compared them and found the closest time. Now we could create a calender reminder.

I quickly found another problem: I might move the car days before street cleaning is scheduled. But that reminder is still on my calendar, and it'd tell me to move my car for the street I parked on days ago. I don't want to manually delete old reminders each time. I needed a way to automatically delete them when I moved my car.

I ended up tagging new calendar events with metadata to identify them as "move car reminders". This way, the Pipedream workflow could find and delete all existing move car reminders before creating a new one. Google Calendar allows you to add this metadata as extended properties:

The fields of the Events resources cover the most common data associated with an event, such as location, start time, etc, but applications may want to store additional metadata specific to their use case. The Calendar API provides the ability to set hidden key-value pairs with an event, called extended properties. Extended properties make it easy to store application-specific data for an event without having to utilize an external database.

In code, you can just attach this metadata in the create event API request:

{
  extendedProperties: {
    private: {
      streetCleaning: true
    }
  }
}

Finally, I add my wife to the invite, add reminders 24 and 12 hours ahead of the cleaning time, and add the location of the car, which you can view in Google Maps.

Wrapping up

These real-world data problems can be more challenging than typical automations, but they're rewarding to solve. I couldn't have done this without SFData - huge thanks to them for collecting and exposing the cleaning schedules.

Let me know what problems you're automating in the comments!

I wrote this article with VS Code, on my Mac, and published it by running git push. All my DEV posts are tracked by Git, and I can manage them from my local machine. Learn how to do this yourself!

Publish DEV articles from a Git repo, with Github + Pipedream

Dylan J. Sather ・ Jun 10 ・ 7 min read

#pipedream #github #meta #git

Show your most recent DEV post on your Github profile with Pipedream

Dylan J. Sather — Sun, 12 Jul 2020 15:58:08 +0000

Github just launched READMEs for your Github profile. This is pretty powerful. Since you can add any Markdown to a README, you can now add images, links, and all sorts of other content to your profile. Check out Monica Powell's profile, for example:

What if you could take this a step further, and update your profile automatically in response to specific events? For example, what if you could always display your most recent tweet on your profile? What if you could list the Spotify track you're listening to right now?

To test this idea, I'm going to show you how to display a link your newest DEV post on your profile:

Each time you publish a new post, this Pipedream workflow will automatically update the link on that README. Pipedream is an integration platform for developers, and it's great for building serverless, event-driven workflows like this. Pipedream workflows are written using pre-built actions and custom Node.js code, each of which can connect to hundreds of API integrations. Workflows run for free.

Let's get started!

How this works

First, add a README to your Github profile. Stefan Natter published a great article on how to set that up:

How to add a README to your GitHub profile

Stefan Natter 🇦🇹👨🏻‍💻 ・ Jul 9 ・ 1 min read

#github #tutorial

Pick a place in that README where you'd like to display your most recent DEV posts. In my README, I've included this Markdown:

## DEV blog

I blog on [DEV](http://dev.to/). Check out my most recent post:

<!-- dev -->
<!-- devend -->

Notice the comments:  and . This is where the link to your newest DEV post will be added. Put these comments wherever you'd like the link to live, and push that to your profile README.

Visit https://pipedream.com and press the Sign In button in the top-right to sign up for a Pipedream account:

Once you've signed up, open this Pipedream workflow. Click the Copy button near the top-right to create a copy of the workflow in your Pipedream account.

Once you copy the workflow, you'll be asked to enter information specific to your DEV account and your Github profile. The first step of the workflow is the trigger step. This workflow is triggered on new posts in an RSS feed. DEV generates an RSS feed for every user's posts at the URL https://dev.to/feed/{username}, so we'll use this to track new posts.

Replace {username} with your DEV username, and enter your DEV RSS URL in the Feed URL section of this step. By default, Pipedream will poll your RSS feed for new items every 15 minutes, which you can change:

The next step asks you to enter an Owner and Repo where your profile README lives. Enter your username in both fields:

The remaining steps fetch the current version of the README, inserts the link for the newest post within the comments, and updates the README with the newest content, pushing the newest link live. Since these steps interact with the Github API, you'll need to connect your Github account to the relevant steps. Scroll through the workflow and press the Connect Github button in each step that requires it:

Now, your workflow should be ready to run. Let's test it. At the top of your workflow, in the trigger step, you should see a sample event for the most recent post. You'll also see a Send Test Event button. Press it:

This should run your workflow, adding the link to your profile:

One final step - by default, the trigger step is turned Off so you can test your workflow without events triggering your code until you're ready. Now that the workflow works, turn it On:

And that's it. New posts should trigger the workflow and update your profile automatically.

You can extend this idea to any data source, updating your profile when that data changes. Pipedream event sources allow you to trigger workflows on any event from any API, so you could trigger a profile rebuild on new tweets, new Github stars, changes to an Airtable table, and more. Try creating another workflow to see what else you can build, and share your workflows (or questions) in the comments!

Publish DEV articles from a Git repo, with Github + Pipedream

Dylan J. Sather ・ Jun 10 ・ 7 min read

#pipedream #github #meta #git

Publish DEV articles from a Git repo, with Github + Pipedream

Dylan J. Sather — Wed, 10 Jun 2020 21:18:35 +0000

I wrote this article with VS Code, on my Mac, and published it by running git push. All my DEV posts are tracked by Git, and I can manage them from my local machine.

Below, I'll tell you why I set this up, and show you how easy it is to configure for your own posts.

How this works

I'll show you the finished product first so you see how this works end-to-end.

First, I create a new Markdown file:

---
title: My First DEV Post
---

Hello, world!

and push it to Github:

git add my-first-post.md
git commit -m "Adding first post"
git push

As soon as I push, this Pipedream workflow creates a draft of that post in my DEV account:

I can edit and git push again, and the draft will update. When I'm ready to publish, I add published: true to the top of my YAML front matter:

---
published: true
title: My First DEV Post
---

Hello, world!

Then run:

git add my-first-post.md
git commit -m "Publishing first post"
git push

and it's published:

How cool is that!

Why manage DEV posts in a Git repo?

I like writing code in VS Code, with all its keyboard shortcuts, plugins, and other goodies I've added to make it my own. When I write Markdown in the DEV editor, I miss my local setup. That's the primary reason I set this up: I get to write articles in my own editor, with my own shortcuts and tools.

But storing your articles in a Github repo carries other benefits:

Every change you make to an article is tracked by Git. You can compare the changes you made between versions, or revert to an older version. If you make your repo public, anyone can open a pull request to fix typos, broken links, and more.
You can run Git hooks or Github Actions to automate basic tasks: for example, you could run a script to validate your Markdown or spell check it before your change gets commited your repo.
You can trigger other, more complex automations on every git push using Pipedream workflows. For example, once you publish your article to DEV, you could automatically post its link to Twitter.

How to set this up

To get started, you'll need a Github account and a DEV API key.

First, create a new Github repo to manage your DEV posts:

Then clone the repo locally:

git clone git@github.com:[YOUR_USERNAME]/dev-to-posts.git

Next, you'll configure a Pipedream workflow to publish your articles using the DEV API.

Pipedream is an integration platform for developers. For this flow, Pipedream works like Github Actions: each time you push new Markdown files to your repo, the workflow runs. Pipedream workflows are written using pre-built actions and custom Node.js code, each of which can connect to hundreds of API integrations. You can run this workflow for free on Pipedream's free tier.

Visit https://pipedream.com and press the Sign In button in the top-right to sign up for a Pipedream account:

Once you've signed up, open this DEV workflow and click Copy near the top-right. This creates a copy of my workflow in your account, that will run for your DEV repo.

First, you'll be asked to configure the Trigger step. This workflow runs every time Markdown files are added or modified in your repo. You'll just need to connect your Github account and select your repo from the dropdown menu:

Then click Create Source at the bottom right of the trigger step. This will configure a webhook in your Github repo that notifies this workflow anytime a push happens.

You'll also need to turn this trigger step On to make sure it runs automatically on new pushes:

The next step of the workflow - create_and_update_dev_posts - runs Node.js code to push this Markdown to the DEV API. If you add a new article to your repo, the workflow creates a new DEV article. If you're pushing an update to an existing article, the workflow updates that DEV article.

To get this step working, you'll just need to connect your DEV API key. Press the Connect Account button near the top of this step. Like in the trigger step, this will prompt you to enter your DEV API key:

Now you're ready to push your first article. Create a new Markdown file in the Git repo you cloned above, adding the following contents:

---
title: My First DEV Post
---

Hello, world!

Add, commit, and push this file to Github:

git add my-first-post.md
git commit -m "Adding first post"
git push

As soon as you push these changes, you should see a new event appear in your Pipedream workflow:

This triggers the workflow, creating a new draft article in DEV:

To publish the article, add published: true to the YAML front matter section at the top of your file:

---
title: My First DEV Post
published: true
---

Then run:

git add my-first-post.md
git commit -m "Publishing first post"
git push

You should see your brand new article show up, published, in DEV:

To unpublish, set the published flag to false:

---
title: My First DEV Post
published: false
---

Since you own this copy of the Pipedream workflow, you can modify the code in any way you'd like. You could update the DEV step to change how articles get published, or add a step to send a message to Slack, or post a tweet, any time you publish a new article. If you make any edits, share them in the comments below or publish your own article about it!

The rest of this post addresses other details of the integration, like managing images in articles, and other YAML front matter you can use to change details of your posts.

Making changes to articles in the DEV UI

Any change you make to articles in the DEV UI will get overwritten unless you also make those changes to the file in your repo.

How to include images in posts

Currently, DEV doesn't have an API for uploading images, so I'm using Cloudinary to host mine, referencing the Cloudinary URL in my Markdown:

![My image](https://res.cloudinary.com/dkbxegavp/image/upload/v1590355743/dev.to%20posts/dev-to-draft_bmqlgb.png)

In theory, this workflow could be changed to reference local images in your article Markdown:

![My image](./images/my-image.png)

Once you commit the images to your repo, the Pipedream workflow could upload the images to Cloudinary directly, and change the article references to point to the hosted version, instead.

Remember — the Pipedream workflow you copied above is yours to modify. You can add any logic you'd like to that workflow, uploading images to Cloudinary or another hosting provider, or include anything else custom to your publishing process.

Default branch only

The workflow processes commits on the default repo branch only (master, unless you've changed it). Commits to other branches are ignored, until they're merged to your default branch.

One Markdown file : One post

Each Markdown file you push to your DEV repo generates its own DEV post. Your files must have a .md extension for them to be processed. Any non-Markdown files are ignored by the workflow.

Front Matter

The Pipedream workflow uses the same YAML front matter that DEV supports to update the published state, title, tags, and more.

None of the front matter variables are required. If you include no front matter, your article will be saved to DEV as a draft, with a title based on the filename (see below).

Titles

The title front matter variable is always used if present.

If title is absent, the workflow will use the filename of the article, converting hyphens and underscores as spaces. Case is preserved:

my-first-post.md -> my first post
My_First_Post.md -> My First Post
My First Post.md -> My First Post

If you rely on this convention and then later want to set the title front matter variable, the workflow will update the title according to the variable's value, and use that instead of the filename moving forward.

The simplest way to run Node.js code on a schedule

Dylan J. Sather — Fri, 03 Apr 2020 17:58:17 +0000

I remember the first time I discovered cron. I’ve been obsessed with automation my whole life, and as a fledgling developer cron became my workhorse.

At the time, I worked at an IT help desk. I used cron for email reminders, automated reports, backups: anything I could schedule, I did.

cron is simple: you just tell it what you want to run, when you want to run it:

0 0 * * *  node script.js  # run script.js once a day

But it comes with tradeoffs: primarily, you have to operate a server to run it, which costs money and time. If you don’t already have a server where you can run jobs, setting one up just for cron is far from ideal.

In the era of the cloud, it’s also outdated. Cron jobs are simple only if you cut your teeth on Linux and can effectively administer a server. This is an inaccessible option for many modern devs who operate far up the stack.

So it’s no surprise that we have other choices for scheduling code in 2020:

These options work well in their intended context: Lambda is great when you’re glueing together AWS resources. Kubernetes CronJobs or ECS Tasks are ideal when your code runs in a container. Airflow is great when you have complex dependencies between jobs, for example in data pipelines.

But when you just want to run a script on a schedule, they’re overkill.

Enter Pipedream

Pipedream is an integration platform for developers. You run serverless workflows - any Node.js code - triggered by HTTP requests, timers, emails, and more. You can also use pre-built actions to connect to hundreds of APIs and apps - actions are just Node functions that perform common operations against these APIs.

You can sign up and run any Node code on a schedule in less than one minute. Watch this video or follow the step-by-step instructions below:

First, install the Pipedream CLI:

curl https://cli.pipedream.com/install | sh

cd into a directory with a Node script you’d like to run, or just create a simple one-line script:

echo 'console.log("Hello, world")' > cronjob.js

then deploy that script to Pipedream, running it every 15 seconds:

pd deploy --run cronjob.js --timer --frequency 15s

This step will prompt you to sign up for Pipedream if you haven’t already. The pd CLI will deploy your code to Pipedream, and print logs as the script produces them. You can press Ctrl-C to quit the real-time stream, and listen for new logs later by running:

pd logs cronjob-js

You can delete this job and all its logs by running:

pd delete cronjob-js

You can list all running jobs with pd list, update their code or schedule with pd update, and more. See the docs to learn more.

A practical example: send an HTTP request on a schedule

If you already host code at some URL, and just want to trigger a job to run via HTTP request, you can use Pipedream to send that request on a schedule. In this example, we’ll hit the Star Wars API once a day.

First, open your editor and create a file called http.js with the following contents:

const axios = require("axios");

const resp = await axios({
  method: "GET",
  url: `https://swapi.co/api/films/`  // replace with your URL
});

console.log(resp.data);

Then run

pd deploy --run cronjob.js --timer --cron “0 0 * * *”

This will make a GET request to SWAPI, once a day at midnight, but you can modify this script to send any HTTP request, at any schedule. See our guide on making HTTP requests in Node for more examples.

See the README on cron jobs or the Pipedream docs to learn more about the platform at large.

We ❤️ Feedback

We're eager to hear your feedback. We're shipping new features every day - check out our roadmap and reach out anytime via email or Slack to let us know what we can improve.

Analyzing Github Issue Reactions

Dylan J. Sather — Wed, 27 Nov 2019 19:45:18 +0000

Prioritizing issues with 👍s, ❤️s and 🎉s

At Pipedream, we use Github Issues to track feature requests, bugs, and new app integrations.

Anytime a user has a new idea, we send them to the roadmap to make sure the idea gets captured.

Anytime someone suggests an idea that’s already been captured, we send them to the roadmap to add a reaction (a 👍, ❤️ or 🎉) to the issue.

We’re diligent about pushing people to the roadmap because we want to prioritize the most requested items. Reactions are the best way to collect that data.

Unfortunately, Github doesn’t provide high-level dashboards on issue reactions. You can sort issues by the total number of reactions in the Issues UI:

is:issue is:open sort:reactions-desc

but you can’t see the reaction count without digging into the issue itself:

Nor can you compare the count of reactions across issues.

To help us prioritize the right issues, we needed to answer question like:

What are the top issues this week? (What should we be focused on?)
What issues are trending this week? (Even if it’s not in the top 5, is there an issue getting lots of love this week we should be paying attention to?)
Who’s opening the most issues? Who’s reacting to the most issues? (Who are the most engaged users, and how can we prioritize their issues and get more feedback?)

We built a workflow to collect this data, and a Google sheet and Jupyter notebook to drive the analysis. I’ll show you how this works and how to use it for your own repo.

Pulling Issue Reactions, saving to Google Sheets

This Pipedream workflow pulls reactions for all open issues in your repo once a day, saving them to a Google Sheet where can run more analysis:

This gives us the basic data we need to run analysis on reactions by issue, author, and more.

Follow the instructions in the workflow’s README to connect your Github and Google Sheets accounts, and enter the necessary values in the fields of each step (for example, the Github repo and spreadsheet you’d like to save data to).

Once that’s done, press the Run Now button to collect your first set of reaction data:

This workflow uses some built-in Pipedream actions to save data to Google Sheets. But when you forked the workflow, you created a copy you can modify however you’d like.

For example, you can swap out the Google Sheets steps if you’d like to save data to a database, Airtable, or any destination (you can use any pre-built actions or run any Node.js code).

How we make requests to the Github API

I use the amazing octokit/rest.js package to facilitate interaction with the Github API. This sets the necessary HTTP headers to enable the reactions API (it’s still in preview), and handles pagination and retries transparently.

When the workflow runs the fetch_issues_reaction_data step, Pipedream provides a fresh OAuth access token in the variable auths.github.oauth_access_token that you can use to authorize requests (read more about connected accounts in the docs).

This all means you can fetch any data from the Github API with just a few lines of code:

const Octokit = require("@octokit/rest").plugin(
  require("@octokit/plugin-retry")
)

const octokit = new Octokit({
  auth: auths.github.oauth_access_token,
  previews: ["squirrel-girl-preview"] // See https://developer.github.com/v3/previews/#reactions
})

# Retrieve owner and repo name from the form params passed by the user
const { owner, repo } = params

let options = octokit.issues.listForRepo.endpoint.merge({
  owner,
  repo,
})

const issues = await octokit.paginate(options)

Exploratory Analysis in Google Sheets

I like to explore my data with a pivot table before ever jumping into a more complex analysis with SQL or a Jupyter notebook.

Pivot tables support grouping, aggregate functions, sorting, and more, all in a friendly GUI.

What issues have the most reactions?

It’s nice to have a single, ordered table of the top issues.

What issues have seen the most reactions in the past week?

We need to see what issues are trending. If it’s not a large project and can be tackled quickly, fixing a trending issue proves our responsiveness to users and helps build trust.

Reactions by User

It’s nice to know who our top “reactors” are, so see if that correlates with engagement in Slack or other channels.

A deeper dive with a Jupyter notebook

I wanted to observe how the cumulative sum of reactions for an issue move over time. This helps us understand whether the top issue recently moved to the top, or whether it’s been the top issue for weeks.

I was also interested to analyze the number of total reactions over time, to see if it’s increasing, on average, or whether it correlates with dates when we’ve promoted our roadmap in our Slack community.

As a new product, the number of reactions per day is small, so this doesn’t tell a rich story yet. You can help us by using Pipedream and 👍 your favorite issues!

I created these charts with pandas and matplotlib, using a a Jupyter notebook. You can see the code in this Github repo.

Extending this analysis

There are a number of other questions that would be interesting to analyze in the future:

Under what conditions are different types of emojis ( 👍, ❤️, 🎉, and more) used? Does the use of a specific emoji correlate with issue closure or activity?
Does an issue get attention on a project after a certain number of reactions?
Do issues with labels (or a specific label) get more reactions than issues with no labels?

Questions like these have been explored in research papers, so there’s a lot of prior art you can read up on to get ideas to apply to your own repo.

Let us know what kind of analyses you end up doing in the comments below!

This article was originally posted on Medium

Airflow is overkill

Dylan J. Sather — Tue, 29 Oct 2019 17:42:09 +0000

Originally published on Medium.

Last year, I taught a Data Analytics bootcamp at UC Berkeley Extension. In 6 months, students who'd never programmed before learned Excel, Python, pandas, JavaScript, D3, basic machine learning, and more.

What would you guess was the toughest thing for most students to learn? Machine learning? JavaScript and D3?

Deploying a Flask app to Heroku.

I empathize. When I first learned to program, and got through the pain of setting up my environment, it was easy to iterate. I used a REPL to experiment:

>>> name = "World"
>>> print(f'Hello, {name}')
Hello, World

or wrote a script that I could quickly run and test on my local machine:

$ python hello.py
Hello, World

Now I'm asked to deploy an app to a service like Heroku. Suddenly, I need to learn:

git
The Heroku CLI
The Procfile
and Gunicorn

What's more, each deploy can take minutes, so each mistake takes longer to troubleshoot and fix.

I dealt with the pain of deploying apps before Heroku, so I appreciate its simplicity. But my students lacked that context, so Heroku seemed needlessly complex.

Airflow was my Heroku

At Pipedream, we use Airflow to run scheduled jobs.

When I first used Airflow, I needed to run a simple Python script on a schedule. I wrote that script in 15 minutes. Then I setup Airflow. This involved:

Reading the README for the Helm chart (we use Kubernetes)
Realizing I failed to include some specific config, trying again
Realizing there was a typo in the documented config, trying again
Setting up some Kubernetes secrets
Troubleshooting a PersistentVolumeClaim
Setting some env vars and airflow.cfg config
And lots of deep dives into the Airflow docs and StackOverflow

I felt a lot like my students did with Heroku. I just wanted it to work, but had to learn a handful of new concepts and tools all at once.

Now, I love Airflow. I appreciate its dependency management, backfill, automatic retries, and all the things that make it a great job scheduler. But I didn't need any of that for my original use case. I just wanted to run a cron job.

cron has limitations. It doesn't have built-in error handling or retry, and has to be run on a machine that someone has to maintain. Modern job schedulers improve on it substantially, but lose its simplicity.

Keep it simple

As a teacher and developer, I care very deeply about improving developer tools. I joined Pipedream for this reason.

When we built the Cron Scheduler, we tried to marry the simplicity of cron with a powerful programming environment. I believe it's the easiest way to run a job on a schedule. There's no infra or cloud resources to manage, and it's free.

When I schedule a job on Pipedream, I:

Create a workflow
Set the schedule
Write the code

I created this 1 minute video to show you how this works:

Every workflow on Pipedream begins with a trigger: HTTP requests, emails, or a cron schedule.

After I select my trigger, I add steps: Run any Node.js code (Python coming soon), send an HTTP request or an email, or interact with the APIs of built-in apps like Slack, Github, Google, Reddit, Shopify, AWS, and more.

Standard output and error logs appear directly below the step that produced them. If a job fails, I'm notified via email, and I can replay that job in one click.

Workflow templates are also public. I can share them with anyone, and they can fork, modify, and run them their own accounts.

Take a look at these examples and try running one on your own:

We ❤️ feedback

Our beta release is our first effort at making job scheduling and workflow management easy, but we want you to use the product and give us honest feedback about how it can improve.

We'd love if you joined our Slack community and added new feature requests on our backlog. And you can reach out to our team anytime. We'd love to hear from you.