DEV Community: Transposit

Embracing the SaaS unbundleverse

Transposit — Wed, 15 May 2019 20:56:21 +0000

Written by Adam Leventhal

Previous eras of software consumption were defined by their bundles. Our current era is defined by its diverse application ecosystem, its unbundling. Even recently, corporate consumers wore their fealty to a vendor with pride: we’re a Microsoft shop, we’re an Oracle shop, we’re an IBM shop. While devoted, the relationship wasn’t monogamous. The right tool for the job, but guided by relationship as the north star. Bundling had the benefit (at least perceived) of interoperability — many tools, intermeshed, helping us work well as a team.

SaaS changed both the economics of software, and the adhesion of the bundle. Even if you wanted to be a mono-shop, it’s harder to see how to do that (or why you’d want to). Teams are picking the tools they want. Applications are able to be deeper and narrower than ever before. Consequently everyone has many more applications from a much more distributed collection of vendors.

Established and new companies are repositioning for this distributed, diverse unbundleverse. We started Transposit to help build the apps composed of the APIs from unbundled software. We’re betting on this trend. But I heard a new vision for this modern era of software a couple of weeks ago at Slack’s Frontiers conference: that there can be hubs even amid unbundling, centers in an era of decentralization. Rather than grouping like-applications, these hubs group like-activities. In an unbundled world, the monopolies will be horizontal, aggregating similar kinds of interactions across applications.

Slack on Slack

At the conference, Slack laid out a significant and compelling vision about becoming a major epicenter amid this sectoral unbundling. While the conference felt bold and enlightening, Slack’s summary blog post was more pedestrian: mostly new features and non-specific enthusiasm. It listed shared channels, Workflow Builder, and Gmail/Outlook gateways as some big pieces. Under the heading “Building the tools for modern work”, the video featuring CPO, Tamar Yehoshua, touches on a common theme of the conference:

“But communication at work doesn’t just happen between people. We need access to data and other workflows. A typical enterprise uses more than 1000 cloud applications. These software tools fragment attention and make it harder for information to flow across an organization. So we want to do more and more to connect Slack to the critical tools behind every business.”

1071

To anyone at Frontiers, the number 1071 will feel at least familiarly significant. I counted it in no fewer than 6 different talks by various members of the Slack ensemble. It’s the average number of cloud applications in an enterprise. (Apparently from this Netskope report.) This was the strong theme of the conference: companies have tons of SaaS applications and Slack is the hub for collaboration, approvals, workflows, etc.

Not sure which session; could have been any of them.

As an aside, Slack’s famously humble, empathetic culture was also on display. The second most common theme was speakers thanking the audience for laughing — just take your yuks and move on folks! (And we have the perfect hype man!)

You’re welcome.

Apps form silos. Silos impede cross-team communication and collaboration. If a task needs approvals from a bunch of folks in a bunch of tools, it’s going to be faster if it happens in Slack. If people use tools infrequently, offer common tasks through Slack. And rather than gathering data from multiple apps, bring it all together in the tool everyone’s already got open, Slack. It’s the aspiring epicenter of work.

Hub at the center of the unbundleverse

SaaS has fractured the traditional bundles. The new monopoly opportunity is horizontal. Rather than taking over collections of applications, these hubs will own common tasks across the landscape of apps. Slack wants to own collaboration and lightweight interactions across apps. It’s a monopoly that embraces a sprawling ecosystem rather than trying to own it. These catalyze expansion and heterogeneity where bundles sought to contain it. Every major software vendor is looking for their hub opportunity.

How will Slack become this hub? How will anyone? In their S-1 (published the day after Frontiers), Slack cited several components of their growth strategy. The first 6 are different aspects of sales and marketing. The 7th addresses their platform vision (and could be copied almost verbatim from other aspiring hubs):

Grow our application platform and developer ecosystem

We will continue investing to expand the number of developers building applications that integrate with Slack and to make Slack work with an increasing number of third-party and internally developed custom applications.

(The 8th and final growth strategy is “Leverage artificial intelligence, machine learning and advanced search”, a shibboleth of our times I suppose.)

For any hub, there’s the core platform, the surround of apps and APIs, and a yawning gap in between. Platforms and users alike are dependent on developers to fill that divide, to write the software that wires together hub and edge. From Zapier, to Airtable, and Transposit, that’s why we’re here. As much as the hubs will define our future so will the developers, tools, and platforms that bind it all together.

—

Building Slack bots or other connections between hub and edge? Try Transposit.

Superglue for Gaps in Your AWS Infrastructure — automation scripts using SQL and Javascript

Transposit — Tue, 07 May 2019 21:50:36 +0000

Written by Yoko Li

Superglue for gaps in your AWS infrastructure — automation scripts using SQL and Javascript

Introduction

I’m a big fan of AWS, but I wish there was a bit more “glue” within the AWS system itself, and between AWS and services like Slack and GitHub. An example of the sort of glue I’m looking for: a mechanism for defining a more complex lifecycle policy, where I could write some code to control resources I want to delete on AWS.

To solve this problem, we built a Transposit app that purges unused resources on ECR and ECS and then posts the results on Slack so the whole team gets notified. This enabled us to ditch the lambda function we built previously that was hard to debug and iterate on.

Before diving in, a little background. Our deployment works like this: on every build success we deploy to our test environment, under the hood these things happen:

Build a new image based on latest code, and put that image in ECR
Create a new task in ECS, and tell the task to use the image we just created

The problem with this system is that because each build creates ECR images and ECS tasks, and both ECR and ECS provide limited amount of storage, we eventually need to clean up old resources so our build doesn’t fail from running out of space.

The old world with Lambda

To clean up old tasks and images, our earlier solution was to write a lambda function. A lambda function let us have custom logic to check what’s currently in use by ECS tasks and delete images and tasks not in use. A problem arose when the function failed without us knowing; we hadn’t built in active notifications so didn’t realize it failed. This led to unused resources piling up, and one day, caused our build to fail. The build kept failing until we freed up some space in ECR.

Not only was it failing, but it actually didn’t do everything we wanted: it didn’t account for inter-account dependencies on our ECR repositories and was deleting more images than it should. Besides, based on the failures above, it would be nice to have notifications!

In our lambda, the code to figure out which images are in use is complicated. It needs to use a pagination token to get a full list of ECS tasks, iterate through them, and find out what tasks and images are both old and not currently in use. It’s a pain to have to do that pagination by hand and look up how to use each AWS API. I won’t bore you with the details of how it works, but you can click through the screenshots if you want to see all the code.

Rather than investing more into this complex lambda function, we rebuilt the functionality in a Transposit app. Based on what we learned from building on lambda, we built an app called cleanup_task using SQL and Javascript on Transposit.

The new world with Transposit

To make sure we don’t delete any image in use, we have to first find out which images are currently being used. Here is how I built it in Transposit:

The code below gets all the ECS tasks that are currently in use by iterating through all active services within our cluster and looking at task definitions.

We can then use the above operation (which we call get_tasks_in_use ) to get the images that those ECS tasks are attached to.

Once we have all the images we want to keep, we iterate through all the images in an ECR repository, and delete the ones that are both old and no longer in use. We call GitHub to verify image commit time, since each image was built from a single commit:

Get notified!

I mentioned earlier that it would be helpful to build Slack notification on top of this project, which posts to a Slack channel about what has been done to AWS resources. Trying to build this into a lambda function is a struggle: I would need to setup OAuth from scratch and figure out how to use Slack APIs. With Transposit, I was able to build this with very little code:

The output is kind of raw, but it tells me what I need to know. When it fails, it also tells me that.

With our old lambda I would lose sleep worrying about whether our build would fail. I’d obsessively check out Cloudwatch logs to make sure nothing had gone wrong. The new Transposit version is easier to understand, easier to write, easier to extend, and tells me when things fail. I still get the benefits of using scheduled tasks to run it and not needing to worry about the underlying infrastructure. I now have the confidence that if something fails I’ll be able to address it before it becomes a problem. Each new feature — Slack notifications, GitHub integration, event queuing through AWS Dynamo — has become much quicker to add.

You can fork the full version of this app here if you want to see how it works and try it out against your own infrastructure. We also made a sample app for purging objects stored in S3, which you can find here.

For the next bit of glue you need to hold your AWS infrastructure together, try out Transposit.

Google Hire Has An API. Finally.

Transposit — Fri, 12 Apr 2019 15:46:00 +0000

Written by Adam Leventhal

We had just convinced our Seed round investors that Transposit was a big idea. A big idea needs a team to build it, so when the first round closed we started hiring that team. We’d talk to candidates and write down our feedback to share with the team. At first we used Google Docs. The price was right (free), but we quickly outgrew it: we needed more structure and access control. My experience with application tracking systems (ATS) is that they’re all infuriating in their own way, imposing their structure on your process rather than adapting to it. So if we weren’t going to get what we wanted, at least we didn’t want to pay a lot for it.

Enter Google Hire.

Hell is other people… trapping your data with no API

As anticipated, Google Hire wasn’t exactly what we wanted. What it was though was cheap. It’s not a terrible product; it just doesn’t do anything quite the way we’d like. Filter by recruiter, handoff candidate ownership, find folks we haven’t pinged in awhile — Hire didn’t do any of those things.

No problem, right? That’s what APIs are for. Your service might not suit my needs, but an API lets me work around the parts I hate and build the extensions I need. Even if the product improved to meet our needs, there would always be a next need. But — amazingly for a Google product[1] — there was no API.

So we suffered. And in the meantime we hired our team and built our product that — with some irony — made it easy to work with APIs. By a happy coincidence, Google Hire released their beta API right around the time Transposit opened our beta. Transposit puts a SQL and JS interface in front of APIs, manages auth, and makes it easy to build API-driven APIs. Here are the first couple of queries I ran against the Google Hire API. (You can try everything out live once Google gives you API access.)

I can list all the candidates we’re tracking:

SELECT \* FROM google\_hire.list\_candidates
  WHERE tenant=‘my\_tenant’

I can see the past employers of all the folks we’re talking to:

SELECT employmentInfo.employer FROM google\_hire.list\_candidates
  WHERE tenant=’my\_tenant’ EXPAND BY employmentInfo

(Amusingly in the context of this post, the top hit is Google)

Solving problems

After being held at arm’s length for far too long, I had a lot of fun exploring the newly public(-ish[2]) API. The first real problem I wanted to solve was to show the candidates whom we haven’t pinged recently. Why is this not built in? I have no idea. It was easy enough to get to with Transposit. We treat each API like a view or table within a database; getting the data I wanted amounted to a JOIN between the candidates and applications tables:

https://medium.com/media/0103d33ee62c48d82b569ab9af8c2747/href

[
  {
    "name": "tenants/abc/candidates/def",
    "job": "tenants/acb/jobs/ghi",
    "candidate": "tenants/abc/candidates/jkl",
    "createTime": "2019-04-02T00:31:48.630Z",
    "status": {
      "state": "ACTIVE",
      "processStage": {
        "stage": "Initial Screen",
        "reportingCategory": "SCREEN"
      },
      "updateTime": "2019-04-03T05:27:29.321Z"
    },
    "personName": {
      "givenName": "David",
      "familyName": "Lightman"
    },
...

I can focus that in on a particular job by adding another SQL filter:

https://medium.com/media/1f3bac392e9c743010c9b25872a34158/href

… and then get the data I want by plucking out particular fields and adding an ORDER BY:

https://medium.com/media/1b85f16bf4ff02fa2f40b489465d7810/href

[
  {
    "updateTime": "2019-03-12T17:25:33.176Z",
    "givenName": "Michael",
    "familyName": "Jennings"
  },
  {
    "updateTime": "2019-03-21T20:33:08.454Z",
    "givenName": "Kate",
    "familyName": "Libby"
  },
...

Finally! The data I’ve wanted since we first started using Hire!

All Hail Our Benevolent Slack Overlords

Slack is rapidly becoming the central interface across so many tools we use, so I wanted the data from Google Hire connected up with Slack. Building a Slack slash command is a snap (check out the Transposit Quickstart). You just need to point Slack at a webhook that returns the data you want:

https://medium.com/media/5aed6a0835a234647aaba99de0820df2/href

Now when I worry if we’ve dropped the ball on a candidate and they haven’t heard from us in a while, I can just type my Slack slash command:

As excited as I was for the Google Hire API, the endpoints in the first beta are pretty basic. It will be extremely powerful when they open up the ability to do things like create new candidates, transition candidates between stages of the hiring process, and add comments (the mechanism we use to hand off candidates between members of our team).

An API is a critical feature of any tool we use; I’ll never again make the mistake of choosing a tool that doesn’t give me access to my data via an API. Transposit let me explore the Google Hire API interactively, check out the data, and poke around at different endpoints. You can try it out with the Hire API or any of the other APIs it connects to. And if this sounds like something you’d like to get involved in, we’re hiring:

SELECT title FROM google\_hire.list\_jobs
 WHERE tenant=’my\_tenant’
 AND filter=’state=OPEN’

[
  {
    "title": "Developer Advocate"
  },
  {
    "title": "Growth Marketing Manager"
  },
  {
    "title": "Product Marketing Manager"
  },
  {
    "title": "Senior Site Reliability Engineer"
  },
  {
    "title": "Software Engineer"
  }
]

[1] I call it a Google Product — and I knew this beforehand — but Hire came from the Bebop acquisition that brought Diane Greene into Google. I suspect acquiring an amazing leader at the head of GCP may have been the primary goal.

[2] You need to sign up and get approved for the Google Hire API. I wrote a pleading, but firm mail to our sales rep to nudge things along.

Hypebot is the hype man who never lets me down

Transposit — Tue, 19 Mar 2019 15:36:00 +0000

Written by Sarah Brown

The birth of hype

It’s the middle of our company hackweek, and my hackweek project is not working. I’m trying to convert dwarf fortress release notes into emoji, but I keep hitting rate limits and — what’s maybe more detrimental — there is no emoji symbol for dwarf 🤷. I’m also vastly outnumbered in a Slack poll for what flavored waffles we’re buying for the office.

Salted Caramel Conspiracy

I know what I must do: abandon my hackweek project and write a slackbot that votes Apple Cinnamon. Unfortunately, the Slack API does not allow bots to click buttons 😒. I change tack and instead create hypebot , a bot who reacts emphatically to everything I say in slack.

And Tuan too because he’s starved for hype.

The hype-cycle

Hypebot is built using Slack’s event subscription api, with the backend running on Transposit. You can check out all of hypebot’s code here: https://console.transposit.com/t/transposit-sample/hypebot

The Transposit Console

When hypebot receives a message event from Slack, it splits the string into discrete words and looks each word up in the thesaurus. Then, armed with hundreds of synonyms, it compares each word to all potential Slack emojis. When it finds a match, it reacts. Thus the hype-cycle is complete.

Implementing this in Transposit meant I didn’t need to deploy my own server, and Transposit made it easy to keep track of bot OAuth scopes when making API calls. No infra; no auth; pure hype.

Taking it to the next level

The hype train doesn’t end here. I’ve got upcoming hypebot posts about how we addressed performance problems when things were overhype, and how we can modify hypebot for his eventual debut on the Slack app directory. Of course if you’re too hype to wait, go ahead and fork hypebot on Transposit to build your own slackbot.

Sadly hypebot can’t add claps to a medium post via the api (believe me, I checked), so you’ll just have to do this the old fashioned way: 🙏👏

The Life Changing Magic of Migrating Github repositories with Transposit

Transposit — Fri, 15 Mar 2019 15:31:00 +0000

Written by Nina Yang

I’m not a hoarder. I just happen to manage a ton of GitHub repositories all shoved into one organization. But we need all of these repositories, I swear!

To help developers easily interact with APIs, Transposit has built an assortment of connectors. Each connector lived in its own repository within a single GitHub org. But as our library of connectors grew, our org became more and more cluttered. It was time to KonMari them and tidy up.

Photo credit: twotwentyone.net

The goal

Create a new GitHub org and migrate our ~100 connector repositories over to it. Do this quickly and comprehensively to minimize disruption to developers interacting with these connectors.

The implementation

Of course I wanted to automate this — making processes like these straightforward was one of the reasons we built Transposit! But I was still surprised by how simple it was to translate the task into code.

Step 0: Create a new GitHub org

I did this manually.

Step 1: Fetch the repository names from the existing org

In Transposit, it’s one line of JavaScript

… or one line of SQL

One of the great features of Transposit is that it takes care of the OAuth dance and credential storage for the developer. After storing a client ID and secret it was just a few clicks to allow Transposit access to my GitHub account.

Step 2: For each connector repository, create its namesake in the new org

When we were crowding all our repos into one GitHub org, we used the prefix “app_” to distinguish our connector repositories from other code (our main codebase and supporting projects). Thus, I could use this prefix to identify the correct repositories to migrate and then drop it because it was no longer needed.

I made an operation that created the new repo:

This operation can be invoked as “this.create_repo_in_org”

My main operation applies it to all repos with the “app_” prefix:

Not bad for <20 lines of code.

I hit the “Run” button, and off it went.

Step 3: Realize that I’ve accidentally created all the new repositories as public and run a fix

Some of our connectors are open source, and some are still a work-in-progress and not quite ready to be public. Whoops. Fortunately, with Transposit it was simple to change them back to private.

Whew!

Step 4: Migrate repository contents

I used a python script to copy the repository contents into their new homes. However , while writing this blog post I discovered that GitHub provides an actual repository transfer API I could’ve used which would have taken care of this step, and combining it with Transposit would’ve made the process even easier. Welp… good information for next time.

Step 5: Notify my team to update their local repository remotes to point to the new org

I wrote another script to do this. Again, it would’ve been better to use the repository transfer API, which will warn you about the updated remote and still push to the new repository.

Step 6: Delete the old repositories

Look familiar?

The end

There are other ways this migration could’ve been done. I could’ve manually transferred ownership of all the repos via the GitHub UI, but that would’ve taken forever. I could’ve done the migration using raw requests to the GitHub APIs, but that would’ve required poring over API and authentication documentation, and would’ve created more opportunities for error. There are also GitHub-specific CLIs that would have achieved the same outcome, but necessitated learning syntax that I was not likely to use again.

With Transposit, I used JavaScript and SQL to complete the migration in a few hours. The details of my API calls were abstracted away and I got to focus on the logic of my implementation rather than the minutiae of the setup steps. The process was quick and efficient and now all of our repositories are in their proper place.

Your use case might be different, but your sentiment of wanting to build without getting sidetracked in tedium could be the same. Maybe Transposit can spark joy for you too.

SQL order from API chaos

Transposit — Thu, 07 Mar 2019 20:44:27 +0000

Written by Tina Huang @kmonkeyjam

Developing any application in a world powered by SaaS requires you to work with and cater to numerous APIs. That wasn’t always the case.

When I first started out at Apple as an application frameworks engineer, components that people built would render from a single data source, almost always a local database. But by the time I joined the engineering team at Twitter in 2009, the world had moved to an API-driven ecosystem. At Twitter, we used APIs for internal microservice communication as well as to publicize data to external developers. Having to glue together all these APIs, both internal and external, was pure drudgery that resulted in a lot of spaghetti code. Composing APIs for developers who haven’t been working with them for the past decade can be mind-bogglingly complex.

It’s hard to believe there’s not a good solution to the problem of API composition. There are so many platforms to help you find APIs, make that initial connection, and even call a single API. But there is no dedicated platform that helps weave data from APIs together in a streamlined way — while also handling the numerous operational chores required from modern APIs, like authentication, pagination, retries, and so much more.

Introducing Transposit

This is a layer that needs to exist, and for the past two years, we’ve been building Transposit as the solution to this problem.

Transposit is a zero-ops platform that brings the power of a relational database to the API ecosystem. Our relational engine provides the ability to write SQL and JavaScript to query and transform your data as though each data connection were a virtual table in a single relational database.

While a database has unfettered access to its raw data on disk, APIs protect your data with authentication. To make a relational engine work in this new data landscape, we’ve built in authentication management not only for you as a developer accessing your own data, but also for all your application’s users and their data sources. The result is a seamless way to query data regardless of where it lives.

Transposit abstracts away the details of specific API mechanics and lets you focus on the code that makes your application unique. It does this by allowing you to easily express your intent in a higher-level language of joins and filters, then translating that intent into a number of optimized API calls. It deals with all aspects of communicating with modern APIs such as retries, re-auth, and pagination. It implements security best practices so that you don’t have to. The UI provides an interactive and playful way for you to explore your data as you build your application. For example, teams often use Slack to communicate and triage issues that happen during production incidents. So let’s say you want to make it easy to file a bug in Jira and attach some production logs stored in AWS Cloudwatch, all from within your Slack triage channel.

A Transposit webhook operation that files an issue in Jira with Cloudwatch logs

To do this without Transposit, you would need to learn the intricacies of the Jira, AWS, and Slack APIs. You’d also need to host a server to receive the Slack webhook, securely store credentials, and implement a bunch of tedious logic. All this work makes it too daunting and too frustrating to build — even though this would be an amazing time saver during an actual incident.

Enter Transposit: now you have the ability to rapidly compose logic across multiple APIs without having to handle any of the API complexities. You can quickly build an application and have Transposit manage all of the deployment details. The joy of Transposit is that it’s fast and fun to turn your idea into an application, and it’s also easy to debug, improve, and share it with others.

The Transposit UI in action

Today, we’ve opened a public Beta for Transposit. We’ve also raised $12.2M in Series A funding, which gives us the resources we need to grow and engage with the community.

Tell us what you wish you could build with APIs, and what’s holding you back. What is the most annoying challenge you have to deal with? How is your creativity being hampered by API drudgery? We hope you’ll sign up for our Beta, tinker around, and let us know what you think — and what you build!