DEV Community: Prismatic

How to Prioritize Integrations for Your B2B SaaS Product

Kristin Ides DeMar — Tue, 21 Feb 2023 22:46:02 +0000

One of the most frequent integration questions we hear from SaaS teams is, "Which integrations should we build first?"

This question often comes up as they're getting started with our embedded integration platform (since it substantially increases how quickly they can launch new integrations), but it's relevant no matter how you build integrations to the other products your customers use.

Outside of core product development, integration development may be the most important thing you can do to define the future of your product.

You may have identified dozens of integrations you'd like to build, but which ones need to come first? Sales can give you all sorts of data about which missing integrations impact the pipeline. Your success team has a list of customer requests they would like to deliver. Engineering has a feel for the level of effort and knows which ones it would like (or not like) to build.

Similar to setting the roadmap for your core product, you need a way to determine which integrations to build first and which can wait. To help you with that decision, let's look at common strategic and tactical approaches to setting integration development priorities. But first, let's recap why integrations are so important.

Why are integrations important for SaaS?

With the average mid-market company using 137 different software tools, it's no surprise that integrations are increasingly important. Without integrations, apps turn into data silos. That, of course, is inefficient, making product integrations essential for the modern software ecosystem.

You are already familiar with SaaS (software as a service) companies like Slack and Hubspot that have leveraged integrations to make themselves indispensable to their customers while growing at phenomenal rates.

In today's market, with few exceptions, a successful B2B SaaS product needs integrations – often quite a few. Let's look at a few critical strategies for prioritizing your SaaS integrations.

Strategies for prioritizing integration development

We have found three common strategies that successful SaaS companies use – often in conjunction with one another – to determine which integrations to build first:

Which integrations are most needed by customers and prospects?
Which integrations will extend your product's functionality and SAM?
Which integrations will increase your competitive advantage?

Which integrations are most needed by customers and prospects?

Depending on your product, you may have a largely homogeneous customer base, or your customers may have substantially different integration needs based on the other software they use. That can vary greatly based on company size, industry, sophistication, and region.

Some situations are straightforward. Perhaps 40% of your customers use the same third-party accounting system, with the other 60% spread over a score of different accounting systems. Integrating your product with that first accounting system will bring needed efficiencies to a substantial portion of your customer base.

But what if the data isn't that obvious? What if only 10% of your customers use the same third-party accounting system? Is that enough reason to work on that integration first? Knowing that one of those accounting systems is gaining huge market share among your ICP or that another 10% of your customers will shift to it in the next year can give you the data you need to prioritize.

As you can see, a big part of using the customer need strategy is getting good customer data. Do you have a regular touchpoint with your customers, like a quarterly business review (QBR), where they can tell you how everything is going and share concerns and upcoming needs? If you aren't getting regular, solid customer feedback, you might be trying to make strategic decisions based on anecdotes.

Along with customer data, you also need prospect data. If some non-trivial number of your prospects are also using or planning to use the same third-party accounting system, that should factor into your decision. Do you have a mechanism for your sales team to pass this data to your product team? If you use a tool like Gong or Chorus to record sales calls, you can set specific terms (such as "integration") to be flagged in those recordings.

The bottom line is that you can only focus on the integrations that are most needed if you know what other SaaS apps your customers and prospects use and how they use them.

Which integrations will extend your product's functionality and SAM?

That is, which integrations will supply desired functionality to your product and help it do more for your customers? In many cases, integrations can expand your serviceable addressable market (SAM) by providing functionality that solves the problems of larger prospects, more sophisticated prospects, or prospects in a new industry or region.

In some situations, building that functionality directly into your product makes sense. But sometimes, adding certain functionality would take your product in a direction that detracts from your core value proposition. Or you could add that functionality but lack the resources to do it right now.

In either case, building an integration to another app that supplies that functionality can often extend your product's capabilities and SAM for substantially less work and cost than building that functionality in your product right now.

Let's say your product is an industry-specific classification system. However, your product lacks the functionality that more sophisticated customers need to automate the scheduling of classification interviews. As a result, you need to decide whether to add scheduling to your product.

In this scenario, integrating with a scheduling system (ideally one that your prospects are already using) will quickly address those scheduling needs and expand your SAM, while buying you time to determine whether there are long-term benefits to having scheduling as part of your core product.

Which integrations will increase your competitive advantage?

Maybe you've had your product on the market for a while, and it's mature, and you've built a few integrations for it but are struggling to differentiate yourself from your competitors. Or you don't have any product integrations, but you are considering an upcoming major release of your product and would like to do something to make your product stand out from the competition.

How many times have you seen "integrates with X" as a value proposition on a SaaS homepage? If you don't have the integrations your prospects want/need, then they'll go with someone who does. To keep this from happening, find out what integrations your competitors have that you don't, and then focus on closing those gaps as quickly as possible.

But beyond matching what your competitors offer, build integrations to outpace the competition. Each SaaS company that builds an integration can choose (within limits) what that integration does, so integrations with the same third-party app can vary immensely. Are there integration categories where you could offer more, better, broader, or deeper integrations than are available from your competitors? If so, building those integrations, and detailing how/why they offer greater value than your competitors’ integrations with the same systems may be the edge you need.

To make some noise in the market, you might also consider launching an integration hub or marketplace or releasing several integrations in an entirely new category all at once. Doing so lets everyone know that you are making a significant investment in integrations as part of your long-term strategy and signals the market that your product is the one that integrates with everything else they use.

Tactics for prioritizing integration development

These strategies are essential for big-picture, long-term planning. Most SaaS companies use a combination of them and ultimately look at the overlap to choose which integrations to build first.

Your planning should also include flexibility for handling integration needs that pop up periodically, often more frequently than you might want them to. Let's look at some common scenarios where a tactical approach is necessary.

You need an integration to win an in-flight deal

Sometimes, saying yes to adding a prospect's integration requirement makes the difference between winning and losing the deal.

The key here is to agree to those integrations that are also needed by other prospects and customers, add value to your product, or help your competitive position.

Sometimes, you may also say yes to an integration to win an important customer, knowing that it will be a custom or one-off integration but that you can pass along the costs to the customer.

In other cases, you may need to tell the customer "No" if the request doesn't fit with the direction of your product or what you can charge the customer won't cover your costs. Or you may need to tell the customer that you will build the integration, but it will be a few months until you can deliver it.

Whatever you agree to, make sure it's something you can execute in the promised time frame to get the new customer relationship off to a good start.

You need an integration to keep a customer

Sometimes, a missing integration causes enough pain or inefficiency to a customer that the customer considers churning – especially if a competitor has or agrees to build that integration.

Leaving aside the question of whether you want to keep that customer, you should ask questions about the integration. Is it a current gap in your lineup? Does it make sense for your overall product plan? Will it add value to only this customer, or will it be valuable for other customers or prospects? Can you fit it into the short-term schedule, or is it better to revisit it in a quarter or two?

Delaying other priorities to build an integration to keep a customer may or may not be the right decision. But if you decide to do it, your decision should be backed up by as much good data as possible.

You need to replace an existing integration

Sometimes, an existing integration no longer meets customer needs. Ideally, you've planned for integration replacements and worked them into a long-term strategy. However, there will be times when new industry regulations or third-party changes (whether to languages, libraries, or APIs) can break something that worked fine yesterday.

Sometimes you won't need to replace an integration. But in many cases, you may only know once your devs investigate whether a small code change will do the trick or if a complete rewrite is required.

Communication is key. Let your customers know what's happening, how you plan to address it, and when and why they'll need to migrate to the new integration.

Change the plan as you need to

Planning product integrations is challenging, in no small part, because systems and technology are constantly changing. Setting up a strategy for prioritizing integrations doesn't tend to get any easier if you wait, but you also need the flexibility to change that strategy over time.

Integration usage and churn provide essential data for adjusting that strategy to account for changes to the market and your customers. And yes, regular communication with your customers is the key to staying current with how they use your product and its integrations.

Whatever strategic approach you put in place for prioritizing integrations is a starting point. You'll need to periodically revisit your strategy to ensure it isn't stale and that you are not missing market opportunities.

Contact us or request a demo to see how Prismatic's embedded integration platform can help you deliver on your prioritized list of B2B SaaS integrations more quickly and respond immediately when more urgent situations arise.

White Label iPaaS for B2B SaaS Companies

Kristin Ides DeMar — Wed, 08 Feb 2023 19:05:10 +0000

As the number of SaaS apps increases and your customers keep trying to do more without greatly expanding their employee numbers, it’s more important than ever that you have the right tools to support native integrations between your product and the other apps your customers use.

A white label iPaaS is one of those tools and is much more than an iPaaS you can brand as part of your product. A white label iPaaS takes the idea of a traditional (enterprise) iPaaS and builds on it with a framework focused on your customers and how they do business.

What is white label iPaaS?

A white label iPaaS is a set of tools that enables software companies to build reusable, configurable, native integrations and deliver them to your customers as features of your application. It’s called “white label” because critical elements of the platform can be branded and configured to match your core app, ensuring that your customers don’t know where your app stops and the white label iPaaS, with all your native integrations, begins.

The important thing to remember about a white label iPaaS is that it is a purpose-built platform software companies use to create native product integrations for their customers.

This distinguishes it from a traditional or enterprise iPaaS, a general-purpose platform businesses use to create integrations for internal use.

A white label iPaaS, in addition to the features of an enterprise iPaaS (integration designer, connectors, and infrastructure), includes integration deployment and customer support tools, customer management, and an embedded integration marketplace. As a result, it is often called an embedded iPaaS or embedded integration platform.

How white label iPaaS benefits SaaS companies

A white label iPaaS can help you build integrations for customers faster. Development teams used to take weeks or months to build a single integration using the traditional from-scratch approach have reported that, with a white label iPaaS, non-developers can build and deploy native integrations to their customers in days.

As a result, you could realize substantial cost savings, not just from the original build time shrinking dramatically but also because non-devs can build a single integration and deploy it dozens or hundreds of times to different customers with their own configurations. Meanwhile, your devs are freed up to work primarily on your core app.

In addition, a white label iPaaS allows you to provide an excellent integration experience for your customers. No longer is an integration a black box, addressable only by engineering if there is a problem. Integrations are available for customers from the integration marketplace and can be searched, selected, enabled, and configured directly by your customers.

You and your customers can rely on the purpose-built infrastructure of the white label iPaaS to handle security and scalability concerns. Best of all, everything from the integration designer to support and troubleshooting tools is embedded seamlessly in your product.

How to choose a white label iPaaS provider

When it comes to choosing a white label iPaaS provider, here are a couple helpful steps:

Identify your integration needs
Evaluate white label iPaaS providers

1. Identify your integration needs

Before you can determine what tool would be best, you should first determine the integration requirements you will likely be working with. Here are some important questions you’ll want to answer:

What apps do your customers need to integrate with your product?
Will they all need the same integration with your product (differing only in configuration), or will they need many different integrations (different apps)?
What API do you have in place for integrations, or do you have another approach, such as daily file exports to an FTP?
Are your integrations going to be straightforward (data from A to B), or will they be more complex (data from A to B, but with data manipulation between A and B)?
How much control over integrations do your customers need (for example, will customers need to interact with integrations as individual users and not systems)?

2. Evaluate white label iPaaS providers

Once you've determined your integration needs, you'll need to evaluate the capabilities and commitment of each white label iPaaS vendor in addition to seeing if they can meet your integration needs. Here are a few essential questions for that evaluation:

Does the white label iPaaS support low-code only, or does it allow devs to code whatever is not available through built-in connectors and components?
Does the platform fit in with your existing dev environment (code repository, deployment cycles, etc.), or will you need to run the integrations development space in parallel with your current software development?
Is the provider dedicated to providing a white label iPaaS solution, or is this simply one of its many products?
Will the white label iPaaS platform scale from where you are now to where you are planning to be in a year, two years, or five years?
Is the provider using a sensible approach to pricing, where you can pass along the costs to your customers by tying your pricing to the value you are providing?

Prismatic’s white label iPaaS solution

White label iPaaS has gained much traction in the last few years among software companies serving all verticals and organization sizes. Prismatic is proud to be a leader in this space.

Our sole focus has always been to create and support a market-leading embedded integration platform that enables software companies to integrate with their customers’ other critical business apps. To do this, our white label iPaaS solution:

Empowers companies to build and maintain integrations easily, regardless of market vertical. CRM or niche, it makes no difference.
Abstracts the complexities of integration deployment to varied customers. We leverage the power of configuration to make integrations work for every one of your customers without making you create unique integrations for every customer.
Provides a best-integration-class integration UX to your customers’ customers. Customers show up for functionality but stick around for the user experience.

If you want to see precisely how our white label iPaaS could help you jumpstart your integration project, request a demo, and we’ll be glad to get into the details.

What Is an API Integration Example?

Kristin Ides DeMar — Wed, 08 Feb 2023 17:56:21 +0000

An API integration is the code that allows one system to transfer data to or from another system while using an API (application programming interface) to securely access the system. Some API integrations may only an API on one side of the integration, while others might use two or more APIs.

Devs build API integrations for different reasons, but those integrations generally fall into one of two categories: they are either intended to run inside a company to automate internal business workflows, or they are meant to connect systems from different companies for external data sharing.

Since we work with software companies that need to create native integrations connecting their products to the other systems their customers use, our example will cover an external data sharing scenario – but the concepts apply to internal integrations as well.

We’ll lay out this example as follows:

Integration business needs (the why)
Integration technical requirements (the what)
Integration details for execution (the how)

Integration business needs

For this API integration example, your company provides a SaaS product for monitoring building security. Among other things, your app regularly records temperature and humidity levels from sensors installed at critical points in each of your customers' buildings. Your customer needs an integration to export these temperature and humidity values from your product daily to its building maintenance app (StructManager). The customer will then determine if there are correlations between temperature and humidity levels and unplanned maintenance tickets.

Integration technical requirements

When building an integration, you’ll want to start with the technical requirements. These are the basic questions you’ll need to answer before you begin:

What data will be transferred? (Data)
Is this a one-way or two-way integration? (Direction)
How often will the integration run? (Frequency)
What APIs will be used? (API)
What transfer protocols will be used? (Protocol)
What transport languages will be used? (Language)
How will auth be handled? (Auth)

Let’s fill out all of these for our API integration example:

Data. Humidity and temperature records per building for the prior day (24-hr-period).
Direction. One-way export from your product to StructManager.
Frequency. Once per day at 7 AM building local time.
API. Your product uses a SOAP API. StructManager uses a REST API.
Protocol. Both APIs support HTTP.
Language. Your product outputs SOAP XML. StructManager accepts JSON as input.
Auth. Both the SOAP API and REST API use OAuth.

In addition, while the humidity data is provided as percentage points of relative humidity (and is the same in both apps), the temperature data coming from your product uses Celsius, while StructManager is set up to use Fahrenheit. Finally, the temperature and humidity data is collected once per minute by your product, but StructManager only needs to know the values every 15 minutes.

So, for this API integration example, we have two APIs, two data formats, two scales for temperature values, and way more data than required. Sounds like we’ll need to do more than just grab the data from one API and hand it to the other.

Integration details for execution

At 7 AM local time, the integration trigger causes the integration to send a query to your product’s SOAP API requesting the records for the specified customer per building for the last 24 hours. Thankfully, your SOAP API supports OAuth, which is built into the connection string. Once the SOAP API receives the request, it sends back XML via HTTP with all the matching records.

A single record sent from your SaaS product looks something like this:

<environmental>
    <customer_id>AA8312</customer_id>
    <building_id>H265<building_id/>
    <sensor_id>1323</sensor_id>
    <sensor_loc>5W2NAB</sensor_loc>
    <timestamp>10:30</timestamp>
    <temperature>27.3</temperature>
    <humidity>55</humidity>
</environmental>

The integration first performs a data format conversion to translate everything from XML to JSON, giving us the following pattern for a single record.

{
"environmental": {
    "customer_id": "AA8312",
    "building_id": "H265",
    "sensor_id": 1323,
    "sensor_loc": "5W2NAB",
    "timestamp": "10:30",
    "temperature": 27.3,
    "humidity": 55
    }
}

For the sample customer, who has 30 sensors distributed over a single building, the daily export from your SOAP API will include 43,200 records, but the integration only needs to send 2,880 of them to StructManager. As a result, we'll need the integration to filter the 43,200 records and strip out every record that doesn't have timestamp values that match the following patterns: hh:00, hh:15, hh:30, and hh:45. And yes, we could figure out a way to request only these records from the SOAP API in the first place, but it’s cleaner in this case to get the superset of data and go from there.

Once the reduced dataset contains only the records we want to keep, the integration will need to loop over the data once more to convert the temperature values from Celsius to Fahrenheit using a bit of simple math. Our sample data now matches the format StructManager needs to import.

{
"environmental": {
    "customer_id": "AA8312",
    "building_id": "H265",
    "sensor_id": 1323,
    "sensor_loc": "5W2NAB",
    "timestamp": "10:30",
    "temperature": 81.1,
    "humidity": 55
    }
}

At this point, all 2,880 records, encoded in JSON, are wrapped up in an HTTP request for the StructManager REST API. Once again, the integration uses OAuth to connect with the API before submitting the data.

Our API integration example has run successfully and will wait until tomorrow at 7 AM for its next run.

Additional API integration resources

Of course, one example can hardly do justice to a topic as complex as API integrations. With that in mind, here are some resources to help further your understanding of API integration concepts:

APIs by technology (REST, XML-RPC, SOAP, and GraphQL)
APIs by access type (private, partner, public, and open)
Integration transfer protocols and transport languages
Integration media types (formerly called mime types)
What goes on in the middle of an integration

It’s all about the tools

APIs are hugely helpful in building data integrations between SaaS products. But having the right tools to work with those APIs is critical. As a software company providing in-app integrations, those tools can be the difference between implementing the bare minimum integrations that meet your customers' needs, and implementing integrations that are so much a part of your SaaS product, that your customers can't tell where your product stops and the integration starts. One of those tools is an embedded integration platform.

Request a demo if you'd like to see how an embedded integration platform can help you build API integrations – or integrations where there's not an API in sight.

The Best Authentication Methods for B2B SaaS Integrations

Bru Woodring — Tue, 12 Jul 2022 16:33:19 +0000

From the earliest days of software development, authentication (also called auth) has been essential. To ensure system and data security, you must ensure that only properly identified users are permitted to log in to a system.

If you’re building native integrations to connect your SaaS product to the other apps your customers use, one of the tricky pieces is dealing with the nuances of the third-party apps, such as authentication. Sometimes you'll be the one setting up authentication for your own app, and sometimes you'll need to configure your integrations to use whatever auth pattern has been provided. In either case, knowing how user authentication methods work and what to look for can save you time and prevent integration headaches.

In working with our customers, my team has helped SaaS teams enable native integrations using basic authentication, API keys, and Open Auth 2.0 (aka OAuth 2.0), including non-spec variations of OAuth 2.0. Auth type preferences have certainly changed over time, with basic authentication being the go-to approach for a long time, though most developers have moved on to API keys and OAuth 2.0 for the reasons we'll get into shortly.

Let's look at how each of these main authentication methods works and dig into the details within the context of native integrations. We'll also touch on authorization (permission to access specific resources), but that is not the focus of this post.

Understanding the basic authentication method

Basic authentication uses the classic username and password approach. As noted, this is no longer common in modern SaaS apps, but we still see it for many legacy systems, including FTP or older HTTP-based apps.

For the simplest use case for basic auth in HTTP, take the username and the password, put a : between them, and use base64 to encode the whole. Then, place the entire base-64 encoded string as a header in your HTTP request:

curl https://example.com \
  --header "Authorization: Basic bXl1c2VyOm15UEBzU3cwUmQ="

To ensure it conforms to standards, always use the Authorization prefix for the header with the Basic {base64 encoded username:password} value.

An even simpler way is to place the username and password at the beginning of the URL:

curl https://myuser:myP%40sSw0rd@example.com

However, this pattern has long been deprecated because typing credentials in the open, where anyone can read them, isn’t a good security practice.

Why SaaS teams use the basic authentication method for integrations

The draw of basic authentication for integrations is simple: basic auth is super easy to implement. All you need to do is decode the base64-encoded header and verify that the username/password combination is correct. That's it. Because of its simplicity, basic authentication makes sense when you build custom integrations in-house without using an API or integration platform.

Things to consider when using the basic authentication method for integrations

While basic auth is simple to implement, it also comes with a few negatives. Every integration gets the same credentials, and you cannot limit the scope of the credentials. That is, you can't change the authorization tied to the credentials. As a result, each integration can do everything permitted by the credentials. And, if you were to change the credentials, each integration that uses them would need to be individually updated to the new credentials. This largely manual approach to setting and updating credentials means that basic authentication does not lend itself to being used for integrations at scale.

Understanding the API key authentication method

An API key is a single string used both for identification and authentication, particularly with reference to integrations that use an API (application programming interface). Auth based on API keys is often referred to as token-based auth and is common in modern SaaS apps.

To set up auth using API keys, your app creates a key for you to use with your integration. You can often generate a key in an app under a Settings or Profile menu.

An example of an API key in an HTTP request looks something like this:

curl https://example.com \
  --header "Authorization: Bearer mF_9.B5f-4.1JqM"

You may notice that the pattern is very similar to what we used for the basic auth, having the Authorization header, but this time using the Bearer {token}value.

An API key may also be passed in as a parameter for some APIs:

curl https://example.com?token=mF_9.B5f-4.1JqM

Or, you could even use a custom header for the API key:

curl https://example.com \
  --header "x-acme-api-key: mF_9.B5f-4.1JqM"

Why SaaS teams use the API key authentication method for integrations

Though API authentication methods require a bit more to set up than basic authentication, they aren’t hard to implement. Usually, the generating app stores the API keys (or hashes of those keys) in a table and matches the keys up with the corresponding users.

One benefit of using API keys is that you can set the scope of the permissions (authorization), unlike with basic auth. You might set up a single token to have read-only access to specific resources while setting another token to access all resources available through the API. As a result, you can use different tokens for each integration, and it doesn't matter if a user changes a password – the API key remains mapped to that username. If an integration no longer needs access to the API, you can delete/disable the corresponding API key from the app.

API keys are ideal for integrations with your app if you are using an embedded integration platform (also known as embedded iPaaS).

Things to consider when using the API key authentication method for integrations

Users need to copy and paste API keys from one app to another. This manual step doesn't take much time, but it can cause issues when data isn't cut and pasted correctly. Another drawback is that API keys don't usually expire. As a result, someone could compromise an API key, which would continue to function until someone realizes and goes into the source app to disable/delete it.

Understanding the OAuth 2.0 method

OAuth 2.0 is everywhere. You've no doubt used it with numerous apps yourself. In general, it is set up so that you click a button in the App A, and App A sends you over to App B to ask if you wish to enable sharing of something (email, etc.) with App A. You click the button to agree to this data sharing and App A is granted permissions to App B on your behalf.

While this is surprisingly simple on the surface, the complexity is handled behind the scenes. Here’s how it works:

Why SaaS teams use the OAuth 2.0 method for integrations

One of the most powerful characteristics of OAuth 2.0 is that authorization (permissions) can be easily scoped to the specific access needed: read access to accounts, read/write access to contacts, etc. Each integration can use a different access token, and it doesn't matter if a user changes passwords; the OAuth access token will still work.

Revoking the refresh token is also straightforward, thereby disabling a user's ability to generate a new access token. Along with this, if an access token is somehow compromised, it usually expires quickly and limits the damage that could result.

The user experience with OAuth is essentially effortless since the user doesn't need to enter any data, such as credentials or an API key. Instead, the user only needs to approve the authorization request between apps.

While OAuth 2.0 is well suited for building integrations to third-party apps using an embedded integration platform (embedded iPaaS), it’s also the most powerful and secure approach to authentication — no matter what you do for integrations.

Things to consider when using the OAuth 2.0 method for integrations

There is more overhead to using OAuth 2.0 than either basic authentication or API keys. You'll need to build out the infrastructure to track the client IDs and client secrets for any of your apps that use OAuth 2.0. In addition, your app/API will need to be set up to create a callback URL that takes the Authorization Code as input and exchanges it for the access token and refresh token (when applicable).

Finally, you'll also need to build something that will periodically refresh the access token. This could be a cron job, AWS Lambda, etc.

Conclusion

If you are building custom in-house integrations between your app and third-party apps, it probably makes the most sense to use OAuth 2.0 because of the user experience and built-in protection. If you can't take that path for whatever reason, then using API keys should still provide a good user experience and decent security. Basic auth, while it has its use cases, is not optimal for most modern SaaS apps.

If, however, you use an embedded iPaaS to create, deploy, and maintain native integrations with your app, then the platform should come with built-in connectors to many applications that handle most of your authentication needs without requiring you to write additional code. You might even be able to set up your app so that an API key is created, and then that key is provided to your customer within the integration config when the customer activates the integration.

Whether your build your integrations in-house or use an embedded integration platform, a basic understanding of user authentication methods will help you execute your integration plan.

Anatomy of a Webhook HTTP Request

Bru Woodring — Thu, 09 Jun 2022 15:59:50 +0000

An HTTP message is a common means by which two systems, usually a server and a client, exchange data. We typically refer to each HTTP message as an HTTP request or an HTTP response.

Webhook HTTP requests are a specific subset of HTTP requests which transfer data between systems based on events in those systems. Webhooks are used with many event-driven integrations.

When working with our customers, we find that some are new to webhooks and want to learn more about what comprises a webhook HTTP request. If you're in that category, this post should help.

What are the sections of a webhook HTTP request?

A webhook HTTP request generally consists of the following:

Start line
Header(s)
Body (payload)

Start line. Each request has a single start line. It comes at the beginning of the request and includes the method, URL, and version. Here's an example of a start line:

POST /webhook/E474BA38/58E1/4544 HTTP/2

Header(s). Each request can have zero or more headers. Headers usually describe something about the request (such as the type of data or the HTTP client), but you can create custom headers for almost any purpose. Here are example headers:

Host: example.com
user-agent: curl/7.79.1
accept: */*
myapp-hmac-sha1: f237e4a4062590a674b0adc1e84614196aae79f4
myapp-api-key: 90B649F2-70F2-4180-95BC-951F5D832F0D
content-type: application/json
content-length: 188

Body (payload). Each request (except in the case of GET and DELETE) has a single body that could be JSON, XML, or some binary file, though a multipart request may encode multiple types of data into one request. Here's an example of a body:

{
  "orderId": "abc-123",
  "state": "update",
  "updates": [
    { "action": "remove", "item": "widgets", "quantity": 5 },
    { "action": "add", "item": "gadgets", "quantity": 20 }
  ]
}

When we put all the pieces together, a webhook HTTP request (and corresponding response) might look like this:

curl https://example.com/ \
 --verbose \
 --request POST \
 --header 'myapp-hmac-sha1: f237e4a4062590a674b0adc1e84614196aae79f4' \
 --header 'myapp-api-key: 90B649F2-70F2-4180-95BC-951F5D832F0D' \
 --header 'content-type: application/json' \
 --data '{
  "orderId": "abc-123",
  "state": "update",
  "updates": [
    { "action": "remove", "item": "widgets", "quantity": 5 },
    { "action": "add", "item": "gadgets", "quantity": 20 }
  ]
}'

> POST / HTTP/2
> Host: example.com
> user-agent: curl/7.79.1
> accept: */*
> myapp-hmac-sha1: f237e4a4062590a674b0adc1e84614196aae79f4
> myapp-api-key: 90B649F2-70F2-4180-95BC-951F5D832F0D
> content-type: application/json
> content-length: 188
>
* We are completely uploaded and fine
< HTTP/2 200
< accept-ranges: bytes
< cache-control: max-age=604800
< content-type: text/html; charset=UTF-8
< date: Thu, 02 Jun 2022 20:26:44 GMT
< etag: "3147526947"
< expires: Thu, 09 Jun 2022 20:26:44 GMT
< last-modified: Thu, 17 Oct 2019 07:18:26 GMT
< server: EOS (vny/044E)
< content-length: 1256

Examining the start line

Each start line consists of the following:

Method
URL
Version

Method

Request methods (verbs) define the action performed by an HTTP request. At present, HTTP supports eight methods:

DELETE
GET
HEAD
OPTIONS
PATCH
POST
PUT
TRACE

However, webhooks only use a subset of these. POST is used most of the time, even when data is being updated or deleted instead of created. Occasionally, we might see a webhook use GET to verify that a webhook endpoint exists. Less commonly, PUT and PATCH are used to modify/replace data. And probably least common of all, some webhooks use DELETE.

URL

The most common URL for a webhook is something like https://example.com/my-webhook.

Some apps append an absolute path to the URL to let you know the record type that's being requested: for example, https://example.com/my-webhook/order-confirmation when an order is confirmed.

The URL with a query string is a widespread pattern for web page requests (such as search engines) where some value is appended to the end of the standard URL https://example.com/my-webhook?param1=Param-Value1&param2=Param-Value2. While infrequent, some apps use query strings to send metadata via the URL instead of in custom headers.

Version

The version is just that, the version of the HTTP protocol used for the request. It will generally be HTTP/1.1 or HTTP/2. While webhook HTTP requests include the version, it usually has no impact and is there to ensure that the HTTP request is valid.

Examining the headers

Headers for a webhook HTTP request may either be default (standard) headers or custom headers.

Default headers

Many headers are default and are automatically generated by the source system. Here are a few default headers commonly used with webhooks:

Content-Type: Describes the data sent in the body (example: application/json)
User-Agent: Describes the HTTP client used for the request (example: Mozilla/5.0)
Content-Length: Defines the size of the request in bytes.
Accept or Accept-Encoding: Defines the type of response expected.

Custom headers

Custom headers for webhook HTTP requests can vary quite a bit but are used frequently to sign the body, send some other type of authentication (such as an API key), or send other data (such as a Customer-ID) that, for whatever reason, didn't make it into the body of the request. A custom header may also be used for an HMAC signature to secure the webhook endpoint.

Here's an example of a couple custom headers for a webhook HTTP request:

myapp-hmac-sha1: f237e4a4062590a674b0adc1e84614196aae79f4
myapp-api-key: 90B649F2-70F2-4180-95BC-951F5D832F0D

Examining the body

The body of a webhook HTTP request contains the data sent via POST (in most cases) or sometimes PUT or PATCH.

This data is often in JSON format but can also be XML, CSV, a PDF, or any other format you'd like to use. If you need to send several types of data at once, you can set up the body as a multipart body. Doing this allows for files such as PDF or MP3 to be transmitted via HTTP requests alongside JSON, etc. Please note that a multipart body requires a corresponding multipart Content-Type header.

Here's an example of a simple body:

{"renderId":51266,"s3Bucket":"test-customer-renders","status":"complete"}

Here's an example of a multipart body (with the multipart header):

curl 'https://example.io/webhook/' \
  --request POST \
  --header "Content-Type: multipart/form-data" \
  --form person='{"firstname":"Sam","lastname":"McElhaney"};type=application/json' \
  --form photo=@sam.jpeg \
  --form resume=@resume.pdf

> POST /webhook HTTP/2
> Host: example.com
> user-agent: curl/7.79.1
> accept: */*
> content-length: 73686
> content-type: multipart/form-data; boundary=------------------------0c985f7380ec6342

--------------------------0c985f7380ec6342
Content-Disposition: form-data; name="person"
Content-Type: application/json

{"firstname":"Sam","lastname":"McElhaney"}
--------------------------0c985f7380ec6342
Content-Disposition: form-data; name="photo"; filename="sam.jpeg"
Content-Type: image/jpeg

SOME BINARY DATA...
--------------------------0c985f7380ec6342
Content-Disposition: form-data; name="resume"; filename="resume.pdf"
Content-Type: application/pdf

%PDF-1.3
MORE BINARY DATA
%%EOF
--------------------------0c985f7380ec6342--

Conclusion

With an increasing number of companies (from Salesforce to Shopify) implementing webhooks, this technology is fast becoming the standard for SaaS integrations. Webhooks are straightforward to understand and implement, and they are highly flexible – allowing you to make them as simple or complex as the data requires.

How to Secure Webhook Endpoints with HMAC

Bru Woodring — Wed, 25 May 2022 21:21:04 +0000

Webhooks are ubiquitous in SaaS integrations, and there’s a good reason for that. They are a simple and speedy way to transfer data via HTTP callbacks between systems based on changes to data in those systems. My company helps SaaS teams build native integrations to their customers’ other apps, and many of our customers use webhooks for their integrations.

Over the last few weeks, we’ve helped several of our customers who needed to ensure that their webhook endpoints were secure. In this post, we’ll describe the approach we recommend. But first, let’s lay some groundwork.

How do webhooks work?

In short, the source app has a webhook, and the destination app has a webhook endpoint; based on some event occurring in the source app, the webhook sends an HTTP request to the webhook endpoint.

Here's a simple example of an HTTP request body (or payload):

{
  "event": "WAREHOUSE_UPDATE",
  "updates": [
    {
      "item": "gadgets",
      "action": "add",
      "quantity": 20
    },
    {
      "item": "widgets",
      "action": "remove",
      "quantity": 10
    }
  ]
}

But how do you ensure that the destination app receives valid data from the source app and not bogus data from a bad actor who has spoofed the webhook?

The short answer is that you need to set up the webhook to provide the endpoint with the HTTP request and a unique key that the endpoint can use to verify the data. But, before we get into the details, let's briefly cover hashing.

What is hashing?

At its simplest, hashing is the process of converting a value (or key) into another value. Even if you've not worked extensively with hashing before, you are probably aware of MD5, SHA-256, or RipeMD-128. Each of these is the name of a hashing algorithm (aka cryptographic hash function).

Let's see what each algorithm does to a classic string:

MD5 hashes Hello World! to ed076287532e86365e841e92bfc50d8c
SHA-256 hashes Hello World! to 7f83b1657ff1fc53b92dc18148a1d65dfc2d4b1fa3d677284addd200126d9069
RipeMD-128 hashes Hello World! to 24e23e5c25bc06c8aa43b696c1e11669

The important part is that an algorithm hashes a value the same way every time. If we don't change our string ('Hello World!'), the resulting hash value doesn't change either.

However, if anything in the string changes, the hash will also change. For example, let's lower-case the 'H' so we have 'hello World!' and see what that does:

MD5 hashes hello World! to 41d0c351efedf7fdb9a5dc8a1ed4b4e3
SHA-256 hashes hello World! to e4ad0102dc2523443333d808b91a989b71c2439d7362aca6538d49f76baaa5ca
RipeMD-128 hashes hello World! to b5cf338f17d6796ba0312e0d78c70831

A slight change, but the resulting differences are evident.

Though hashing doesn't allow us to completely solve our original problem (someone sending bogus data to a webhook endpoint), it does lead us directly to HMAC.

What is HMAC?

HMAC, or hashed message authentication code, is an authentication method that uses not one but two keys. The first key is the HTTP request body, while the second one is a secret cryptographic key. When you implement HMAC for your webhook, you’ll be using both these keys plus an algorithm such as MD5, SHA-256, or RipeMD-128 to ensure the HTTP request that shows up at your webhook endpoint is legit.

How does HMAC work?

Before the source app sends an HTTP request via the webhook, it hashes the payload (request body) with HMAC using the secret key. The resulting hash is then bundled into the HTTP request as a header, and the entire request (header and body) is sent to the webhook endpoint.

Upon receiving the HTTP request, the destination app hashes the body with the secret key and then compares the result to the hash provided in the header. If the values match, the destination app knows the data is legit and processes it. If the values do not match, the destination app rejects the data and executes whatever code was written for that scenario — perhaps creating a log entry or sending a notification.

If someone tries to spoof the payload, they won't be able to generate a valid hash since they don't have the secret key. Door closed.

Let’s imagine that you have an e-commerce platform connected to your app. Your app regularly sends payloads to the platform’s webhook endpoint to create orders and issue refunds. Using HMAC ensures that you won’t have random (or not so random) people sending bogus orders or refunds to the e-commerce platform.

But, you say, couldn't someone capture an HTTP request and reverse engineer the hash in the header to figure out the secret? Short answer: no. Hashing is a one-way function. To crack a hash with a sufficiently complex secret, we would need more computing power and time than any of us has available.

Apps that rely on HMAC for webhook endpoints

Some well-known apps currently use HMAC to secure their webhook endpoints:

Slack: Provides a Signing Secret when you create a Slack app. When it sends a webhook payload, it hashes both the payload and webhook's timestamp with that secret using SHA256. The webhook request includes the resulting hash as a header called X-Slack-Signature.
Dropbox: Generates an App Secret when you create a Dropbox app and uses that secret to generate webhook HMAC hashes and authenticate users with OAuth 2.0. It hashes webhook payloads using SHA256 and sends the hash as a header called X-Dropbox-Signature.
Shopify: Creates an API Secret Key and hashes its payloads with that key and SHA256. It sends the hash as a header called X-Shopify-Hmac-SHA256.

HMAC has broad language support

You can use just about any modern language to compute HMAC hashes. Here are some links to popular languages with HMAC capabilities:

Example code for HMAC

Finally, what would all of this be without code? Here is an example of how this might be set up in NodeJS using the built-in crypto module:

const crypto = require("crypto");

const SECRET_KEY = "secret-FA782CF7-060E-484E-B3DC-055CF2C9ED99";

const payload = JSON.stringify({
  event: "REFUND_REQUEST",
  user: "realcustomer@notabaddie.com",
  amount: "50.25",
});

const hash = crypto
  .createHmac("sha256", SECRET_KEY)
  .update(payload, "utf-8")
  .digest("hex");

console.log(hash); // Prints d12f95e3f98240cff00b2743160455fdf70cb8d431db2981a9af8414fc4ad5f8

The corresponding HTTP request using HMAC might look like this:

curl https://my.webhook.endpoint.com/callback \
  --request POST \
  --header "x-hmac-hash: d12f95e3f98240cff00b2743160455fdf70cb8d431db2981a9af8414fc4ad5f8" \
  --data '{"event":"REFUND_REQUEST","user":"realcustomer@notabaddie.com","amount":"50.25"}'

Even if a bad actor intercepted your HTTP request, they couldn't issue a refund request of a million dollars to their own email address, since they couldn't sign the request properly without the secret key.

Conclusion

Using HMAC does not require that you learn a new language or gain an advanced understanding of encryption, but it does allow you to protect the integrity of the data you are transferring via webhooks.

Common B2B SaaS Integration Patterns and When to Use Them

Bru Woodring — Tue, 19 Apr 2022 14:56:25 +0000

According to a recent survey, the number of SaaS companies has grown to more than 25,000 worldwide. In general, businesses that subscribe to SaaS products work with multiple SaaS vendors and expect that the apps will integrate with each other. To address this, some SaaS companies build bespoke integrations into their apps, while others go with an embedded integration platform to address their customers' integration needs.

Regardless of the approach you are taking for integrations with your app, you'll find that most integrations should fit into a few common patterns. Knowing these patterns will help whether you find yourself in the midst of an integration project, just getting started with building out your API, or in the early days of planning a new SaaS app.

In this post, we'll briefly examine these patterns and learn the benefits of each.

Common integration patterns

Event-driven – An event occurs in an app, which causes data to be sent to another app. The event may be in your app or the third-party app with which you are integrating. This is the most common type of integration between SaaS apps.
Scheduled – An integration that runs on a regular schedule (every minute, every hour, daily, etc.) These integrations include exports (such as recurring reports) and imports (such as polling an API to request data for import to your app). Scheduled integrations are often used between a SaaS app and non-SaaS application.
Synchronous – One app makes a request, then waits for the integration to complete and respond with an answer. Because of timing, this can be the most difficult type of integration to implement successfully. It is not used as often as the event-driven and scheduled integration patterns.
Hybrid – This one is not so much a pattern as a combination of other patterns.

Event-driven integrations

This is probably the most common type of integration for SaaS apps. When an event occurs in a source system (for example, a customer record is updated in CRM or an invoice is paid in POS), data about the event is sent to a destination system.

This data is generally sent via a webhook. A webhook is an event-triggered request to an app (the third-party app for an import integration or your app for an export integration). The payload is the data sent by the webhook to the destination system.

If the source app is not using an API with webhooks, it may be using a different type of pub/sub system such as Amazon SNS, Apache Kafka, or even email. In those cases, the destination app subscribes to what those systems provide.

For a great example of an event-driven API, consider GitHub, which has a variety of events that can trigger a webhook. GitHub also provides the shape of the payload sent via each webhook. When you integrate with GitHub, you'll create endpoints to receive those payloads.

If you are interested in ensuring that data is transmitted speedily, that your event-driven API is reusable for multiple integrations, and that third-party developers have an easy path to building integrations with your app, you’ll definitely want to set up your app and API to support webhooks.

In the following sections, we’ll look at various event-driven integrations.

Event-driven import integrations

An event-driven import integration is an import of data from the third-party app to your app based on one or more events in the third-party app. The integration may have a single payload or multiple payloads. Within the integration, a payload may move through a single sequence of activities (a flow), or it might go through multiple flows or branching logic to be filtered or otherwise transformed before your app consumes the data.

The key to an event-driven integration for import is that the third-party app publishes the data (via webhooks or another pub/sub system), and your app subscribes to that data.

Event-driven export integrations

An event-driven export integration is an export of data from your app to a third-party app based on one or more events in your app. As with an import integration, you may be handling a single payload or multiple payloads. And the logic may be simple (a single flow) or complex (multiple flows).

The key to an event-driven integration for export is that your app publishes the data (via webhooks or another pub/sub system), and the third-party app subscribes to that data.

Event-driven two-way integrations

An event-driven two-way integration melds the import and export functions into a single integration. The import portion will have its logic, and the export portion will have separate logic, but everything is contained within a single integration. Two-way integrations could be split into separate import and export integrations, but they are usually combined to provide straightforward deployment and support processes.

In an event-driven two-way integration, both your app and the third-party app are publishers and subscribers.

Scheduled integrations

When the system you need to transfer data from does not have webhooks or another pub/sub system to which you can subscribe, you need to use a scheduled integration. A scheduled integration can be either import or export. Instead of being triggered when the integration receives a notification, a scheduled integration is triggered at a specific time or time interval. Your integration can go out and query for data, and then send that data off to another app, a data storage system, or a messaging system like Slack, email or SMS.

The data may be only a minute or two old for a scheduled integration that recurs frequently, but it will not be as fresh as data from an event-driven integration.

In the following sections, we’ll look at various scheduled integrations.

Scheduled data import integration

You'll need a scheduled integration to import data from a third-party API that does not have webhooks or another pub/sub system. In this case, the integration must wrap the third-party API endpoints containing the data the integration will query. The API then processes the query and returns the requested payload.

Scheduled file import integration

Some third-party apps may be configured with file export capabilities instead of an API. This scenario is more likely with non-SaaS (legacy) systems. In this case, the app may write out data as PDF, XML, CSV, or another file type to an external data source such as Dropbox or SFTP. To import this data, you'll build an integration that checks the location(s) according to a regular schedule to see if there are new files to process.

When there is new data within the location (for example, new XML files in a Dropbox folder), your integration will loop over each file, process the files, and then move or delete the processed files. As with other scheduled integrations, this still is not near-real-time but can be current within a few minutes of the file(s) being placed in the external data source.

Scheduled file export integration

You'll use this type of integration when you need to generate and send a regular report to your customers. You set up an integration where you are either querying your API (for your app) or a third-party API for specific data.

Once the data has been returned, your integration generates a file with the data in the necessary format (PDF, XML, CSV, etc.) Finally, this file is packaged up and sent off to its destination (which could range from a filesystem to email or SMS).

Synchronous integrations

A synchronous integration may be what you need if you have data from a third-party app that you must have right now. An integration of this type calls an integration URL and then waits for a response.

The synchronous integration should function well if it works quickly and has little to no lag between the call and the response. However, if the integration takes a while to run (much data to process, the third-party app is slow in fulfilling requests, etc.), the integration becomes susceptible to network disconnects and timeouts.

Synchronous integrations are useful when the order of operations matters. For example, if your sales system receives an order and you send data to a payment processor integration, you probably want to wait for a response that the payment was posted successfully before you send data to your order fulfillment integration. As you build an integration for these scenarios, you’ll need to decide if waiting for a response is right for you or if you should have the payment processing system "call you back" when the payment is successfully run asynchronously.

Hybrid integrations

Hybrid integrations are some combination of other integration types. For example, suppose that your app supports webhooks, but the third-party app you are working with only has an API with no webhooks. As a result, you might build a hybrid integration where you have an event-driven export from your app but are running a scheduled data import from the third-party app. Instead of setting this up as two separate integrations, you would be able to combine them in the same integration with distinct flows and logic.

Conclusion

While integrations may be defined simply as software that transmits data from one system to another, details can vary considerably from one integration to the next. When planning and building integrations between your app and your customer's third-party apps, it's essential to consider integration patterns, ensuring that you use the best pattern for each integration scenario.

SOAP APIs Aren't Scary: What You Should Know Before You Build a SOAP Integration

Taylor Reece — Wed, 23 Mar 2022 20:05:09 +0000

Building your first integration with a SOAP-based API can be daunting.

Recently I’ve helped several companies integrate their apps with third-party apps and services that use SOAP-based APIs. For developers with SOAP experience, the integrations were a breeze. But for the uninitiated, SOAP has a pretty steep learning curve and throws a lot of new terms and acronyms your way: XML, WSDL, envelope, procedure... and a few dozen others.

Today I want to look at some basic concepts of SOAP, then dive into some tools and resources that make SOAP integrations a lot easier to build (after all, the “S” in SOAP stands for “simple”)!

What is SOAP?

First, let’s talk about some SOAP basics. If you feel like reading through the entire SOAP spec and its history, you can over at w3.org. I’ll try to give the tl;dr here.

Originally, SOAP stood for “Simple Object Access Protocol.” It was largely built to handle CRUD (create, read, update, delete) operations and it was a protocol to access your objects (like inventory or customer data) in a simple way. A SOAP API would advertise the types of objects that outside services could fetch and manipulate and would outline exactly how a third party should make CRUD requests.

SOAP has since dropped that acronym because it does a lot more than object manipulation, but the idea is still the same: it’s a protocol for two systems to communicate with one another in a predictable way.

Web services and WSDLs

A SOAP API provides a web service. These web services are made up of one or more procedures (called operations). For example, you might create a “weather forecast” web service, and it could include a “get the 10-day forecast” operation, or an “I’m a weather station, and here’s our latest temperature data” operation.

Requests to each operation take very specific formats for their input, and they have predictable formats for their responses. The “get the 10-day forecast” operation might take a numerical zip code as its input and return an array of floating point temperatures in Fahrenheit that are forecast over the coming ten days for that zip code.

When SOAP-based web services are published, a summary WSDL (Web Services Description Language) file is published. The WSDL, written out in XML, outlines the various operations that the service offers, as well as what inputs and responses you can expect for each operation.

Let’s look at a simple WSDL for a rudimentary “calculator service.” The WSDL (here) defines some basic calculator operations (like addition, subtraction, multiplication, and division). This portion of the calculator WSDL declares that the service has an addition “Add” operation:

<wsdl:operation name="Add">
 <wsdl:documentation xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/">
  Adds two integers. This is a test WebService. ©DNE Online
 </wsdl:documentation>
 <wsdl:input message="tns:AddSoapIn"/>
 <wsdl:output message="tns:AddSoapOut"/>
</wsdl:operation>

If we look further into this XML, we see that the wsdl:input references a message type tns:AddSoapIn. Looking a few lines prior where some parameters are declared, we see this section:

<wsdl:message name="AddSoapIn">
 <wsdl:part name="parameters" element="tns:Add"/>
</wsdl:message>

That portion of the WSDL references an element tns:Add, which is found in another previous section of the WSDL. The tns:Add element defines a complex input type with a couple of integers:

<s:element name="Add">
 <s:complexType>
  <s:sequence>
   <s:element minOccurs="1" maxOccurs="1" name="intA" type="s:int"/>
   <s:element minOccurs="1" maxOccurs="1" name="intB" type="s:int"/>
  </s:sequence>
 </s:complexType>
</s:element>

Now, one criticism of SOAP is that it is incredibly verbose - this WSDL is no exception, but in layman’s terms all that the above XML really says is “this service has an Add operation and it takes two integers, intA and intB, as inputs.”

Consuming a SOAP API

Okay.... there’s an addition procedure out there that we want to invoke and it takes two integers. Great. How do we go about calling it? SOAP APIs are almost always HTTP-based, so we need to craft an HTTP request from the WSDL we looked at.

SOAP APIs always communicate with XML, so we’ll start by adding a header that sets our Content-Type to text/xml; charset=utf-8.

The data we send needs to be in XML format, and SOAP wraps data sent back and forth in a SOAP envelope (an envelope defines the message’s structure). If we look at the calculator WSDL again, we can glean the envelope’s structure. Within our SOAP envelope’s “body” we’ll declare what procedure we’re going to run (Add), and we’ll include values for the inputs that the procedure expects (for intA and intB). Our HTTP request will end up looking like this:

curl http://www.dneonline.com/calculator.asmx \
  --request POST \
  --header "Content-Type: text/xml; charset=utf-8" \
  --data \
'<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
 xmlns:soap="http://www.w3.org/2003/05/soap-envelope">
 <soap:Body>
    <Add xmlns="http://tempuri.org/">
     <intA>15</intA>
     <intB>27</intB>
    </Add>
 </soap:Body>
</soap:Envelope>'

After running that HTTP requests from our command line, we get a response that’s also in XML, and also wrapped in an envelope. The response’s body includes the AddResponse that’s defined in the WSDL, and that has an AddResult - the sum of the two numbers we sent to the server:

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope
 xmlns:soap="http://www.w3.org/2003/05/soap-envelope"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xmlns:xsd="http://www.w3.org/2001/XMLSchema">
 <soap:Body>
  <AddResponse xmlns="http://tempuri.org/">
   <AddResult>42</AddResult>
  </AddResponse>
 </soap:Body>
</soap:Envelope>

Phew... that’s a hell of a lot of parsing through a WSDL and monkeying with XML schema just to add two numbers together. It’s nice to have predictable request and response formats, but a missing xmlns XML attribute or typo’d operation name can result in the SOAP server rejecting your requests. Hand-writing these XML requests is pretty error-prone. Let’s reach for some tools to simplify things.

SOAP tooling

Obviously manually parsing a WSDL with your eyes and hand-writing HTTP requests is burdensome and should be avoided at all costs. I recommend reaching for a SOAP tool that parses the WSDL for you and presents you with a list of operations (and their inputs) that you can invoke. There are several great apps out there built for exactly that purpose:

SoapUI from SmartBear is a popular tool that gives you a great graphical UI for navigating through a WSDL. Popping open any operation gives you a sample request that you can fill in and execute from within their app:
Recent versions of the popular HTTP client Postman also support importing a WSDL, and it even provides you with pre-populated sample requests that you can send to the web service from within the app:

These graphical tools are great for exploring a SOAP API, and I recommend you familiarize yourself with the web service’s operations prior to sitting down and coding up your integration. Once you get a good idea of how the SOAP API works, it’s time to turn to code.

Writing code to interact with SOAP APIs

Manually parsing a WSDL and hand-writing HTTP requests is error-prone and just generally an awful experience. You definitely don’t want to resort to templating out XML with string literals. Let’s avoid that when writing code to interact with a SOAP-based web service.

Whether you’re using Python, NodeJS, C#, or any other modern programming language, the chances are good that there’s a SOAP library you can leverage to make it simple to invoke web service operations. In NodeJS, for example, you can reach for the aptly named soap package on NPM.

This library can take the URL of a WSDL as an argument, and in doing so creates a fully functional HTTP client object that can invoke any of the web service’s operations. We can perform something like client.Add to invoke the “Add” operation, pass in a couple of input arguments, and the rest (parsing the WSDL, formatting the XML request in an envelope, deserializing the XML response, etc.) is handled for you. With just a half-dozen lines of code (I don’t count parentheses!), we can make a request to the calculator API:

var soap = require("soap");
const wsdlUrl = "http://www.dneonline.com/calculator.asmx?WSDL";
const args = { intA: 15, intB: 27 };
soap.createClient(wsdlUrl, function (err, client) {
  client.Add(args, (err, result) => {
    console.log(`The API returned a sum of ${result.AddResult}`);
  });
});

The response to our request contains an AddResult property, and our function prints out The API returned a sum of 42, like we’d expect.

Abstracting SOAP Further

If you’re building integrations that connect your own product to other apps your customers use, an embedded integration platform as a service (embedded iPaaS) like Prismatic can abstract much of the complexity of interacting with SOAP APIs. Embedded integration platforms include low-code integration builders with pre-built connectors for popular SOAP-based apps (like Salesforce), which completely eliminates the need to write code to interact with many SOAP-based APIs.

For less common SOAP-based APIs that don’t have built-in connectors, they typically offer a SOAP component that abstracts out the complexity of making SOAP requests:

In many situations, you can leverage an embedded iPaaS to interact with a SOAP-based API while writing little to no code.

Conclusion

Obviously the calculator web service is incredibly simple, but I think it does a good job illustrating how a SOAP web service can appear daunting at first, but with the right tooling it’s relatively clean and simple to build an integration with a SOAP-based API.

Building an Integration to Another App: How to Get Started

Taylor Reece — Wed, 02 Feb 2022 16:20:51 +0000

[ding 🔔] goes your email notification. You’ve just been assigned a new ticket. The subject line reads “Integrate our app with Acme CRM.” The ticket’s description is pretty sparse - just a few bullet-points saying things like:

Sync users’ appointments in Acme with our calendar app
Track the number of email and phone touch-points users have in Acme on our dashboard

You expected this. A lot of the businesses that use your software also use Acme CRM, and you’ve been hearing from your sales and support teams that customers keep asking for their Acme data to sync with your app.

So.... it’s up to you to build the integration with Acme. Where do you even begin?

I know I’m always inclined to fire up my IDE and dive right in to code, but that always results in misunderstanding expectations, missing details, and having to redo work.

Let’s talk about the stuff you should do before you start coding. Let’s think through the questions you should ask, requirements you should clarify, and tools you can reach for to get a feel for how the integration is going to work.

Gather requirements for your integration project

In an ideal world, the person requesting the integration would provide you with a specification that details what events trigger the integration to run, what type of data gets sent over, and how the data should look when it arrives at its destination. The world’s rarely ideal, though, so you’ll probably have to go digging for clearer integration requirements.

It’s incredibly helpful to have a series of “When foo then bar” statements. For example, “When a user in Acme CRM sets an appointment with a customer, a calendar event is created in our app that contains the customer’s name, phone number, email address, and time of the appointment.” Bonus points if you can tie each step to relevant API documentation!

As you think through the “when foo then bar” statements, there are several companion questions you should be asking with regards to how data should flow from one system to another:

What triggers an integration to run? Does Acme support events like webhooks (where they send data whenever it’s been changed) or are you expected to fetch data on a schedule?
Is this a one-way or two-way integration (that is, will Acme just send data to your app, or does your app also need to send data back to Acme)?
What’s the format of the data the source system is sending? JSON? XML? CSV?
What information is included in the payload that’s sent your way?
What format of data is the destination system expecting? You may need to fetch more information from other third-parties or Acme’s other API endpoints to satisfy the destination’s requirements.

After you have a sense of how data will flow, ask yourself what things could go wrong, and clarify how your integration needs to behave in these common situations:

How should you deal with data “collisions” where the same data gets sent over twice? Should you ignore the second request, create a second record, or update (”upsert”) the existing data?
What should happen if incomplete or “bad” data is shipped (a calendar event that’s missing date, for example)? Do you need a “dead letter queue” to store data that causes your integration to error out so your devs or support folks can examine the bad data later?
What happens if either API is down? Is your integration expected to hold on to data and retry later?

Next, figure out how configurable you need this integration to be. (Occasionally, I’ve needed to build “one-off” integrations for a single customer. Far more often, I’ve needed to build reusable integrations that can behave differently from customer to customer and handle each customer’s credentials for the third-party app.)

How do users authenticate with each system? Do users have their own credentials to Acme, or some shared key? Where are you going to store those secrets securely?
What things are going to be different between customers you deploy this integration to? Do customers have different API endpoints, or data mapping requirements?
What all needs to be configurable to make it so you can deploy the same integration to multiple customers?

Finally, figure out deployment and support concerns. If your company has several app integrations already, you may already have answers to these questions:

How will your customers activate and configure the integration? Do you need to build UX for this?
What are the expectations for logging, monitoring and alerting? Where should the integration’s logs be sent, and under what conditions should someone be alerted if the integration runs into a problem?
Where is the integration itself going to live? Will you need to build custom infrastructure to host this integration?

As you can see, a quick bullet point spec like “Import users’ appointments into the our calendar” seems simple at first, but there’s a ton of things to think through if you want to build, deploy and support your integration thoroughly.

Get familiar with how the third-party app works

Once you’ve thought through all of the questions above and have a solid understanding of the task at hand, it’s time for you to familiarize yourself with the third-party app(s) involved. I recommend logging in to Acme CRM like an end user would. Figure out what exactly it means to “set an appointment” or “log a phone call with a customer” in the context of Acme. This should give you a sense of how different objects that the application stores are related.

If you don’t have an Acme CRM account, ask for one. It’s mutually beneficial for both your company and Acme to have an integration between your systems, and most reasonable companies are willing to provide a free developer account or “sandbox” environment where you can toy around with dummy data, without needing to access your users’ real Acme data.

You may be provided written information about how the other application will provide data to you, but I recommend that you insist on getting a real sandbox environment that at least sends fake data in the format the third party says they use. I’ve been around for plenty of integrations where a third party claims they’ll send XML in thus-and-such a format using REST calls, but informs you after you build your integration that they decided to write out JSON messages to a message broker (like Kafka or Amazon SNS) instead. You can just change your code quick to subscribe to an SNS feed, right? Testing with an actual system is far better than mocking out fake APIs and message streams yourself.

It sounds silly, but also make sure to log in to your own app as well to verify that you know what it means to “create a calendar event” or “update the dashboard.” Verify with the people requesting the integration that you understand what actions should trigger the integration, and where the data should be displayed when it arrives at its destination.

Explore the API with an HTTP client

You’re ready to start fiddling with APIs, but it’s not quite time to code quite yet. Take a look at Acme’s developer docs. If they have an OpenAPI/Swagger or WSDL spec that defines all for their API endpoints, you’re in luck! Figure out how their auth flow works, what data their webhooks send, and what data is available for you to query. If you’re really lucky, Acme will provide a client library you can use when it’s time to start coding (but don’t get your hopes up!).

Fire up your favorite HTTP client (I personally like Postman), and try to make some simple authenticated GET and POST requests against the Acme CRM sandbox you have access to. Verify that the third-party docs are correct, and that you can, indeed, fetch data from their appointment API endpoint. Double-check that the payload you receive has all the information you need to map data over to your app.

Once you have requests to Acme CRM working in Postman (or your favorite HTTP client), finally turn to code.

Time to build!

Now that you have a good idea of the shape of data that comes in, where additional data should be fetched from, how it should be modified and where it should wind up, it’s time to start building your integration.

There’s still a lot to do - you need to write your code, make it configurable enough to handle differences between customers, deploy it, monitor and support it, make the integration available within your product, etc., but asking the right questions and taking the proper steps beforehand can make all that go faster and keep rework to a minimum.

About Taylor

Taylor is a developer advocate at Prismatic (an integration platform for B2B software companies) where he helps customers build integrations connecting their products to their customers’ other apps. If you’d like to chat with Taylor or the folks at Prismatic, please reach out - we’d love to chat about the integrations you need to build for your app!

Why We Moved From Lambda to ECS

Taylor Reece — Wed, 21 Apr 2021 14:46:26 +0000

After many months of development, my team just announced the general availability of our platform. That milestone seems like a perfect opportunity to look back and reflect on how the infrastructure that supports Prismatic has evolved over time. (Spoiler: We ended up moving our most important microservice from Lambda to ECS.)

In this post I'll dive into what went well in Lambda, what challenges we faced, and why we eventually made the decision to migrate some services from Lambda to AWS Elastic Container Service (ECS).

What Problem are We Solving?

For some quick context, our product is an integration platform for B2B software companies. That is, we help software companies build integrations and deploy those integrations to their customers. A simple integration might look something like this:

Step 1: Pull down an XML document from Dropbox.
Step 2: Process the XML with some custom JavaScript code.
Step 3: Use some stored credentials to post the processed data to a third-party API.

Our users can configure integrations to run on a schedule, or they can trigger them via a webhook, and our platform takes care of running, logging, and monitoring the integrations (and a whole bunch of other things).

The Early Days

The first incarnation of Prismatic used LocalStack. We knew that we wanted to eventually host Prismatic in AWS (with the possibility of moving to Azure, GCP, etc. as needed), so the ability to spin up our platform locally to simulate AWS was appealing. The LocalStack service that approximates AWS Lambda was easy to iterate on, and ran without any major hiccups. It gave us a great development feedback loop, so we could prototype and test very quickly.

We used Lambda to execute each "step" of an integration, and steps leveraged SQS to pass data and trigger the next step. So, an integration execution would look like this:

Run a Dropbox "fetch a file" action to grab an XML file.
Save the contents of that XML file to SQS, trigger the next step.
Run a customer's custom JavaScript code to process the XML.
Save the resulting transformed data to SQS, trigger the next step.
Run an action to post the processed data to a third-party API.
Save the results of the final step, trigger the end of the integration.

Within LocalStack, this was a very quick process. We could define a 6-step integration, run it, and see results within a couple of seconds.

Our Migration to Real AWS Lambda

Once we had a proof of concept working, we devoted some time to moving Prismatic to an actual production environment, with real Lambdas, queues, databases, etc. We were still a small team, and we didn't want to dedicate a ton of time to DevOps-y, infrastructure problems yet. We wanted to dedicate most of our time to our core product, and Lambda let us do just that.

Lambda was attractive to us for a number of reasons. We didn't need to worry about CPU or memory allocation, server monitoring, or autoscaling; that's all built-in. We were able to throw .zip files full of JavaScript code at Lambda, and AWS took care of the rest. Lambda let us compartmentalize our code into a series of microservices (a service for logging, a service for OAuth key renewals, a service for SMS/email alerting if integrations error out, etc.), so we could keep a good mental map of what code is responsible for doing what task. Costs were pretty reasonable, too - you just pay for compute time, so rather than running servers 24/7, we just paid when our prototypes were executing something.

After a few days monkeying with Terraform, we had our second incarnation of Prismatic in AWS. Our integration runners ran on real Lambda, and were triggered via SQS. This is the point at which we started running into performance issues with our integration runners.

Why Lambda Didn't Work for Us

We had a number of issues, ranging from speed to SQS size limits and lack of process isolation in Lambda, that caused us to reconsider its effectiveness as our integration runner. Let's talk about each of those issues:

Speed. Remember the 6-step integration that I said took a couple of seconds to run within LocalStack? It took a full minute using real Lambda and AWS. The actual Lambda invocations were quick - usually a few milliseconds. The writing of step results to SQS and subsequent execution of the next step, though, ended up taking multiple seconds every step. For more complex integrations, like ones that looped over 500 files, that was a show-stopper - who wants their integrations to take minutes (hours?) to complete?

We tried a number of things to get our Lambda invocations to go faster. We followed guides to keep a number of Lambda instances "warm", and we cranked up the number of vCPUs powering our Lambdas to the highest we could at the time (6 vCPUs / 10GB RAM), but those things only shaved single digit percentages off of our integration run times.

SQS Size Limits. SQS limits message size to 256 kilobytes. The amount of data being passed between steps of an integration often exceeded that size (after all, it's totally reasonable for an integration developer to pull down a multiple megabyte JSON file to process). We were able to work around this size limitation - the recommended solution that we followed was to write out payloads to S3 and pass references to S3 objects via SQS - but this extra API call to S3 only compounded our slowness issues.

Process Isolation. This was the issue that surprised me the most. At first, AWS Lambda seems appealing as a stateless compute engine - run a job, exit, run another job, etc - scale horizontally as needed. We naively assumed that Lambda invocations were isolated from one another, but that turned out to only be half true. Concurrent invocations are isolated (they run in distinct containers within Lambda). However, subsequent invocations reuse previous "warm" environments, so an integration runner might inherit a "dirty" environment from a previous integration run. That's especially a problem if you let customers write their own code, like we do for our customers' integrations.

It turns out that if one customer writes some bad code into their integration - something like this, global.XMLHttpRequest = null;, then subsequent integration runs on that same Lambda that depend on the XMLHttpRequest library error out. This is a big deal, since one customer could break something like axios for another customer. A customer could even be malicious and execute something like global.console.log = (msg) => { nefariousCode(); }, and other integrations that execute on that same Lambda will run nefariousCode() whenever they invoke console.log(). Yikes!

We tried a few things to get around this issue of shared execution space. We toyed with forcing our Lambdas to cold-start every time (which was a terrible idea for obvious reasons), and we tried spinning up distinct Node processes within chroot jails. Neither option panned out - spinning up child Node processes in a Lambda took 3-5 seconds and partially defeated the purpose of being in Lambda in the first place.

Our Move to ECS

Lambda had served us well with development - we were able to iterate quickly and get a prototype out the door, but with the myriad issues we faced in Lambda we decided to bite the bullet and dedicate some dev time to cloud infrastructure.

Our team got to work expanding our existing Terraform scripts, and moved our integration runner to AWS Elastic Container Service (ECS). Within an ECS container we could easily (and quickly!) chroot and isolate Node processes from one another, solving the process isolation issues we were seeing in Lambda. To get around the SQS size limit issues we faced, we swapped in a Redis-backed queuing service. We had to reinvent some wheels that Lambda had given us for free - like logging, autoscaling, and health checks - but in the end we had our 6-step test integration back to running in under 2 seconds.

Now, ECS hasn't been perfect - there are are some trade-offs. For one, ECS doesn't seem to autoscale as quickly as Lambda. A "scale up" seems to take about a minute or so between API call and AWS Fargate pulling down and initializing a container that's ready to accept jobs. We had to pull one of our devs off of product development to work on cloud infrastructure, and there's a ton more to juggle with regards to CPU and memory usage, autoscaling rules, and monitoring, but at this point in product development the pains are worth the gains to give our customers a speedy integration runner.

What Remained in Lambda

We didn't move all of our microservices out of Lambda - plenty still remain in the serverless ecosystem and will for the foreseeable future. Our integration runner didn't fit Lambda well, but there are other tasks for which Lambda seems like the clear choice. We kept all important integration services that aren't critical to the actual execution of the integration in Lambda. Those include:

A logger service that pulls logs from ECS and sends them to DataDog.
A service that writes metadata about integration executions to a PostgreSQL database.
A service that tracks and queues scheduled integrations.
An alerting service that sends SMS or email notifications to users if their integrations error.
An authorization service that renews customers' OAuth 2.0 keys for third party services.

We didn't want any of these services to block execution of an integration, and for all of them it's fine if they take an additional second or two to run, so services like those fit Lambda perfectly.

Conclusion

Our infrastructure definitely changed over time, but I think the decisions we made along the way were the right ones: LocalStack's "Lambda" service let us develop and iterate very quickly, and our first deployment into AWS was simple enough that our small dev team could Terraform our infrastructure without losing a ton of dev hours to it.

Lambda seemed like an attractive solution for hosting and scaling our microservices, and for many of them, especially asynchronous services that might take a second or two to run, it still remains the correct choice. For our integration runner, though, we learned that the size, speed, and process isolation limitations of Lambda made ECS a better option, and it was worth the dev time it took to create an ECS deployment for that particular service.

Lambda let us concentrate on product development early on, and when the time was right the transition to ECS was a fairly smooth one. Even with the issues we faced in Lambda, I'm glad we took the path we did.

Announcing Prismatic General Availability!

Michael Zuercher — Tue, 30 Mar 2021 11:31:27 +0000

Today, we're excited to announce the general availability of Prismatic, the embedded integration platform for B2B software companies. We're on a mission to provide B2B software companies with a better way to build and deploy integrations for their customers, and today is a big milestone on that path.

A Little History

Almost two years ago, my cofounders and I were sitting around complaining about software integrations. You see, we'd spent fifteen years at my last software company building B2B software with tons of integrations, and it just got more and more painful the larger we got.

Even though the three of us had exited that business, we still had stuck in our heads how important integrations were to our customers and how hard they were to do well at scale. How they made moving core product forward harder, slowed down onboarding of customers, and made it harder to provide the level of support our customers needed and deserved.

At some point, we decided to stop talking and put our heads down at figuring out how to solve it. So we started Prismatic.

Learning and Building

We didn't know exactly what Prismatic needed to be, just that we wanted to fix integrations for software companies. So we started building some early prototypes and had what seemed like a million Zoom calls with some amazing people at great software companies.

It was immediately obvious that we weren't the only ones who had experienced "the integration problem." It affects software teams in every industry. And we heard a lot of the same desires over and over. Things like the need for productized integrations that could be built easily, a desire to become more responsive and efficient by reducing the integration workload on development, and a need for better tooling in order to support integrations well.

We also came to understand that although there are countless integration platforms out there, they don't serve B2B software companies very well. They're largely designed for enterprises automating their own workflows and don’t work for B2B software companies that need to provide integrations to customers. They lack the concept of customers (and their configs, credentials, etc.). They lean too hard on a “no-developer” mindset. And they don’t fit into software companies' existing infrastructure and processes well.

As our understanding grew, we formed a clear viewpoint on what it would take to build an integration platform that actually works in the real world of B2B software companies. And we continued to learn. And we iterated. And learned. And iterated.

Introducing Prismatic

After almost two years of conversations, learning, and building, Prismatic has evolved into the platform that B2B software companies have been waiting for. We've been extremely encouraged by the reception it has seen from software teams and we're eager to get it in the hands of more people.

Prismatic gives software companies a better way to build integrations and also a better way to manage their deployment to customers. It includes:

Purpose-built infrastructure that runs your integrations in a secure, scalable environment
An intuitive integration designer for building reusable integration workflows
Integration deployment and support tools like customer-specific configuration, logging, and alerting
An embeddable, white-labeled customer portal with self-service integration tools

And it does this in a way that gives every person involved exactly what they need:

Devs get the power and flexibility to extend the platform to fit your product and the way you build software (strong custom component support, fully exposed API, robust CLI)
Non-devs use a low-code designer, tailored by your devs to fit your product and industry, to build reusable integrations that can be deployed to multiple customers with different configs
Customers and customer-facing teams get a simple integration configuration and management environment

What's Next

Although today's launch is an important milestone, it's just the beginning. We're improving Prismatic every day with a roadmap that our customers are excited about. We're growing our team. And most importantly, we're moving faster than ever on our mission to provide B2B software companies with a better way to build and deploy integrations for their customers.

We'd Love To Hear From You

We're super passionate about better software integrations and helping software teams solve their integration challenges. If you're a member of a software team and are interested in what we're doing, we'd love to hear from you! Or check out Prismatic for free.

If you share our passion and are interested in joining us, check out our careers page - we're growing!

How to Connect to Private AWS Resources with SSH Tunnels and Bastion Hosts

Taylor Reece — Thu, 17 Dec 2020 15:03:29 +0000

The Problem with Publicly Accessible AWS Resources

When you first develop infrastructure for a new project, you naturally optimize for rapid development. You want to get something - anything - out the door, and you therefore want to be able to write code and debug issues quickly.

Because of that, it's awfully tempting to spin up servers and databases in public subnets so that you can readily connect to them for debugging sessions. It's nice to be able to ssh my-user@my-web-server to do some live code debugging, or psql -U my-user -h my-database-instance to assess the current state of your database.

Sure, AWS allows you to do that. You can expose SSH on your webservers and you can expose your AWS Relational Database Service (RDS) and Redis/ElastiCache servers to the internet for easy access. But should you? Exposing databases and servers publicly yields a pretty obvious security issue: your sensitive data is directly accessible to anyone on the internet.

You might argue "it's fine; we lock down our VPC security groups so that only people from our office IP can access our EC2, RDS, or ElastiCache instances. Other people on the internet are blocked because of their IP addresses". While that's true - locking down your resources this way is better than nothing - this strategy yields two pitfalls:

Allowing anyone from your office IP address to access your AWS resources is still a security issue. Should guests on your WiFi, or John the junior developer really have access to production databases? By locking down your resources by public IP, it's pretty hard to discriminate between DevOps users and the rest of your organization - everyone likely uses the same public IP address. Plus, as your organization grows, more attack vectors are created. A single compromised user on your network can wreak havoc on your production systems.
Your remote workers are going to struggle to hit resources. If your employees are working from home (which, in this COVID-riddled year, they very likely are), you will either need your employees to maintain a VPN connection to your office, or you will need to maintain a whitelist of all of your employees' home IP addresses. If your employees happen respond to an outage from an airport or coffee shop, they might spend minutes (hours?) manually monkeying with VPC security groups before they can even get access to address the outage.

This is a problem that I've had to solve everywhere that I've worked, so I imagine it's a problem others have had to face, too. Let's look at how to do it right, optimizing for both security and accessibility at the same time.

Enter AWS Systems Manager

Let's suppose you've followed security protocols, and have placed your various AWS resources (EC2, RDS, ElastiCache, etc.) into private subnets and removed public access to them. How can you securely grant access to members on your team (like your senior devs and DevOps) who need them? AWS provides Systems Manager to let you inspect and access your AWS resources - even those residing in private subnets. With Systems Manager, users exchange their AWS credentials for temporary shell access to EC2s. They can get that access both on the CLI, and through the AWS web console.

To enable access to your EC2s, simply install the AWS SSM agent onto the servers that you spin up. What's the deal with the extraneous "S" in "SSM", you might ask? It's historical - Systems Manager used to be called "Simple Systems Manager" and the extra "S" stuck - just go with it. Good news - if you're building your servers off of Amazon Linux, macOS, Ubuntu Server, or some Amazon-provided Windows Server images, the SSM agent is already installed for you - you just need to turn on a machine and grant it a specific role defined in AWS's docs.

Once you have an EC2 with the SSM agent installed, you can open a shell into your instance with the aws CLI tool:

$ aws ssm start-session --target i-0b6c737cc21dc01a9

Starting session with SessionId: treece-0bf6ff366c16d651f
sh-4.2$ whoami
ssm-user
sh-4.2$ cat /etc/system-release
Amazon Linux release 2 (Karoo)

When you have the SSM agent installed, you don't need to worry about juggling IP whitelists. Your systems management teams can access resources from anywhere (the office, at home, or even a coffee shop) using their AWS credentials. And, without AWS credentials John the junior dev has no access to production databases. Sorry, John!

What About Accessing RDS or ElastiCache?

With AWS Systems Manager you can remote into EC2s, but what about databases like PostgreSQL/RDS or Redis/ElastiCache? AWS doesn't allow you to directly SSH into the systems running RDS or ElastiCache. Instead, I suggest spinning up a minimal EC2 instance called a bastion in your VPC that you can remote into with Systems Manager. You can remote into the bastion, and once there you can access your databases.

Make sure that you assign your bastion a VPC security group, and create ingress rules so that your RDS and ElastiCache security groups allow access from your bastion security group. Once you have all that set up, you can access your databases from the command line on the bastion:

$ aws ssm start-session --target i-0b6c737cc21dc01a9

Starting session with SessionId: treece-048a3bf2269ad7caf
sh-4.2$ psql -h 10.0.2.88 -U testuser postgres
Password for user testuser:
SSL connection (cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256)
Type "help" for help.

postgres=> CREATE TABLE users (id INT, email VARCHAR(40));
CREATE TABLE

You might ask, "can't I just spin up a bastion in a public subnet, and SSH into it?" Yes and no. Yes, you can totally do that. No, it's not a good idea - that opens up the same security vulnerabilities that leaving your servers and databases in public subnets does.

Tunnels... Tunnels Everywhere

At this point, we have options for remoting in to EC2s directly through Systems Manager, and we can access our databases indirectly via a bastion EC2. The last thing I want to look at is leveraging SSH tunnels through the bastion to access database resources as though they're locally accessible. That way, you can connect your favorite database GUI, like pgAdmin to your RDS instances.

First, we need to get an SSH public key into our bastion. I'll assume you already have a key in $HOME/.ssh/id_rsa.pub. We'll use the aws CLI to temporarily load up that key for user ssm-user into our bastion:

$ aws ec2-instance-connect \
    send-ssh-public-key \
    --availability-zone us-east-1a \
    --instance-id i-0b6c737cc21dc01a9 \
    --instance-os-user ssm-user \
    --ssh-public-key file://$HOME/.ssh/id_rsa.pub
{
    "RequestId": "f8f3db2a-3107-4c0a-ae3d-94e2d7fff620",
    "Success": true
}

Next, I'm going to modify my laptop's $HOME/.ssh/config file so that servers starting with i-will proxy their connections through the aws ssm start-session command we used earlier. I'll add:

host i-*
    ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"

Now, I can simply ssh ssm-user@INSTANCE-ID to access my instance:

$ ssh ssm-user@i-0b6c737cc21dc01a9
Last login: Tue Dec 15 20:27:34 2020 from localhost

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/
10 package(s) needed for security, out of 22 available
Run "sudo yum update" to apply all updates.
[ssm-user@ip-10-0-0-71 ~]$ ls
file.txt system.log
[ssm-user@ip-10-0-0-71 ~]$

Now that I have SSH access to my bastion (even though it's in a private network that's not accessible from the internet!) I can spin up SSH tunnels like I would with any other SSH connection. To create a tunnel to my RDS instance, for example, I can simply run:

$ ssh ssm-user@i-0b6c737cc21dc01a9 -NL 5000:10.0.2.88:5432

And then from a separate shell I can access my database "locally" on port 5000:

$ psql -h localhost -p 5000 -U testuser postgres
Password for user testuser:
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.

postgres=> \dt
         List of relations
 Schema | Name  | Type  |  Owner
--------+-------+-------+----------
 public | users | table | testuser
(1 row)

Cool, huh? Using tunnels through your bastion you can access any resources that reside in your private network without needing to expose them to hooligans on the internet... or John the junior dev.