DEV Community: Taylor Reece

Six OAuth 2.0 Anti-Patterns to Avoid

Taylor Reece — Thu, 19 Mar 2026 19:48:42 +0000

I like to joke that developers who add the OAuth 2.0 authentication code flow to their apps read the spec, implement 90% of it, and then simply wing the last 10%. I say it in jest, but it's not far from the truth.

After building over 200 connectors (84 of which use OAuth 2.0 authentication), we've seen dozens of deviations from the spec that you absolutely should not mimic when adding OAuth 2.0 to your app.

What is OAuth 2.0?

But, before we cover what OAuth 2.0 shouldn't be, let's see what it is. OAuth 2.0 is a standard that allows one app to access data in another app on your behalf.

If you've ever clicked a "Log in with Google" button or selected "Click here to connect your Outlook calendar", you've likely gone through an OAuth flow.

How should OAuth 2.0 auth code work?

Suppose you're logged in to Acme. Acme has an integration with Dropbox, and to enable it, you click a handy button labeled "Click here to link your Dropbox account." Clicking that button will bring you to Dropbox's Authorize URL:

https://www.dropbox.com/oauth2/authorize?client_id=abc-123&state=foo&...

Once you arrive at Dropbox, you'll be presented with a consent screen – a screen that says something like "Acme would like access to files in thus-and-such Dropbox folder. Cool?" You're familiar with these screens, I'm sure.

Once you select "Yeah, grant access to Acme," Dropbox sends you back to Acme's Callback URL:

https://oauth2.acme.com/callback?state=foo&code=bar1234

At this point, Acme has your "auth code" (the code=bar1234 bit). Acme can now use Dropbox's Token URL to exchange that code (along with a client_secret) for an access_token and sometimes a refresh_token.

From there, Acme can use your access token to do whatever you permitted it to do, all without ever handing Acme your Dropbox username and password.

Anti-patterns

The OAuth 2.0 authorization code spec is pretty straightforward: users are redirected through an Authorize URL, consent to some things, return to the calling app with an auth code, and the auth code is exchanged for an access token.

Straightforward, yes? Let's look at some common ways you can deviate from that simple pattern.

Token expiration in nanoseconds?

When an app completes the auth code exchange, it receives a payload that looks something like this:

{
  "access_token": "2YotnFZFEjr1zCsicMWpAA",
  "token_type": "example",
  "expires_in": 3600,
  "refresh_token": "tGzv3JOkF0XG5Qx2TlKWIA"
}

The expires_in property is defined in section 4.2.2 of the OAuth 2.0 spec as "The lifetime in seconds of the access token."

Guess what happened when one of our customers got back an expires_in of 3,600,000,000,000? You guessed it. Our OAuth service set itself a reminder to refresh the token in 3.6 trillion seconds (or 114,077 years).

Obviously, that was not the app developer's intention. They were just pedantic and loved the precision of nanoseconds over seconds. However, doing it this way is an anti-pattern, and resulted in this lovely bit of code in our OAuth 2.0 service:

// Apparently some people feel the inexplicable need to use nanoseconds as
// the unit of measure for the expiration time. For now I think the best
// we can do is recognize a "stupidly large" value and assume that it's
// probably using a silly unit of measure, and "fix" it. We'll do this
// for nanoseconds for now, as guessing should be pretty safe.
if (intExpiresIn > 31_540_000_000) {
  // If it looks like the number of seconds is more than 1000 years,
  // it's quite likely that the unit of measure is something dumb
  // like nanoseconds. So divide it by a billion...
  intExpiresIn /= 1_000_000_000;
}

Token expiration in seconds vs at time

The spec clearly calls for expires_in – the number of seconds from now until the token expires. Despite that, some apps will try to save you some compute cycles by returning an expires_at timestamp instead. They give you something like "expires_at": "2026-03-05T15:36:41.304Z".

The spec doesn't mention an expires_at value, so it's something extra to account for. To compound the problem, there's no standardization of the date/time format for expires_at. While some apps give you a UTC timestamp in ISO 8601 format, others yield the number of seconds (nanoseconds?) since Unix Epoch, which means you need to do some amount of computation anyway, and leads to this part of our OAuth 2.0 service:

// Handle the situation where we got an expires_at but no expires_in
if (!token.expires_in && token.expires_at) {
  const expiresAt = new Date(token.expires_at);
  intExpiresIn = Math.floor((expiresAt.getTime() - Date.now()) / 1000);
}

Auth code search parameters vs fragments

When you return to your original app's callback URL with a ?code=some-auth-code search parameter, the server that responds to the callback request exchanges the auth code for an access token. Section 4.1.2 of the spec is clear: this needs to be a URL query parameter.

Despite that, some apps opt to return a URI fragment (so, /callback#code=some-auth-code instead). Their justification is that fragments aren't stored in browser history (unlike search parameters), so your auth code is more secure. But auth codes are supposed to be invalidated shortly after issuance anyway – so not really a problem.

This creates a headache because fragments aren't readable by the callback server. Instead, the frontend needs to read the fragment and act on it.

To work around this, our customers have created an intermediary endpoint that includes some HTML/JS that converts the fragment into a search parameter, then redirects to our callback URL with the auth code as a search parameter.

Naming conventions

Have you ever gotten into an argument about snake_case vs camelCase vs kebab-case? Developers have strong opinions on this – especially if they have backgrounds in different tech stacks.

When you're creating an OAuth 2.0 service, though, your opinion on the topic doesn't matter – the spec requires snake_case.

Despite this, some apps use clientId or client-id instead of the correct client_id; ditto for grantType or client-secret. Others make up an entirely new term, like appSecretKey for client_secret.

Please use snake_case. OAuth libraries don't map fields gracefully, and using anything other than what's specified requires special treatment for anyone wanting to integrate with your app.

Error responses

Section 5.2 of the OAuth spec outlines the expected error structure when an error occurs during token exchange. You need to provide an error that is an enum with a value like invalid_scope or unauthorized_client. If you need to provide more details, an optional error_description string can be included. An error may look like this:

{
  "error": "invalid_scope",
  "error_description": "You asked for a scope of 'widgets.red' - do you mean 'widgets.read'?"
}

OAuth 2.0 libraries expect this format. Despite the clear spec, I've seen apps throw standards to the wind and return nested objects like:

{
  "error": {
    "code": "invalid_request",
    "description": "Invalid token"
  }
}

This makes debugging tricky. A console.debug(error) yields an oh-so-helpful [object Object]!

Made-up OAuth 2.0 flows

A grant_type defines the OAuth 2.0 flavor you're using, and authorization_code is pretty standard for auth that requires user interaction. Other apps use client_credentials (machine-to-machine OAuth). Some older apps use the deprecated password OAuth flow.

Only one app on the internet uses the account_credentials flow – a made-up OAuth flow that's really client_credentials under the hood.

On to butterflies, now that we've handled snakes

If you're going to add OAuth 2.0 to your app, please read the spec and then follow it. Deviation from that doesn't make you a special butterfly, but it does make your app an edge case that anyone who integrates with it must explicitly handle.

A good practice to follow, once you have an OAuth 2.0 service in place, is to grab a few off-the-shelf OAuth clients, like simple-oauth2 for Node.js or oauth2-client for PHP, and verify that what you've built is compatible with the tools other apps will use to integrate with you. As a bonus, it's also the best way to ensure that your code won't show up in the next version of this post.

SOAP APIs Aren't Scary: What You Should Know Before You Build a SOAP Integration

Taylor Reece — Wed, 23 Mar 2022 20:05:09 +0000

Building your first integration with a SOAP-based API can be daunting.

Recently I’ve helped several companies integrate their apps with third-party apps and services that use SOAP-based APIs. For developers with SOAP experience, the integrations were a breeze. But for the uninitiated, SOAP has a pretty steep learning curve and throws a lot of new terms and acronyms your way: XML, WSDL, envelope, procedure... and a few dozen others.

Today I want to look at some basic concepts of SOAP, then dive into some tools and resources that make SOAP integrations a lot easier to build (after all, the “S” in SOAP stands for “simple”)!

What is SOAP?

First, let’s talk about some SOAP basics. If you feel like reading through the entire SOAP spec and its history, you can over at w3.org. I’ll try to give the tl;dr here.

Originally, SOAP stood for “Simple Object Access Protocol.” It was largely built to handle CRUD (create, read, update, delete) operations and it was a protocol to access your objects (like inventory or customer data) in a simple way. A SOAP API would advertise the types of objects that outside services could fetch and manipulate and would outline exactly how a third party should make CRUD requests.

SOAP has since dropped that acronym because it does a lot more than object manipulation, but the idea is still the same: it’s a protocol for two systems to communicate with one another in a predictable way.

Web services and WSDLs

A SOAP API provides a web service. These web services are made up of one or more procedures (called operations). For example, you might create a “weather forecast” web service, and it could include a “get the 10-day forecast” operation, or an “I’m a weather station, and here’s our latest temperature data” operation.

Requests to each operation take very specific formats for their input, and they have predictable formats for their responses. The “get the 10-day forecast” operation might take a numerical zip code as its input and return an array of floating point temperatures in Fahrenheit that are forecast over the coming ten days for that zip code.

When SOAP-based web services are published, a summary WSDL (Web Services Description Language) file is published. The WSDL, written out in XML, outlines the various operations that the service offers, as well as what inputs and responses you can expect for each operation.

Let’s look at a simple WSDL for a rudimentary “calculator service.” The WSDL (here) defines some basic calculator operations (like addition, subtraction, multiplication, and division). This portion of the calculator WSDL declares that the service has an addition “Add” operation:

<wsdl:operation name="Add">
 <wsdl:documentation xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/">
  Adds two integers. This is a test WebService. ©DNE Online
 </wsdl:documentation>
 <wsdl:input message="tns:AddSoapIn"/>
 <wsdl:output message="tns:AddSoapOut"/>
</wsdl:operation>

If we look further into this XML, we see that the wsdl:input references a message type tns:AddSoapIn. Looking a few lines prior where some parameters are declared, we see this section:

<wsdl:message name="AddSoapIn">
 <wsdl:part name="parameters" element="tns:Add"/>
</wsdl:message>

That portion of the WSDL references an element tns:Add, which is found in another previous section of the WSDL. The tns:Add element defines a complex input type with a couple of integers:

<s:element name="Add">
 <s:complexType>
  <s:sequence>
   <s:element minOccurs="1" maxOccurs="1" name="intA" type="s:int"/>
   <s:element minOccurs="1" maxOccurs="1" name="intB" type="s:int"/>
  </s:sequence>
 </s:complexType>
</s:element>

Now, one criticism of SOAP is that it is incredibly verbose - this WSDL is no exception, but in layman’s terms all that the above XML really says is “this service has an Add operation and it takes two integers, intA and intB, as inputs.”

Consuming a SOAP API

Okay.... there’s an addition procedure out there that we want to invoke and it takes two integers. Great. How do we go about calling it? SOAP APIs are almost always HTTP-based, so we need to craft an HTTP request from the WSDL we looked at.

SOAP APIs always communicate with XML, so we’ll start by adding a header that sets our Content-Type to text/xml; charset=utf-8.

The data we send needs to be in XML format, and SOAP wraps data sent back and forth in a SOAP envelope (an envelope defines the message’s structure). If we look at the calculator WSDL again, we can glean the envelope’s structure. Within our SOAP envelope’s “body” we’ll declare what procedure we’re going to run (Add), and we’ll include values for the inputs that the procedure expects (for intA and intB). Our HTTP request will end up looking like this:

curl http://www.dneonline.com/calculator.asmx \
  --request POST \
  --header "Content-Type: text/xml; charset=utf-8" \
  --data \
'<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
 xmlns:soap="http://www.w3.org/2003/05/soap-envelope">
 <soap:Body>
    <Add xmlns="http://tempuri.org/">
     <intA>15</intA>
     <intB>27</intB>
    </Add>
 </soap:Body>
</soap:Envelope>'

After running that HTTP requests from our command line, we get a response that’s also in XML, and also wrapped in an envelope. The response’s body includes the AddResponse that’s defined in the WSDL, and that has an AddResult - the sum of the two numbers we sent to the server:

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope
 xmlns:soap="http://www.w3.org/2003/05/soap-envelope"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xmlns:xsd="http://www.w3.org/2001/XMLSchema">
 <soap:Body>
  <AddResponse xmlns="http://tempuri.org/">
   <AddResult>42</AddResult>
  </AddResponse>
 </soap:Body>
</soap:Envelope>

Phew... that’s a hell of a lot of parsing through a WSDL and monkeying with XML schema just to add two numbers together. It’s nice to have predictable request and response formats, but a missing xmlns XML attribute or typo’d operation name can result in the SOAP server rejecting your requests. Hand-writing these XML requests is pretty error-prone. Let’s reach for some tools to simplify things.

SOAP tooling

Obviously manually parsing a WSDL with your eyes and hand-writing HTTP requests is burdensome and should be avoided at all costs. I recommend reaching for a SOAP tool that parses the WSDL for you and presents you with a list of operations (and their inputs) that you can invoke. There are several great apps out there built for exactly that purpose:

SoapUI from SmartBear is a popular tool that gives you a great graphical UI for navigating through a WSDL. Popping open any operation gives you a sample request that you can fill in and execute from within their app:
Recent versions of the popular HTTP client Postman also support importing a WSDL, and it even provides you with pre-populated sample requests that you can send to the web service from within the app:

These graphical tools are great for exploring a SOAP API, and I recommend you familiarize yourself with the web service’s operations prior to sitting down and coding up your integration. Once you get a good idea of how the SOAP API works, it’s time to turn to code.

Writing code to interact with SOAP APIs

Manually parsing a WSDL and hand-writing HTTP requests is error-prone and just generally an awful experience. You definitely don’t want to resort to templating out XML with string literals. Let’s avoid that when writing code to interact with a SOAP-based web service.

Whether you’re using Python, NodeJS, C#, or any other modern programming language, the chances are good that there’s a SOAP library you can leverage to make it simple to invoke web service operations. In NodeJS, for example, you can reach for the aptly named soap package on NPM.

This library can take the URL of a WSDL as an argument, and in doing so creates a fully functional HTTP client object that can invoke any of the web service’s operations. We can perform something like client.Add to invoke the “Add” operation, pass in a couple of input arguments, and the rest (parsing the WSDL, formatting the XML request in an envelope, deserializing the XML response, etc.) is handled for you. With just a half-dozen lines of code (I don’t count parentheses!), we can make a request to the calculator API:

var soap = require("soap");
const wsdlUrl = "http://www.dneonline.com/calculator.asmx?WSDL";
const args = { intA: 15, intB: 27 };
soap.createClient(wsdlUrl, function (err, client) {
  client.Add(args, (err, result) => {
    console.log(`The API returned a sum of ${result.AddResult}`);
  });
});

The response to our request contains an AddResult property, and our function prints out The API returned a sum of 42, like we’d expect.

Abstracting SOAP Further

If you’re building integrations that connect your own product to other apps your customers use, an embedded integration platform as a service (embedded iPaaS) like Prismatic can abstract much of the complexity of interacting with SOAP APIs. Embedded integration platforms include low-code integration builders with pre-built connectors for popular SOAP-based apps (like Salesforce), which completely eliminates the need to write code to interact with many SOAP-based APIs.

For less common SOAP-based APIs that don’t have built-in connectors, they typically offer a SOAP component that abstracts out the complexity of making SOAP requests:

In many situations, you can leverage an embedded iPaaS to interact with a SOAP-based API while writing little to no code.

Conclusion

Obviously the calculator web service is incredibly simple, but I think it does a good job illustrating how a SOAP web service can appear daunting at first, but with the right tooling it’s relatively clean and simple to build an integration with a SOAP-based API.

Proxying Github Through a Bastion

Taylor Reece — Tue, 22 Feb 2022 15:37:39 +0000

Despite the status page saying otherwise, GitHub is down for part of the midwest. Though a 4-day weekend would be cool, getting access to code would be even cooler.

So... let's get to GitHub via a SOCKS proxy!

What is SOCKS?

SOCKS simply stands for "socket secure". It's a way by which you can proxy your web traffic on your computer through another system. All of your HTTP and HTTPS requests are piped through an SSH connection, and another system essentially makes connections for you on your behalf.

Why use SOCKS?

If you're either on a network that's locked down, or your network happens to be suffering an outage (like the midwest GitHub outage right now), but other networks are fine (AWS us-east-1 can hit GitHub right now), it's advantageous to have the other network make connections for you on your behalf.

Let's Make a Connection!

Alright, you're going to need a Linux box sitting somewhere that can access GitHub. I spun up an Ubuntu box in AWS's us-east-1 region, which is not experiencing an outage. I made sure I can SSH into the external box, then I ran this command on my command line:

ssh -D 1337 -q -C -N -i ~/my-key.pem ubuntu@my-endpoint

Replace my-key.pem with your SSH private key, and my-endpoint with your EC2's endpoint. The 1337 can be changed to whatever local port you'd like to use.

Configure SOCKS Proxy

Next, on MacOS pop open your network settings, click Advanced, then under the Proxies tab select SOCKS Proxy. Enter localhost and the port you chose (like 1337):

Windows has similar settings, and if you're on Linux you probably already know what you're doing.

Head on over to WTF is My IP to verify that your browser is proxying through your EC2, then pop on over to GitHub. Your connection should proxy through AWS, and you should be able to hit the GitHub website!

Configuring the GitHub CLI

Your GitHub CLI might connect at this point. That depends on it it respects your global SOCKS configuration, and if your checkout is via https:// or git://. You may need to set the http.proxy config flag, explained on this Stack Overflow post.

What if I use SSH?

If you use SSH to interact with GitHub, rather than HTTPS (which is the smart thing to do) you can pop open your SSH config file (likely at $HOME/.ssh/config) and throw in this SSH proxy config:

Host my-proxy
  HostName my-ec2-endpoint
  User ubuntu
  IdentityFile ~/my-ec2-key.pem

Host github.com
  ProxyJump my-proxy

That'll make it so your SSH requests to GitHub proxy through your Bastion in AWS. Make sure you remove this later when you can directly access GitHub!

Building an Integration to Another App: How to Get Started

Taylor Reece — Wed, 02 Feb 2022 16:20:51 +0000

[ding 🔔] goes your email notification. You’ve just been assigned a new ticket. The subject line reads “Integrate our app with Acme CRM.” The ticket’s description is pretty sparse - just a few bullet-points saying things like:

Sync users’ appointments in Acme with our calendar app
Track the number of email and phone touch-points users have in Acme on our dashboard

You expected this. A lot of the businesses that use your software also use Acme CRM, and you’ve been hearing from your sales and support teams that customers keep asking for their Acme data to sync with your app.

So.... it’s up to you to build the integration with Acme. Where do you even begin?

I know I’m always inclined to fire up my IDE and dive right in to code, but that always results in misunderstanding expectations, missing details, and having to redo work.

Let’s talk about the stuff you should do before you start coding. Let’s think through the questions you should ask, requirements you should clarify, and tools you can reach for to get a feel for how the integration is going to work.

Gather requirements for your integration project

In an ideal world, the person requesting the integration would provide you with a specification that details what events trigger the integration to run, what type of data gets sent over, and how the data should look when it arrives at its destination. The world’s rarely ideal, though, so you’ll probably have to go digging for clearer integration requirements.

It’s incredibly helpful to have a series of “When foo then bar” statements. For example, “When a user in Acme CRM sets an appointment with a customer, a calendar event is created in our app that contains the customer’s name, phone number, email address, and time of the appointment.” Bonus points if you can tie each step to relevant API documentation!

As you think through the “when foo then bar” statements, there are several companion questions you should be asking with regards to how data should flow from one system to another:

What triggers an integration to run? Does Acme support events like webhooks (where they send data whenever it’s been changed) or are you expected to fetch data on a schedule?
Is this a one-way or two-way integration (that is, will Acme just send data to your app, or does your app also need to send data back to Acme)?
What’s the format of the data the source system is sending? JSON? XML? CSV?
What information is included in the payload that’s sent your way?
What format of data is the destination system expecting? You may need to fetch more information from other third-parties or Acme’s other API endpoints to satisfy the destination’s requirements.

After you have a sense of how data will flow, ask yourself what things could go wrong, and clarify how your integration needs to behave in these common situations:

How should you deal with data “collisions” where the same data gets sent over twice? Should you ignore the second request, create a second record, or update (”upsert”) the existing data?
What should happen if incomplete or “bad” data is shipped (a calendar event that’s missing date, for example)? Do you need a “dead letter queue” to store data that causes your integration to error out so your devs or support folks can examine the bad data later?
What happens if either API is down? Is your integration expected to hold on to data and retry later?

Next, figure out how configurable you need this integration to be. (Occasionally, I’ve needed to build “one-off” integrations for a single customer. Far more often, I’ve needed to build reusable integrations that can behave differently from customer to customer and handle each customer’s credentials for the third-party app.)

How do users authenticate with each system? Do users have their own credentials to Acme, or some shared key? Where are you going to store those secrets securely?
What things are going to be different between customers you deploy this integration to? Do customers have different API endpoints, or data mapping requirements?
What all needs to be configurable to make it so you can deploy the same integration to multiple customers?

Finally, figure out deployment and support concerns. If your company has several app integrations already, you may already have answers to these questions:

How will your customers activate and configure the integration? Do you need to build UX for this?
What are the expectations for logging, monitoring and alerting? Where should the integration’s logs be sent, and under what conditions should someone be alerted if the integration runs into a problem?
Where is the integration itself going to live? Will you need to build custom infrastructure to host this integration?

As you can see, a quick bullet point spec like “Import users’ appointments into the our calendar” seems simple at first, but there’s a ton of things to think through if you want to build, deploy and support your integration thoroughly.

Get familiar with how the third-party app works

Once you’ve thought through all of the questions above and have a solid understanding of the task at hand, it’s time for you to familiarize yourself with the third-party app(s) involved. I recommend logging in to Acme CRM like an end user would. Figure out what exactly it means to “set an appointment” or “log a phone call with a customer” in the context of Acme. This should give you a sense of how different objects that the application stores are related.

If you don’t have an Acme CRM account, ask for one. It’s mutually beneficial for both your company and Acme to have an integration between your systems, and most reasonable companies are willing to provide a free developer account or “sandbox” environment where you can toy around with dummy data, without needing to access your users’ real Acme data.

You may be provided written information about how the other application will provide data to you, but I recommend that you insist on getting a real sandbox environment that at least sends fake data in the format the third party says they use. I’ve been around for plenty of integrations where a third party claims they’ll send XML in thus-and-such a format using REST calls, but informs you after you build your integration that they decided to write out JSON messages to a message broker (like Kafka or Amazon SNS) instead. You can just change your code quick to subscribe to an SNS feed, right? Testing with an actual system is far better than mocking out fake APIs and message streams yourself.

It sounds silly, but also make sure to log in to your own app as well to verify that you know what it means to “create a calendar event” or “update the dashboard.” Verify with the people requesting the integration that you understand what actions should trigger the integration, and where the data should be displayed when it arrives at its destination.

Explore the API with an HTTP client

You’re ready to start fiddling with APIs, but it’s not quite time to code quite yet. Take a look at Acme’s developer docs. If they have an OpenAPI/Swagger or WSDL spec that defines all for their API endpoints, you’re in luck! Figure out how their auth flow works, what data their webhooks send, and what data is available for you to query. If you’re really lucky, Acme will provide a client library you can use when it’s time to start coding (but don’t get your hopes up!).

Fire up your favorite HTTP client (I personally like Postman), and try to make some simple authenticated GET and POST requests against the Acme CRM sandbox you have access to. Verify that the third-party docs are correct, and that you can, indeed, fetch data from their appointment API endpoint. Double-check that the payload you receive has all the information you need to map data over to your app.

Once you have requests to Acme CRM working in Postman (or your favorite HTTP client), finally turn to code.

Time to build!

Now that you have a good idea of the shape of data that comes in, where additional data should be fetched from, how it should be modified and where it should wind up, it’s time to start building your integration.

There’s still a lot to do - you need to write your code, make it configurable enough to handle differences between customers, deploy it, monitor and support it, make the integration available within your product, etc., but asking the right questions and taking the proper steps beforehand can make all that go faster and keep rework to a minimum.

About Taylor

Taylor is a developer advocate at Prismatic (an integration platform for B2B software companies) where he helps customers build integrations connecting their products to their customers’ other apps. If you’d like to chat with Taylor or the folks at Prismatic, please reach out - we’d love to chat about the integrations you need to build for your app!

Built-in Node Functions Can Be Overridden Between Lambda Runs

Taylor Reece — Fri, 23 Apr 2021 15:31:43 +0000

A couple days ago I wrote up a post on Why We Moved From Lambda to ECS. One thing I was startled by was how a "warm" Lambda can behave between invocations. I thought I'd expand on that issue briefly with a simple example:

How I Thought Lambda Worked

My (incorrect) understanding of Lambda was that each invocation of a Lambda was isolated. I figured, besides the startup code that runs once when a "cold" Lambda gets warmed, that Lambda executions were stateless and wouldn't affect one another.

How It Actually Works

In reality, two Lambdas that run in parallel are perfectly isolated. However, if a Lambda has run and is sitting around "warm", subsequent executions using that Lambda might be affected by a previous execution.

Example Overriding console.log()

Suppose, for example, you have a simple Lambda with this code in index.js:

exports.handler = async (event) => {
  console.log("Hello, world!");

  global.console.log = (msg) => {
    console.error("Your message has been hijacked");
  };
};

The first time this Lambda is invoked, you see a happy "Hello, World!" logged out.

If you test the Lambda again while the Lambda is still warm, though, the first invocation overrode the console.log() function, so you end up seeing an error, "Your message has been hijacked".

You can imagine how someone might exploit this issue if they can invoke their own code in a Lambda that others then use.

Why We Moved From Lambda to ECS

Taylor Reece — Wed, 21 Apr 2021 14:46:26 +0000

After many months of development, my team just announced the general availability of our platform. That milestone seems like a perfect opportunity to look back and reflect on how the infrastructure that supports Prismatic has evolved over time. (Spoiler: We ended up moving our most important microservice from Lambda to ECS.)

In this post I'll dive into what went well in Lambda, what challenges we faced, and why we eventually made the decision to migrate some services from Lambda to AWS Elastic Container Service (ECS).

What Problem are We Solving?

For some quick context, our product is an integration platform for B2B software companies. That is, we help software companies build integrations and deploy those integrations to their customers. A simple integration might look something like this:

Step 1: Pull down an XML document from Dropbox.
Step 2: Process the XML with some custom JavaScript code.
Step 3: Use some stored credentials to post the processed data to a third-party API.

Our users can configure integrations to run on a schedule, or they can trigger them via a webhook, and our platform takes care of running, logging, and monitoring the integrations (and a whole bunch of other things).

The Early Days

The first incarnation of Prismatic used LocalStack. We knew that we wanted to eventually host Prismatic in AWS (with the possibility of moving to Azure, GCP, etc. as needed), so the ability to spin up our platform locally to simulate AWS was appealing. The LocalStack service that approximates AWS Lambda was easy to iterate on, and ran without any major hiccups. It gave us a great development feedback loop, so we could prototype and test very quickly.

We used Lambda to execute each "step" of an integration, and steps leveraged SQS to pass data and trigger the next step. So, an integration execution would look like this:

Run a Dropbox "fetch a file" action to grab an XML file.
Save the contents of that XML file to SQS, trigger the next step.
Run a customer's custom JavaScript code to process the XML.
Save the resulting transformed data to SQS, trigger the next step.
Run an action to post the processed data to a third-party API.
Save the results of the final step, trigger the end of the integration.

Within LocalStack, this was a very quick process. We could define a 6-step integration, run it, and see results within a couple of seconds.

Our Migration to Real AWS Lambda

Once we had a proof of concept working, we devoted some time to moving Prismatic to an actual production environment, with real Lambdas, queues, databases, etc. We were still a small team, and we didn't want to dedicate a ton of time to DevOps-y, infrastructure problems yet. We wanted to dedicate most of our time to our core product, and Lambda let us do just that.

Lambda was attractive to us for a number of reasons. We didn't need to worry about CPU or memory allocation, server monitoring, or autoscaling; that's all built-in. We were able to throw .zip files full of JavaScript code at Lambda, and AWS took care of the rest. Lambda let us compartmentalize our code into a series of microservices (a service for logging, a service for OAuth key renewals, a service for SMS/email alerting if integrations error out, etc.), so we could keep a good mental map of what code is responsible for doing what task. Costs were pretty reasonable, too - you just pay for compute time, so rather than running servers 24/7, we just paid when our prototypes were executing something.

After a few days monkeying with Terraform, we had our second incarnation of Prismatic in AWS. Our integration runners ran on real Lambda, and were triggered via SQS. This is the point at which we started running into performance issues with our integration runners.

Why Lambda Didn't Work for Us

We had a number of issues, ranging from speed to SQS size limits and lack of process isolation in Lambda, that caused us to reconsider its effectiveness as our integration runner. Let's talk about each of those issues:

Speed. Remember the 6-step integration that I said took a couple of seconds to run within LocalStack? It took a full minute using real Lambda and AWS. The actual Lambda invocations were quick - usually a few milliseconds. The writing of step results to SQS and subsequent execution of the next step, though, ended up taking multiple seconds every step. For more complex integrations, like ones that looped over 500 files, that was a show-stopper - who wants their integrations to take minutes (hours?) to complete?

We tried a number of things to get our Lambda invocations to go faster. We followed guides to keep a number of Lambda instances "warm", and we cranked up the number of vCPUs powering our Lambdas to the highest we could at the time (6 vCPUs / 10GB RAM), but those things only shaved single digit percentages off of our integration run times.

SQS Size Limits. SQS limits message size to 256 kilobytes. The amount of data being passed between steps of an integration often exceeded that size (after all, it's totally reasonable for an integration developer to pull down a multiple megabyte JSON file to process). We were able to work around this size limitation - the recommended solution that we followed was to write out payloads to S3 and pass references to S3 objects via SQS - but this extra API call to S3 only compounded our slowness issues.

Process Isolation. This was the issue that surprised me the most. At first, AWS Lambda seems appealing as a stateless compute engine - run a job, exit, run another job, etc - scale horizontally as needed. We naively assumed that Lambda invocations were isolated from one another, but that turned out to only be half true. Concurrent invocations are isolated (they run in distinct containers within Lambda). However, subsequent invocations reuse previous "warm" environments, so an integration runner might inherit a "dirty" environment from a previous integration run. That's especially a problem if you let customers write their own code, like we do for our customers' integrations.

It turns out that if one customer writes some bad code into their integration - something like this, global.XMLHttpRequest = null;, then subsequent integration runs on that same Lambda that depend on the XMLHttpRequest library error out. This is a big deal, since one customer could break something like axios for another customer. A customer could even be malicious and execute something like global.console.log = (msg) => { nefariousCode(); }, and other integrations that execute on that same Lambda will run nefariousCode() whenever they invoke console.log(). Yikes!

We tried a few things to get around this issue of shared execution space. We toyed with forcing our Lambdas to cold-start every time (which was a terrible idea for obvious reasons), and we tried spinning up distinct Node processes within chroot jails. Neither option panned out - spinning up child Node processes in a Lambda took 3-5 seconds and partially defeated the purpose of being in Lambda in the first place.

Our Move to ECS

Lambda had served us well with development - we were able to iterate quickly and get a prototype out the door, but with the myriad issues we faced in Lambda we decided to bite the bullet and dedicate some dev time to cloud infrastructure.

Our team got to work expanding our existing Terraform scripts, and moved our integration runner to AWS Elastic Container Service (ECS). Within an ECS container we could easily (and quickly!) chroot and isolate Node processes from one another, solving the process isolation issues we were seeing in Lambda. To get around the SQS size limit issues we faced, we swapped in a Redis-backed queuing service. We had to reinvent some wheels that Lambda had given us for free - like logging, autoscaling, and health checks - but in the end we had our 6-step test integration back to running in under 2 seconds.

Now, ECS hasn't been perfect - there are are some trade-offs. For one, ECS doesn't seem to autoscale as quickly as Lambda. A "scale up" seems to take about a minute or so between API call and AWS Fargate pulling down and initializing a container that's ready to accept jobs. We had to pull one of our devs off of product development to work on cloud infrastructure, and there's a ton more to juggle with regards to CPU and memory usage, autoscaling rules, and monitoring, but at this point in product development the pains are worth the gains to give our customers a speedy integration runner.

What Remained in Lambda

We didn't move all of our microservices out of Lambda - plenty still remain in the serverless ecosystem and will for the foreseeable future. Our integration runner didn't fit Lambda well, but there are other tasks for which Lambda seems like the clear choice. We kept all important integration services that aren't critical to the actual execution of the integration in Lambda. Those include:

A logger service that pulls logs from ECS and sends them to DataDog.
A service that writes metadata about integration executions to a PostgreSQL database.
A service that tracks and queues scheduled integrations.
An alerting service that sends SMS or email notifications to users if their integrations error.
An authorization service that renews customers' OAuth 2.0 keys for third party services.

We didn't want any of these services to block execution of an integration, and for all of them it's fine if they take an additional second or two to run, so services like those fit Lambda perfectly.

Conclusion

Our infrastructure definitely changed over time, but I think the decisions we made along the way were the right ones: LocalStack's "Lambda" service let us develop and iterate very quickly, and our first deployment into AWS was simple enough that our small dev team could Terraform our infrastructure without losing a ton of dev hours to it.

Lambda seemed like an attractive solution for hosting and scaling our microservices, and for many of them, especially asynchronous services that might take a second or two to run, it still remains the correct choice. For our integration runner, though, we learned that the size, speed, and process isolation limitations of Lambda made ECS a better option, and it was worth the dev time it took to create an ECS deployment for that particular service.

Lambda let us concentrate on product development early on, and when the time was right the transition to ECS was a fairly smooth one. Even with the issues we faced in Lambda, I'm glad we took the path we did.

How to Connect to Private AWS Resources with SSH Tunnels and Bastion Hosts

Taylor Reece — Thu, 17 Dec 2020 15:03:29 +0000

The Problem with Publicly Accessible AWS Resources

When you first develop infrastructure for a new project, you naturally optimize for rapid development. You want to get something - anything - out the door, and you therefore want to be able to write code and debug issues quickly.

Because of that, it's awfully tempting to spin up servers and databases in public subnets so that you can readily connect to them for debugging sessions. It's nice to be able to ssh my-user@my-web-server to do some live code debugging, or psql -U my-user -h my-database-instance to assess the current state of your database.

Sure, AWS allows you to do that. You can expose SSH on your webservers and you can expose your AWS Relational Database Service (RDS) and Redis/ElastiCache servers to the internet for easy access. But should you? Exposing databases and servers publicly yields a pretty obvious security issue: your sensitive data is directly accessible to anyone on the internet.

You might argue "it's fine; we lock down our VPC security groups so that only people from our office IP can access our EC2, RDS, or ElastiCache instances. Other people on the internet are blocked because of their IP addresses". While that's true - locking down your resources this way is better than nothing - this strategy yields two pitfalls:

Allowing anyone from your office IP address to access your AWS resources is still a security issue. Should guests on your WiFi, or John the junior developer really have access to production databases? By locking down your resources by public IP, it's pretty hard to discriminate between DevOps users and the rest of your organization - everyone likely uses the same public IP address. Plus, as your organization grows, more attack vectors are created. A single compromised user on your network can wreak havoc on your production systems.
Your remote workers are going to struggle to hit resources. If your employees are working from home (which, in this COVID-riddled year, they very likely are), you will either need your employees to maintain a VPN connection to your office, or you will need to maintain a whitelist of all of your employees' home IP addresses. If your employees happen respond to an outage from an airport or coffee shop, they might spend minutes (hours?) manually monkeying with VPC security groups before they can even get access to address the outage.

This is a problem that I've had to solve everywhere that I've worked, so I imagine it's a problem others have had to face, too. Let's look at how to do it right, optimizing for both security and accessibility at the same time.

Enter AWS Systems Manager

Let's suppose you've followed security protocols, and have placed your various AWS resources (EC2, RDS, ElastiCache, etc.) into private subnets and removed public access to them. How can you securely grant access to members on your team (like your senior devs and DevOps) who need them? AWS provides Systems Manager to let you inspect and access your AWS resources - even those residing in private subnets. With Systems Manager, users exchange their AWS credentials for temporary shell access to EC2s. They can get that access both on the CLI, and through the AWS web console.

To enable access to your EC2s, simply install the AWS SSM agent onto the servers that you spin up. What's the deal with the extraneous "S" in "SSM", you might ask? It's historical - Systems Manager used to be called "Simple Systems Manager" and the extra "S" stuck - just go with it. Good news - if you're building your servers off of Amazon Linux, macOS, Ubuntu Server, or some Amazon-provided Windows Server images, the SSM agent is already installed for you - you just need to turn on a machine and grant it a specific role defined in AWS's docs.

Once you have an EC2 with the SSM agent installed, you can open a shell into your instance with the aws CLI tool:

$ aws ssm start-session --target i-0b6c737cc21dc01a9

Starting session with SessionId: treece-0bf6ff366c16d651f
sh-4.2$ whoami
ssm-user
sh-4.2$ cat /etc/system-release
Amazon Linux release 2 (Karoo)

When you have the SSM agent installed, you don't need to worry about juggling IP whitelists. Your systems management teams can access resources from anywhere (the office, at home, or even a coffee shop) using their AWS credentials. And, without AWS credentials John the junior dev has no access to production databases. Sorry, John!

What About Accessing RDS or ElastiCache?

With AWS Systems Manager you can remote into EC2s, but what about databases like PostgreSQL/RDS or Redis/ElastiCache? AWS doesn't allow you to directly SSH into the systems running RDS or ElastiCache. Instead, I suggest spinning up a minimal EC2 instance called a bastion in your VPC that you can remote into with Systems Manager. You can remote into the bastion, and once there you can access your databases.

Make sure that you assign your bastion a VPC security group, and create ingress rules so that your RDS and ElastiCache security groups allow access from your bastion security group. Once you have all that set up, you can access your databases from the command line on the bastion:

$ aws ssm start-session --target i-0b6c737cc21dc01a9

Starting session with SessionId: treece-048a3bf2269ad7caf
sh-4.2$ psql -h 10.0.2.88 -U testuser postgres
Password for user testuser:
SSL connection (cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256)
Type "help" for help.

postgres=> CREATE TABLE users (id INT, email VARCHAR(40));
CREATE TABLE

You might ask, "can't I just spin up a bastion in a public subnet, and SSH into it?" Yes and no. Yes, you can totally do that. No, it's not a good idea - that opens up the same security vulnerabilities that leaving your servers and databases in public subnets does.

Tunnels... Tunnels Everywhere

At this point, we have options for remoting in to EC2s directly through Systems Manager, and we can access our databases indirectly via a bastion EC2. The last thing I want to look at is leveraging SSH tunnels through the bastion to access database resources as though they're locally accessible. That way, you can connect your favorite database GUI, like pgAdmin to your RDS instances.

First, we need to get an SSH public key into our bastion. I'll assume you already have a key in $HOME/.ssh/id_rsa.pub. We'll use the aws CLI to temporarily load up that key for user ssm-user into our bastion:

$ aws ec2-instance-connect \
    send-ssh-public-key \
    --availability-zone us-east-1a \
    --instance-id i-0b6c737cc21dc01a9 \
    --instance-os-user ssm-user \
    --ssh-public-key file://$HOME/.ssh/id_rsa.pub
{
    "RequestId": "f8f3db2a-3107-4c0a-ae3d-94e2d7fff620",
    "Success": true
}

Next, I'm going to modify my laptop's $HOME/.ssh/config file so that servers starting with i-will proxy their connections through the aws ssm start-session command we used earlier. I'll add:

host i-*
    ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"

Now, I can simply ssh ssm-user@INSTANCE-ID to access my instance:

$ ssh ssm-user@i-0b6c737cc21dc01a9
Last login: Tue Dec 15 20:27:34 2020 from localhost

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/
10 package(s) needed for security, out of 22 available
Run "sudo yum update" to apply all updates.
[ssm-user@ip-10-0-0-71 ~]$ ls
file.txt system.log
[ssm-user@ip-10-0-0-71 ~]$

Now that I have SSH access to my bastion (even though it's in a private network that's not accessible from the internet!) I can spin up SSH tunnels like I would with any other SSH connection. To create a tunnel to my RDS instance, for example, I can simply run:

$ ssh ssm-user@i-0b6c737cc21dc01a9 -NL 5000:10.0.2.88:5432

And then from a separate shell I can access my database "locally" on port 5000:

$ psql -h localhost -p 5000 -U testuser postgres
Password for user testuser:
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.

postgres=> \dt
         List of relations
 Schema | Name  | Type  |  Owner
--------+-------+-------+----------
 public | users | table | testuser
(1 row)

Cool, huh? Using tunnels through your bastion you can access any resources that reside in your private network without needing to expose them to hooligans on the internet... or John the junior dev.