DEV Community: Patrick Walsh

A Checklist to Quickly Evaluate SaaS Security

Patrick Walsh — Mon, 27 Sep 2021 17:31:08 +0000

Large companies have security teams that scrutinize every partner and vendor they use. They put the vendor through the wringer with security reviews that usually involve long and tedious spreadsheets.

The result of those painful security reviews is generally an analysis on the relative level of risk that comes with the solution. Then the business stakeholder can choose to take the advice of security or overrule it (spending political capital in the process) to say that the benefits outweigh the risks.

The review process is painful for everyone and, frankly, despite the intentions of various startups and others, it seems to be getting worse, not better.

And we have another problem: only the biggest companies even get to evaluate the security of their vendors. The rest of us aren't privvy to the information needed to do a full risk analysis and we may not be sophisticated enough to do it even if the vendor is willing.

Security Evaluation for the Rest of Us

It's ludicrous to think that security only matters to large enterprises. Yet that's how the industry operates. So the rest of us are left to scrape together the breadcrumbs of publicly available information as part of our purchasing process.

My Credentials

I've been on both ends of security evaluations, I've run engineering teams building large, complex enterprise cloud software, and I'm the CEO of a company that helps SaaS providers harden their apps. I've seen a lot of the skeletons hiding in the closets of software vendors and I've developed a nose for approximating a risk analysis based on publicly available information.

These are the things I look for and the steps I take when evaluating cloud software if the vendor might hold any data that I feel is private.

Seeing Through the B.S.

The hardest part of evaluating software and services for security maturity is the sheer amount of bullshit that gets spewed. Companies frequently have polished web pages that say things like, "we take your security seriously." They then go on to talk up minor things as if they're extremely important. For example, if someone tells you they're using "military grade encryption", run away. Seriously. It's a big "B.S." signal.

Step 1: Find the security page

Note: for now, we're only evaluating their basic security posture and how well they're able to keep confidential data safe. We are not looking at integrity or availability.

Most companies have a page (or sometimes a downloadable PDF) that gives a high-level marketing-oriented view of their security. We're going to use that page to evaluate them, but the first trick is to find it. And that's not always as easy as you might hope. Here's what I do, in order:

Go to the website and look for a "Security" or "Trust Center" link in the footer. If you don't find it there, it's already a bit of a bad sign, but don't give up yet.
I do a Duckduckgo (or Google or whatever) search that looks something like site:company-x.com security to see what comes up. I'm just looking at page titles here.
If that fails, it's sometimes because this sort of information gets pushed into "resources" sections of a website as a download. That's the third place I check.
Finally, if all that fails, I go to the support pages. Sometimes a company's knowledge base isn't indexed for search by the major search engines or is on a different domain. Find the knowledge base and search it for "security" and often you'll find details about what the company does to be a trustworthy partner.

At this point, if you still haven't found a page discussing how a company is going to protect your data, they get an F and you should disqualify them with extreme prejudice.

Step 2: Check the boxes

Next you have to wade through statements like, "we care about your security" and "we use state-of-the-art encryption for all data, whether at rest or in transit" and "plans include SSL encryption to keep your data safe" to find where they make specific security claims. For example, "we care about your security" is not a specific claim, but saying they use SSL or encrypt "at rest or in transit" are statements that you can map back to the checklist.

We've made a spreadsheet that has both the checklist and some of the phrases you're looking for to figure out if each row should be checked or not. You can freely copy and modify the spreadsheet to suit your needs. It is licensed under the Creative Commons Attribution 4.0 International License, which just means we'd like you to give us credit if you redistribute it with or without modifications.

There are currently 23 checkboxes. This may evolve over time, but we'd like to keep the number low rather than comprehensive so it remains a lightweight way to evaluate a company's seriousness about security.

Step 3: Score it

The spreadsheet will do this for you. For the moment, we score one point for basic checkboxes, three points for intermediate, and five points for advanced. We put a maturity level at the bottom where anything under 15 is considered "weak," with "reasonable" and "exceptional" levels above that.

Example Scores

As of the time of writing, here's what we gleaned from a few well-known services. Note: the specifics with notes are in tabs in the spreadsheet. We welcome corrections if we missed something.

Company	Total Score	Maturity Level	Notes
Intuit	18	Reasonable	Disappointingly, Intuit only just barely cleared the "reasonable" line.
Salesforce	40	Exceptional	Salesforce crushes the maturity test.
GitHub	35	Exceptional	Github could do more, particularly with data protection, but is otherwise a leader.

Now you try. And let us know what you think.

👉 Free SaaS Security Evaluation Spreadsheet

Application-Layer Encryption Defined

Patrick Walsh — Mon, 27 Sep 2021 17:27:03 +0000

Let's Not Overcomplicate It

Definition: Application-layer Encryption (ALE) is when you encrypt data before sending it to a data store. There are many variations, but at its core, that's it.

Variations

There are three main ways in which implementations may differ:

Code vs. Service: In some implementations, the encryption/decryption truly happens inside the application, which calls encrypt() or decrypt() routines. In others, an intermediate service or proxy essentially does that for you.
Client vs. Server: Some systems are strict about only encrypting and decrypting data in the client, which brings zero-trust data systems using end-to-end encryption (E2EE). E2EE is a subset of application-layer encryption. But encryption/decryption may alternately happen on the server. For server-side application-layer encryption and SaaS, the most secure option is customer-held keys where each customer keeps their master KEKs in their own KMS. We've also seen hybrid models where some data is end-to-end encrypted and other data is server-side encrypted.
Key Management: This is where systems differ the most. Ideally, each piece of data gets its own key, called a document encryption key (DEK), and that DEK is wrapped by a master key or key encryption key (KEK). This is called envelope encryption. When done right, the KEK is held in a Key Management Server (KMS) where it can't be exported and is backed by a Hardware Security Module (HSM). There can be multiple different KEKs at a per-row, per-field, or per-tenant basis, and the KEKs should rotate regularly so the number of KEKs increases over time. But while that's a strong application-layer encryption approach, plenty of implementations reuse keys to encrypt everything the same way and store keys in insecure places.

Don't Fall for the B.S.

There's a pattern repeated by security solution vendors: as soon as some term starts catching on, the whole industry puts it on their website and says that's what they do. They expand and stretch the definition and muddy the waters until the original meaning is essentially lost. It can be frustrating.

For example, the term "zero-trust" is now prominent industry-wide, but few deserve the label. If you're curious, we break down trust definitions in our trust models blog.

"Homomorphic encryption" is another example. It has a very specific definition that's based on a mathematical abstraction called a homomorphism. But security vendors have stretched the definition to mean pretty much anything that involves the use of encrypted data. We used to get excited when we saw someone claiming homomorphic encryption. Now we mostly roll our eyes and move on. 🙄

The latest casualty is "application-layer encryption", aka application-level encryption, aka ALE. Which is what spurred us to write this blog.

But this is an easy one to check.

A Simple Test

Next time someone says they're using application-layer encryption, apply this simple test: if you had (possibly stolen) credentials and network access that let you query a data store, and you use that to fetch data, is any of that data encrypted when you get it back? It should be.

An alternate test: if your infrastructure provider gets a law enforcement warrant for your data, and they make the query, will they get back meaningful data? Or will it be garbled?

More concretely, if you've turned on encryption for your databases, key-value stores, or object storage in AWS, GCP, or Azure, that's great, but that isn't application-layer encryption. It doesn't matter if the KMS only accepts requests from certain applications. If the data is transparently encrypted at the storage level, it's not application-layer encryption.

Why It Matters

We've recently seen flaws in Azure that caused large numbers of Cosmos DB instances to be publicly accessible. Those impacted companies would have had much less to fear if their exposed data was non-transparently encrypted.

Another reason it matters is that it's surprisingly easy to accidentally expose data in complex cloud environments. One in five data breaches is due to cloud misconfigurations (per Ponemon). The classic example is when much of the Fortune 500 was found to have open AWS buckets. These issues would have been mitigated by non-transparent encryption.

As a final example, we have the latest T-mobile breach. A hacker got a foothold inside their network and then ferreted out their saved database credentials. Querying the database gave back unencrypted data including social security numbers.

Remember, transparent encryption is useful, but it mostly protects against stolen hard drive attacks and it doesn't help at all in these scenarios.

Application-layer encryption brings an important layer of protection that means extra hurdles and challenges for attackers. It completely foils many common attacks.

From Here to There

Application-layer encryption isn't a silver bullet, but it's critically important. And while the definition is simple, implementing application-layer encryption can be complicated and full of pitfalls. We've seen application-layer encryption schemes where people put unwrapped encryption keys directly in the database. We've seen a single key get used and reused for all values. We've seen designs where keys can never rotate. And that's the tip of the iceberg.

At IronCore, we've worked hard to make it straight-forward for anyone to add application-layer encryption to their app, even if it's multi-tenant SaaS. We've made it developer-proof so no one can accidentally do the wrong thing. If you'd like to understand how we might help you accelerate and supercharge your data security, let's talk.

The Lack of Security Layers in SaaS

Patrick Walsh — Tue, 24 Aug 2021 21:11:11 +0000

Running applications using a cloud service should be far more secure than running the same apps on your own servers. In theory, the cloud company will keep everything up-to-date, deal with upgrades and migrations, and understand deeply what security is needed.

But if the cloud app holds sensitive data, then it's a rich target. If breached, an attacker will likely get the data of many customers. One vulnerability in the code, the infrastructure configuration, or any unexpected behavior at the intersection of complex systems, and an attacker can get at that data. The resulting breach can be devastating for both the cloud app company and their customers.

This blog will look at the typical approach to securing cloud apps, the weaknesses inherent in it, and will consider what can be done to improve things.

The Typical SaaS Architecture

In most cases, a SaaS app will consist of one or more application-layer services that talk to one or more backend data stores. Each application-layer service enforces permissions to make sure that the user making a request only gets back the data that he or she is allowed to see. The data stores don't usually enforce this or even know who the end user is that is making the request.

Because developers make mistakes, and because of attacks like SQL injections, most SaaS companies will purchase a security solution that sits out in front of the application. Typically, that means a network-layer firewall plus a web application firewall (WAF). It's the responsibility of the latter to block common attacks targeting web applications.

After traffic is scanned, it travels onward to some kind of ingress controller or load balancer. This is responsible for routing traffic to the appropriate services, limiting direct access to some service endpoints, and helping the app scale by adding more instances of the services. (Note: sometimes the WAF comes after the load balancer.)

Finally, the individual services will typically talk to the data stores as needed. Here's what this looks like:

If you're an attacker looking at this, what you'll see is an outer shell with security rules that need to be evaded and a set of services each of which likely contains yet-to-be-discovered vulnerabilities. That's two layers of security to get through in order to get to the good stuff.

The Overreliance on WAFs

So that Web Application Firewall (WAF) carries a lot of the burden. But here's the thing: not only are WAF evasion and bypass techniques plentiful, but most don't require much skill on the part of the hacker.

Even worse, WAFs can do more harm than good. For example:

They reduce performance;
they produce false positives that break your app (causing you to shut off some of the protections that you may need);
they have their own vulnerabilities that can give an attacker a foothold in your network;
they promote an attitude in developers that makes security someone else's problem;
and they introduce added complexity that enables attacks like desyncs and rewrites.

Some security researchers, like those who wrote about the desync attacks linked above, believe WAFs are actively harmful. I think they still have value and I wouldn't run a service without one, but they do introduce some risks even as they mitigate others. The lesson here: if you lean too heavily on your WAF, you'll fall over.

But maybe the cloud app is well-designed, with flawless permissions enforcement, always validated inputs, and no vulnerabilities. And the libraries and other software that the app uses are equally bulletproof. And the continuous stream of changes to the stack don't adversely affect its security. And pigs might fly.

The Swiss Cheese Model

There's a concept in risk analysis that helps to visualize layers of safety and risk reduction. The idea is that any single measure will inherently be imperfect, but layered together, a risk can be mitigated. For example, you might think of a seatbelt as a slice of cheese. An airbag is another. A crumple zone is a third. ABS brakes are a fourth. The goal of each of these measures is to prevent deaths in the case of a car accident. A death only occurs if the holes (the weaknesses) in each measure all line up and occur at the same time. This is called the swiss cheese model.

Swiss Cheese Model Graphic By BenAveling - Own work, CC BY-SA 4.0, Wikimedia Commons

Essentially this is a way to visualize one of the core pillars of security by design: Defense in Depth. So why, then, are there only two layers of meaningful security between an attacker and the data they seek to reach in the typical SaaS app? Why do most cloud app companies consider it "good enough" to transparently encrypt the data in its data stores, when that encryption does nothing to prevent unauthorized access to the data if an application is compromised?

It's time to add some more cheese to SaaS apps.

Application-layer Encryption (ALE)

One layer to add sits between the application and the data stores, and it's often called application-layer or application-level encryption (ALE). The key concept is that data is encrypted before being stored, which prevents someone with direct access to a database from being able to browse the encrypted data.

ALE can take many forms, but at its best, it adds another layer of permissioning and data segmentation to the data. That is, it can be used to virtually isolate sets of data so that an application breach does not immediately expose all the data. Instead, a layer remains that contains the blast radius from the application vulnerabilities. It will also prevent any access to sensitive data if the network is the exploited component or the hacker gets into the data store.

Using our earlier systems diagram, it might look something like this:

Segmenting Data by Service or by Tenant

To segment data, you use keys and restricted access to those keys to create the virtual isolation. For example, you could restrict Service 1 to only access the data in the database that it needs to do its job. Other data stored in the database or in the file store would be meaningless to it. And you can do this on a per-row or per-column basis.

In a multi-tenant SaaS world, you can use a different master key for each and every tenant (customer) and use an ALE solution that only allows the fetching of data for one tenant at a time.

This extra precaution is a game changer in terms of what happens when a breach occurs. And it's even more of a game changer if the SaaS app lets their customers hold their own master keys and monitor access of their data. The customers retain control of that data and can revoke access at any time.

It's Time For Action

It's time that we as a software industry stop hoping for magic vendor solutions that sit outside the application and intercept all hacking attempts. At best, they work partially and at worst, they're part of the problem. Applications need to add their own layers of security, and there's no more meaningful component to add than a data protection layer.

California's new privacy law explained

Patrick Walsh — Wed, 27 Mar 2019 23:40:40 +0000

The California Consumer Privacy Act takes effect on January 1st, 2020. But it has provisions that reach back to January 1st, 2019. If you’re a software developer or work for a software company, it’s reasonably likely that CCPA is going to impact your roadmap, your website, and existing or planned features in the near future.

This post originally appeared in Hacker Noon. Read the whole post there.