DEV Community: vidhi

HSTS (HTTP Strict Transport Security) - A buddy to HTTPS 🛡️

vidhi — Sat, 27 Feb 2021 12:51:19 +0000

The first time I came upon HSTS or whatever it was supposed to mean was when I was trying to be a developer like everyone else - improving our product's Lighthouse score. We then later concluded that Lighthouse only gives us the basic few requirements and we needed something more muscled up to check for other vulnerabilities along with basic performance. This is also how we ended up updating our third-party CSS and JS libraries but that is not the point we are discussing today.

We had a little warning that we did not have a specific HSTS header set. Well, I hadn't heard of it before (a point to note here would be that it is because of my lack of exposure to security headers in general). And then I did, again, what any other developer would - Google the hell out of it!

This was quite some time back, but I'm revisiting this to oil the information on it. I tend to forget stuff if I don't write it down. And also, who knows, it might help someone understand something! That's always a good thing to have.

🛡️ It all starts with HTTPS

Alright alright, I know the protagonist of this blog is HSTS, and not HTTPS. But it is impossible to talk about HSTS without tagging its buddy HTTPS along in the conversation. We will not go into the details of how it works, but instead of why it matters and its importance in the way the web works. If you're familiar with https, the next section might be your starting point.

There's http and there's https. Both are the same protocols with the difference of an 's' between them. Literally, and also conceptually. It is a what-you-see-is-what-you-get kinda thing if you want to be funny about it. The "s" is for security. HTTPS is a secure version of HTTP. Secure in what sense, you ask?

While on the internet, we transfer A LOT of data. We keep on sending requests, receiving responses for every information, for every page. And it would be much easier if the only data we were trading were cat pictures (which ultimately we'd want to share with as many people as possible), but it's not. We trade details like login details, bank information, and so much more. These are sensitive, something we don't want to share with people, but something that still has to flow through the wires and be communicated between you and the server. HTTPS is what helps us send this over securely.

With https, the communication between the client (browser) and the server is encrypted such that the information being sent over the wire cannot be intercepted by someone other than the server for which the information is intended to. If anyone tries to look at the encrypted data, it will seem like a string of random characters. But because of the way encryption works, the server has some additional knowledge that lets it decrypt the information and consume the actual information. This additional knowledge that the server has needs to be kept secret and private.

Because of what we discussed before, the use of https becomes especially important when dealing with websites that log you to your account. Since HTTP is stateless (ie, everytime you request a page, it knows nothing about you other than what you send in the current request), the server has to somehow recognize you and determine your identity from the information you send with every request. Which also means that with every request, you're sending some sensitive data over to the server which you do not want anyone else to get hold of. (if someone gets hold of what you send with each request to authenticate/identify yourself, the attacker could also easily use that in their request to pose as you).

🛡️ Present Day: HSTS

Let's continue HTTPS' story. We now know how important it is for websites to serve and receive content over https. But what if users access the site over http?

One way to get around it and enforce that communications take place over https would be to redirect any request on http to https. We can permanently redirect all http requests to https and it would work for us most times. But what could be the problems?

Redirection affects performance. We first request http which redirects us to https, after which our request is processed. Let's say we care more about security than performance, and we don't mind the redirection. Any problems then?
Yes, there still can be problems - security problems. Even if we set up a redirection, the initial request goes over http. And if you're logged in or you're trying to submit a form with sensitive data, it is still available for someone in the middle to intercept or attack it.

There is another way to enforce HTTPS, and that is (finally!) with HSTS.

HSTS (HTTP Strict Transport Security) is a security-enhancement achieved with the use of a response header (headers that are sent by the server in response to a client's request).

Suppose you're trying to access a website example.com, you can do it either via http or https. Let's assume it does it over http, so this initial request is not secure. Now, the server redirects to the https endpoint like we discussed previously.

At this stage, HSTS comes into picture as the response includes the HSTS header. This header instructs the browser to use HTTPS for every connection to the site and its subdomains (if includeSubDomains specified in the header) from now on. This rule is valid for a period of time which is specified n the response header as well.

What this means is that after receiving this header, the browser will absolutely not load any resource over http for the particular domain. If the browser is asked to load a resource over HTTP, it will try using HTTPS instead. In case HTTPS fails, the connection must be terminated.

There only needs to be one request to the domain, and with the help of redirection + HSTS, we can ensure that any more connections to the domain for that browser happen via https.

Note from the MDN Docs:

The Strict-Transport-Security header is ignored by the browser when your site is accessed using HTTP; this is because an attacker may intercept HTTP connections and inject the header or remove it. When your site is accessed over HTTPS with no certificate errors, the browser knows your site is HTTPS capable and will honor the Strict-Transport-Security header.

Is there a caveat here as well? Hmm, we'll see. ut first let's have the last piece of HSTS together, - how to set it up, and its components.

🛡️ HSTS! Dis-assemble!

The HSTS header can be set in a response like you'd set u any other header. For a more technical how-to, visit your preferred backend tech documentation. The syntax for the header is:

Strict-Transport-Security: max-age=<expiry_time>; [includeSubDomains] [preload]

In the above syntax, the directives enclosed in square brackets [] are optional.

max-age: This is what tells the browser the duration for which it has to remember to exclusively access a domain over https. This is specified in seconds and the maximum value is 2 years.

We usually specify the HSTS header for every response delivered over https. Thus, whenever the header is sent in the response, the browser will update the expiration time for that site, and this duration is refreshed which prevents it from timing out and expiring.

When the specified time elapses, the next attempt to request resource over http will proceed normally without automatically using https.
includeSubDomains: This is optional. If this parameter is included in the header value, the security enforcement will be applied to all the subdomains of the site.
preload: This is optional as well. This is an additional layer of protection that helps to tackle the issue that still remains after using the HSTS header. Adding this directive indicates that you want to be preloaded. We will discuss it in detail in the next section.

🛡️ HSTS Preload List

Let's talk about the caveat that remains. We now know that with the help of redirection and HSTS, the user only has to visit the site over https once before it expires to enforce communications over https. But it leaves open a small window of opportunity for attackers to sniff data and modify it if they want to.

Suppose you visit a site for the first time using http, and then the website redirects you to the https version and sends you the HSTS header in response. You are safe for any subsequent connections, but that first connection that you had made insecurely, is what might ultimately prove to be a point of failure.

HSTS preload is not part of the HSTS standard spec though. The HSTS preload list is a list maintained by the Chromium project which is used by most major browsers.

The browser has this internal preload list which is checked when making a connection. So suppose you have ever visited a website before but you have the domain added in the preload list, and you try to access the website over http for the initial request - it will not happen. The website then is never accessed via HTTP, and that includes the first connection which was the tiny window for an attack.

In short, if your domain is added to the HSTS preload list, most browsers will never connect to your domain without https.

🛡️ How to add to the HSTS Preload List

There are certain requirements that need to be fulfilled before you can request your domain to be added to the HSTS Preload List.

Your site must serve a valid certificate
If you are listening for HTTP requests, your site should redirect all requests to HTTPS
All the subdomains must be served over HTTPS.
For your base domain, serve the HSTS header with: 1) max-age of at least 1 year; 2) includeSubDomains directive; 3) preload directive.
Make sure the site continues to satisfy the above requirements at all times. Removing the preload directive will make your site available for the HSTS preload removal.

So after you have made sure that all your subdomains are served over HTTPS with a valid certificate, and that nothing breaks with the addition of HSTS headers (try it out with max-age of a few minutes first), add the preload directive to the header.

Then, head over to hstspreload.org and submit your domain. You can also find more cautions listed on the page and the step for removal if anything goes wrong, which is not a process they recommend. So try to be sure and test HSTS without the preload before you request to add a domain to the preload list.

This list is not immediately updated by the browser by downloading it over everyday or at intervals. It is a hard-coded list which is distributed with new browser versions, which means that it might take some time for your site to appear/remove in one of the browser's list.

🛡️ Tip for tiny rewinds

When testing HSTS with smaller max-age (and without the preload coming into the picture), you might occasionally want to remove the HTTPS-only restriction of your site. appuals has a very helpful section on how to remove a domain from HSTS cache in browser. I will brief it here ut you can find more information for more browsers in the given link. Note that this will not be effective if the domain is included in the preload list.

Chrome

Visit chrome://net-internals/#hsts from your browser. Enter the domain name in the "Delete domain security policies" section and hit Enter. Restart Chrome.

Firefox

One way to delete HSTS cache for a site would be to "Forget About this Site" but that would remove a lot more data than just the HSTS restriction. For more methods, visit the above link.

Original post available @ https://veedhee.com/blog/https-hsts-http-strict-transport-security

Random Python Tricks for Random Moods ✨

vidhi — Sat, 29 Feb 2020 20:18:13 +0000

If you have been using Python for quite a duration, you might already be familiar with all, or most of the tricks that we'll discuss below. But I guess it's always best to go over them briskly if it has been days you've used something in particular.

This will also include snippets, so you can just open your Python Interactive Shell and get started along with the examples.

✨ Let's start with something small and one-liner

One of the major concepts of code is to write as less code as possible and make the code as reusable as possible. Functions is one way, yes - the one example everybody gives when talking about reusability. Let's take it down one notch.
Say you want a list in Python of length 5 with all elements as 1. Writing it as lst = [1, 1, 1, 1, 1] would be the way to go, right? But isn't it... repeating and we're diverting from the reusability everybody talks about?
Good news, we can cut our efforts here.
We can write it as:
lst = [1]*5
This is followed for strings too. If you want, for example, to hide a phone number and show only the last 3 digits of the number to the user, you can do something like:

number = '1234567890'
hidden_number = "x"*7 + number[-3:]
'xxxxxxx890'

✨ Reverse a string/ list in a snap

If you're coming from C/C++, you would probably remember iterating from the last element explicitly and copying it over to a new variable to get the reverse of a string/array. With Python, you can:

a = [1, 2, 3, 4, 5]
a[::-1]
[5, 4, 3, 2, 1]

✨ Else is not only if's sibling but also for's sibling

else is something all the tutorial covers with the conditionals. It's the same old if this, then this; else this. But did you know that we can use else with our other favourite member of the family - for?
Here's how it goes.
Do you remember times when you would break out of the loop when you found something, or a condition satisfied, and you'd use a flag right before you broke out o the loop to let the code after the loop know that you hit a break specifically and it was not the normal end of the loop? The use of for..else eliminates the use o flags wth a simpler syntax.

a = [1, 3, 5, 7, 9]
b = [1, 3, 2, 5, 7, 9]

for i in a:
    if i % 2 == 0:
        print("list contains even elements")
        break
else:
    print("list has no even elements")

>>> list has no even elements

for i in b:
    if i % 2 == 0:
        print("list contains even elements")
        break
else:
    print("list has no even elements")

>>> list contains even elements

✨ Revelio

Suppose you have a Python object, it can be anything - functions/ modules/ dictionaries, and it can be created by you or fetched via an API or third-party code. Often times, we need to know the list of attributes and methods that are available on the object. Instead of looking at the code, we can simply list those out using dir.

class Example:
    name = "example"
    ex_id = 1
    def ex_print(self):
        print("example instance")

>>> ['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'ex_id', 'ex_print', 'name']

The above list is the list of attributes and methods of the object without the values. Notice the ex_id, ex_print, name in the list - this could come very handy while dealing with objects and to know what methods/attributes are available on them.

✨ Passing attributes to a function for the unknown

Let's take a function where you have a bunch of optional arguments and you need to call the function a couple of times, every time with a different set of arguments.

If you've worked with Django ORM, you'd know that filter() accepts a bunch of filter values. For the ones who are new to this:
In Django, databases are made easier with an Object Relation Mapper. This also makes querying the database easier for certain records. Suppose we have a table of users who can be a staff user or not, and belong to a certain department of the organization. Long story short, we want a function that fetches the records based on different filter combinations and calculates their ranking points for the organization's benefit plans.
One way to do this would be to fetch records separately for each filter combination we need and then calculate the ranking points, or the other would be to perform the filtering inside the function. We'll go with the latter option for this article. Let's take a look in more detail:

For filtering, the syntax goes

records = UserTable.objects.filter(is_staff=<value>, department=<value>)```
{% endraw %}


Instead of saying {% raw %}`department=None`{% endraw %} (which would search for None values) when not wanting o filter on the department, we could just not pass the department to the filter - it's optional! But we do need only one instruction that would handle all the combinations and cases. Maybe an example would help:
{% raw %}

```python
args1 = {'is_staff': True}
args2 = {'is_staff': True, 'department': 'Mgmt'}
args3 = {'department': 'Ops'}

Let these be the 3 filters you want to perform but you have a common function where you want to pass the filtering values:

def get_ranking(args):
   members = UserTable.objects.filter(**args)
   for member in  members:
        # do something
        pass

With this, we do not need to worry about calling get_ranking() with different sets of arguments. We can:

rankings = get_ranking(args1)
rankings = get_ranking(args2)
rankings = get_ranking(args3)

Here's what the function actually unpacks to:

members = UserTable.objects.filter(**args)
members = UserTable.objects.filter(is_staff=True)    # with args1
members = UserTable.objects.filter(is_staff=True, department='Mgmt')    # with args2
members = UserTable.objects.filter(department='Ops')    # with args3

That's all for this post, maybe I will come across some more exciting stuff (which I'm sure Python is full of) and continue this series for any and every mood that you might have. Let me know where and how you've used the tricks or plan to use them, so we can all bathe in the magic of Python.

BREAKING DOWN REGULAR EXPRESSIONS[0] — ONE STEP AT A TIME

vidhi — Sat, 20 Jul 2019 15:59:40 +0000

BREAKING DOWN REGULAR EXPRESSIONS — ONE STEP AT A TIME

For this article, I will be building this all out right from the ground up. You might be an experienced developer reading this article, a college student with some idea of what the name means, or someone who’s not a programmer but want to know what the term Regular Expression is. Anyone and everyone is welcome.

Before proceeding, I’d like to put something to light and that is that I cannot write regular expressions for a pattern at the top of my head. At times, I forget the most basic syntax of writing a particular element in a regex (the shorter form of regular expression). Which is why, this is more of a documentation for myself than for the public. I learn better when I read something that is written by me than looking up for distributed resources for a topic. And if I am going to write a detailed piece on regular expressions, why not write it like an article.

Feel free to add suggestions or provide feedback (constructive, please).

The Restaurant Scenario

Let us begin with a scenario first. You travel to a new city and you enter a very popular restaurant for lunch. So popular that they have 50 pages of menu. You take the menu in your hand and you just freeze. You have not visited the place before, you don’t know what the place serves, and you’re hungry. Very hungry.

Lucky for you, the staff at the restaurant is super helpful. That is how they became popular in the first place, you think. Instead of going through every item from the menu, which would practically be impossible, you just ask for some basic ingredients you want in your food.

You call for the waiter and ask them to bring you something that has both cheese and peas in it. Now because (a) the staff is super friendly — they accept your request which is not exactly an order (b) the staff is super talented and know their menu really well — so they return to you with a list of dishes that has both cheese and peas in it.

Great! This list is relatively smaller and you can now go through the menu and decide what you want and place the order.

If you understand the above scenario, you now have a basic idea of what regular expressions are. Let me draw the analogy.

The Parallel

When you first asked (queried) the waiter to bring you items that had cheese and peas, you gave them a combination of ingredients or a pattern you wanted (you could’ve asked for food that was barbecued or roasted, and they’d still come up with a list for you). The waiter now had two inputs. They had the very large menu they have been dealing with for years, and a combination you wanted.

The waiter matched your combination to find the menu you wanted.

In the world of software, this isn’t so different. Here, you give a sequence or a pattern of characters to a system and ask them to return all the *matching * results.

Let’s connect this to the example above. Suppose you are being served by a new waiter. Let’s call her Lucy. Lucy doesn’t have the entire menu memorized yet and is struggling to get you the complete list. She has other tables to attend and also cannot sit and match it for you physically. Her colleagues are busy attending tables too.

But she’s smart with computers.

Because she works for a rich restaurant, they have their menu digitally on a computer. She writes a quick regular expression to match all the dishes that contain both cheese and peas in their name, and voila, she now has the list ready for you!

And this is a real life use case of regular expressions. You take the combination/ pattern/ sequence of characters (“cheese”, “peas”), and you match it with the entire menu (“peas dipped in cheese”, “cheese honey & peas special”, etc).

Now we’ll extend the above example for another real life use case, just so we can be sure that we understand what’s going on. Lucy is going through the menu and being new to the crew, she suddenly notices something. While noting down “shredded cheese and peas with creem” , she notices that cream has been spelled wrong. Because she now has the power of regular expression, she enters a search for “cream” — the actual spelling, and finds no result. But she does remember learning a few dishes with cream in them. Where are they?

She then does a search for the dishes which have “creem” in them and there they are. All the dishes have cream spelled out the wrong way! Again, she’s good with computers. After completing her orders, she finds all the dishes with the misspelled word “creem” and replaces those with the correct “cream”. She later notifies the management about it and there’s a happy ending with bonuses which she later uses to buy herself a whipped cream machine.

The Definition

I love the resources of MDN, so let’s take a look on how they define regular expressions and see if we can understand the technical definition now that we know what it is and how it can be used.

Regular expressions are patterns used to match character combinations in strings.

And, that’s it. You now know what regular expressions are used for.

Wait a minute, stop scrolling, and lift your finger off the screen or mouse. Now take a moment to think of some real life incidents where you could’ve used regular expressions. Make it as absurd as possible — finding all the blue-colored jackets in a mall, getting carpets with either a black or blue border with flowers in the center.

The Where & Why Part of It

Now that you’ve given it a thought, we’ll end this part by taking a look at some very concrete scenarios where regular expressions are used extensively. This will help us get a more technical perspective towards this.

While filling out online forms have you ever come across constraints that go: “please input digits only”, I mean that’s a bummer, right? Well no, they are important and are achieved using regular expressions. The system checks if the value entered matches with a predefined string that goes from zero to nine [0–9]. We will check examples like these later in the series.
Did you forget where you have placed a file on your computer and the file was important? But you don’t usually panic, do you? You simply click the search option and type the few letters you are confident the file name contains. The system takes the input (the cheese and peas) and tries to find the relating items in your file storage system (the menu).
It is also useful in web scraping — when you want to get some information from a web page using code. You can enter what pattern you are looking for — for instance, a heading which contains the word “NEWS” in it. A very very basic use of regular expression in web scraping can be found here for a small and personal project.

This article starts everything from the very beginning and will probably be not useful to people with some experience. The next article in the sequence will be dropping soon, and it will focus on turning your patterns and text to actual code we can use. So, stick around for that if you’d like to.

To the ones starting out, just a note of caution, I’ve witnessed senior developers struggling with regular expressions as well. And it is completely fine to not grasp it in your first, second, third or even the seventieth attempt. The more and more you (we) use it in projects, the friendlier the concept becomes and honestly that should be the end goal. Not to master it, but to befriend it.

I forget the regex syntax almost for every other project that I use, but I know what to do and just have to verify and modify if the syntax works properly. I’ve a long way to go too, but it has been okay for any project that I’ve wanted to include regular expressions in.

You’ll be fine too.

— — — — — — — — —

Please leave a comment if this could help you in your journey of understanding regular expressions. If you’d like to chat on topics, you can find me here at twitter, or you can check out my other links listed at_ vidhibagadia.me.