DEV Community: Rui Figueiredo

How to get perfect intellisense in JavaScript

Rui Figueiredo — Tue, 20 Oct 2020 10:13:18 +0000

TypeScript is often described as the solution for making large scale JavaScript projects manageable. One of the arguments supporting this claim is that having type information helps catch a lot of mistakes that are easy to make and hard to spot.

Adopting TypeScript might not always be an option, either because you are dealing with an old codebase or even by choice.

Whatever the reason for sticking with plain JavaScript, it is possible to get a nearly identical development experience in terms of having intellisense and development time error highlighting. That is the topic of this blog post.

VS Code and JavaScript intellisense

If you create a new index.js in VS Code and type conso followed by Ctrl+space (or the Mac equivalent) you'll see something similar to this:

The source of the intellisense data is from the type definition files that that are bundled with VS Code, namely console is defined in [VS Code installation folder]/code/resources/app/extensions/node_modules/typescript/lib/lib.dom.d.ts. All the files with the .d.ts extension in that folder will contribute for what you see in the intellisense dropdown.

TypeScript definition files are one of the sources of intellisense in VS Code.

They are not the only source though. Another source is what VS Code infers from your code.

Here's an example of declaring a variable and assigning it a value. The intellisense is coherent with the type of that value:

(and yes, you can call .blink() or .bold() on a string, even in Node.js)

Here's another example where the type is inferred from the usage of a variable in a class definition:

And additionally to type inference, VS Code will add all the unique words on the file you are editing to the intellisense dropdown:

Even though the type inference available in VS Code is very clever, it's also very passive.

It won't warn you if you call myInstance.pethodName() instead of myInstance.methodName():

We usually only figure this out at runtime when we get a TypeError: myInstance.pethodA is not a function.

Turns out that VS Code has a flag that is turned off by default that when turned on will enable type checking to run through your code, and report errors:

The flag name is called checkJs and the easiest way to enable it is to open "Show all commands" (Ctrl+Shift+p) and type "Open workspace settings" and then activate checkJs:

You might discover that after turning on checkJs your file turns into a Christmas Tree of red squiggles. Some of these might be legitimate errors, but sometimes they might not. It doesn't happen often but I've encountered instances where the type definition files for a JavaScript library don't match the latest version (how this happens will become clearer later in the blog post).

If this happens and you are sure that the code you have is correct you can always add at the very top of the file:

//@ts-nocheck

This will turn off type checking for the whole file. If you just want to ignore a statement you add this immediately before the statement to be ignored:

//@ts-ignore
variableThatHoldsANumber = false; //this won't be reported as an error

Manually providing type information in JavaScript

There are situation where it is impossible for type inference to figure out the type information about a variable.

For example, if you call a REST endpoint and get a list of orders:

const orders = await getOrdersForClient(clientId);

There's not enough information available for any useful type inference there. The "shape" of what an order looks like depends on what the server that hosts the REST api sends to us.

We can, however, specify what an order looks like using JsDoc comments, and those will be picked up by VS Code and used to provide intellisense.

Here's how that could look like for the orders:

/** @type {Array<{id: string, quantity: number, unitPrice: number, description: string}>} */
const orders = await getOrdersForClient(clientId);

Here's how that looks like in VS Code when you access an order:

Even though this can look a little bit cumbersome it's almost as flexible having TypeScript type information. Also, you can add it just where you need it. I found that if I'm not familiar with a legacy codebase that has no documentation, adding this type of JsDoc annotations can be really helpful in the process of becoming familiar with the codebase.

Here are some examples of what you can do with JsDoc type annotations:

Define a type and use it multiple times

/**
* @typedef {object} MyType
* @property {string} aString
* @property {number} aNumber
* @property {Date} aDate
*/

/** @type {MyType} */
let foo;
/** @type {MyType} */
let bar;

If you use @typedef in a file that is a module (for VS Code to assume this there only needs to be an exports statement in the file) you can even import the type information from another file.

For example if @typedef is in a file named my-type.js and you type this from another-file.js in the same folder:

/** @type {import('./my_type').MyType} */
let baz;

The intellisense for the baz variable will be based on MyType's type information.

Function parameters and return values

Another scenario where type inference can't do much is regarding the parameter types in function definitions. For example:

function send(type, args, onResponse) {
    //...
}

There's not much that can be inferred here regarding the parameters type, args and onResponse. It's the same for the return value of the function.

Thankfully there's JsDoc constructs that we can use to describe all of those, here's how it would look like if type is a string, args can be anything and onResponse is an optional function function with two arguments, error and result and finally the return value is a Promise or nothing.

It's a pretty involved example, but it serves to illustrate that there's really no restrictions on the type information we can provide. Here's how that would look like:

/**
 * You can add a normal comment here and that will show up when calling the function
 * @param {string} type You can add extra info after the params
 * @param {any} args As you type each param you'll see the intellisense updated with this description
 * @param {(error: any, response: any) => void} [onResponse]
 * @returns {Promise<any> | void} You can add extra an description here after returns
 */
function send(type, args, onResponse) {
    //...
}

And here it is in action:

Class and inheritance

One thing that happens often is that you have to create a class that inherits from other classes. Sometimes these classes can even be templeted.

This is very common for example with React where it's useful to have intellisense for the props and state of a class component. Here's how we could do that for a component named ClickCounter whose state is a property named count which is a number and that also has a component prop named message of type string:

/** @extends {React.Component<{message: string}, {count: number}>}  */
export class ClickCounter extends React.Component {
    //this @param jsdoc statement is required if you want intellisense
    //in the ctor, to avoid repetition you can always define a @typedef
    //and reuse the type
    /** @param { {message: string} } props */
    constructor(props) {
        super(props);
        this.state = {
            count: 0,
        }
    }

    render() {
        return (
            <div onClick={_ => this.setState({ count: this.state.count + 1 })}>{this.props.message} - {this.state.count} </div>
        );
    }
}

This is how it looks like when you are using your component:

This also possible in function components, for example this function component would have the same intellisense on usage than the class component from the example above:

/**
* @param {object} props
* @param {string} props.message
*/
export function ClickCounter(props) {
    const [count, setCount] = useState(0);

    return (
        <div onClick={_ => setCount(count + 1)}>{props.message} - {count} </div>
    );
}

Casting

Sometimes you might want to force a variable to be of a particular type, for example imagine you have a variable that can be either a number or a string and you have this:

if (typeof numberOrString === 'string') {
    //there will be intellisense for substring
    const firstTwoLetters = /** @type {string} */ (numberOrString).substring(0, 2);
}

Use type information from other modules

Imagine you are writing code in Node.js and you have the following function:

function doSomethignWithAReadableStream(stream) {
    //...
}

To enable intellisense for the stream parameter as a readable stream we need the type information that is in the stream module. We have to use the import syntax like this:

/** @param {import('stream').Readable} stream */
function doSomethindWithAReadableStream(stream) {
    //...
}

There might be cases though where the module you want to import the type from isn't available out of the box (as stream is). In those cases you can install an npm package with just the type information from DefinitelyTyped. There's even a search tool for looking up the correct package with the typing information you need for a specific npm package.

For example, imagine you wanted typing information for mocha's options, you'd install the type definition package:

npm install @types/mocha --save-dev

And then you could reference them in JsDoc and get intellisense for the options:

Providing type information to consumers of your module/package

If you were to create a module that exposed functions and classes with the JsDoc type annotations that we've been looking at in this blog post, you'd get intellisense for them when that module is consumed from another module.

There's an alternative way of doing this though, with type definition files. Say you have this very simple module using CommonJS and this module is defined in a file named say-hello.js:

function sayHello(greeting) {
    console.log(greeting);
}

module.exports = {
    sayHello
}

If you create a file named say-hello.d.ts (and place it in the same folder as say-hello.js) with this inside:

export function sayHello(message: string): void;

And you import this function in another module, you'll get the the typing information defined in the .d.ts file.

In fact, this is the type of file that the TypeScript compiler generates (along with the .js files) when you compile with the --declaration flag.

As a small aside, say that you are creating an npm module written totally in JavaScript that you want to share. Also, you haven't included any JsDoc type annotations but you still want to provide intellisense.

You can create a type declaration file, usually named index.d.ts or main.d.ts and update your package.json with the types (or typings) property set to the path to that file:

{
    "name": "the-package-name",
    "author": "Rui",
    "version": "1.0.0",
    "main": "main.js",
    "types": "index.d.ts"
}

The type declarations that you put in index.d.ts define the intellisense you'll get when you consume the npm package.

The contents of index.d.ts don't even have to match the code in the module (in fact that's what the type definition packages in DefinitelyTyped do).

I'm intentionally leaving the topic of how to write typescript definition files very light here because it's a very dense topic and it's usually easy to find how to provide type information in most cases in the official docs.

A quick note about TypeScript definition files: a .d.ts file does not affect the file it "describes", i.e. if you create a type declaration file for module my-module.js and in that type declaration file you specify that functionA receives a parameter of type number and you invoke that function from functionB also inside my-module you won't get intellisense for functionA. Only modules that require/import my-module will take advantage of the type information in the type declaration file.

That's it, now think about that large 30+ property configuration object for which you can never remember the exact name of the property you want to set (is it includeArrayIndex or enableArrayIndex and does it take a boolean or a string?). Now you don't have to worry about mistyping it and you don't have to look it up everytime.

The story of how I created a way to port Windows Apps to Linux

Rui Figueiredo — Thu, 09 Apr 2020 22:19:57 +0000

Some day during a weekend sometime around the summer in 2018 I was doing house chores while listening to a podcast.

The podcast I was listening to is called Coder Radio, and I was specifically listening to episode #322 Not so QT.

That episode is about using QT to develop a cross-platform GUI for a .NET application. In the end they decided to give up on the idea, mainly because it was very complicated to setup, required it to be developed on Windows (QT does not support cross compilation) and in the end the license was prohibitively expensive.

When I heard this I though, humm, I think I know of a way to solve this problem. I think I can come up with a solution that would work well in this context, specifically for business applications where memory usage is not too constrained.

A bit presumptuous and naive of me to think like this? Perhaps, but let me take you through that journey. I promise it won't disappoint.

The idea

.NET does not have a solution for developing cross-platform GUIs. There are a few options, but they are not easy to set up and develop for.

On the other hand there's a technology that has been super popular for developing cross-platform apps which is Electron.

Electron has been heavily criticized because of its heavy memory use (mostly because of Slack), but there are great applications written in it that feel super smooth (VSCode) and are probably responsible for enabling people to be able to choose a different operating system than what they normally use.

The problem is, you can't develop using .NET in Electron, it's all JavaScript and Node.js (I know, I know, there's Electron.NET, but trust me, what I'm talking about here is completely different).

So the idea was, if Electron is basically Node.js and we can start a .NET process from Node why can't we use Electron to build the UI and have all the behavior written in .NET. We just need a (non-convoluted) way of sending commands/requests between Node and .NET and it all should work, right?

Turns out that yes, it works and you probably already use this approach all the time.

Any time you pipe the output of a command to another in the shell, you are basically using the same idea I'm going to describe next.

And if you are skeptical about how robust this is, let me tell you that people do database restores/backups using this technique (e.g.: cat backup.archive | mongorestore --archive).

Ok, no more beating around the bush: the idea is to use the stdin and stdout streams to create a two way communication channel between two processes, in this case between Node.js and .NET.

In case these streams are news to you, the stdin (standard input stream) is normally used to read data from the terminal (like when a program asks you for input) and the stdout (standard output stream) is where you write to in your program to get data to show up in the terminal. These can be redirected (piped) so that the output of one becomes the input of the other.

Node.js has a module named child_process that contains a function, spawn, that we can use to spawn new processes and grab hold of their stdin, stdout and stderr streams.

When using spawn to create a .NET process we have the ability to send data to it through its stdin and receive data from it from its stdout.

Here's how that looks like:

const spawnedProcess = spawn('pathToExecutable', [arg1, arg2]);
spawnedProcess.stdin.write('hello .NET from Node.js');
spawnedProcess.stdout.on('data', data => {
    //data from .NET;
});

Very simple idea, very few moving parts and very simple to set up.

Obviously, the code above in that form is not very usable. Here's an example of what I ended up creating:

const connection = new ConnectionBuilder()
        .connectTo('DotNetExecutable')
        .build();
connection.send('greeting', 'John', (err, theGreeting) => {
    console.log(theGreeting);
});

The code above sends a request to .NET of type "greeting" with argument "John" and expects a response from .NET with a proper greeting to John.

I'm omitting a lot of details here, namely what actually gets sent over the stdin/stdout streams but that's not terribly important here.

What I left out and is important is how this works in .NET.

In a .NET application it's possible to get access to its process' stdin and stdout streams. They are available through the Console's properties In and Out.

The only care that is required here is reading from the streams and keeping them open. Thankfully StreamReader supports this through an overload of its Read method.

Here's how all that ended up looking in the first implementation of this idea in .NET:

var connection = new ConnectionBuilder()
                    .WithLogging()
                    .Build();

// expects a request named "greeting" with a string argument and returns a string
connection.On<string, string>("greeting", name =>
{
    return $"Hello {name}!";
});

// wait for incoming requests
connection.Listen();

First experiments

I called the implementation of this idea ElectronCGI (which is probably not the best of names given that what this idea really enables is to execute .NET code from Node.js).

It allowed me to create these demo applications where the UI was built using Electron + Angular and/or plain JavaScript with all non-ui code running in .NET.

Calculator Demo:

PostgreSQL database records browser:

On that last one on every keystroke a query is being performed and the results returned and rendered. The perceived performance is so good that it totally feels like a native application, and all the non-UI code is .NET in both examples.

One thing that might not be obvious by looking at the examples is that you can maintain the state of your application in .NET.

One approach that is common with Electron apps is to use Electron to display a web
page, and the actions you perform end up being HTTP requests to the server that hosts that web page. That means you have to deal with all that is HTTP related (you need to pick an port, send http requests, deal with routing, cookies, etc etc).

With this approach however, because there's no server and the .NET process "sticks" around you can keep all your state there, and setup is super simple, literally two lines in Node.js and .NET and you you can have the processes "talking" to each other.

All in all, this gave me confidence that this idea was good and worth exploring further.

Pushing on, adding concurrency and two-way communication between the processes

At the time of these demos it was possible to send messages from Node.js to .NET, but not the other way around.

Also, everything was synchronous, meaning that if you sent two requests from Node.js and the first took one minute to finish, you'd have to wait that full minute before you got a response for the second request.

Because an image is worth more than a thousand words here's how that would look visually if you sent 200 requests from Node.js to .NET and where every request took an average of 200ms to complete:

Enabling request running concurrently involved dealing with concurrency. Concurrency is hard.

This took me a while to get right but in the end I used the .NET Task Parallel Library's Data Flow library.

It is a complicated subject and in the process of figuring it out I wrote these two blog posts, in case you are curious about DataFlow here they are: TPL Dataflow in .Net Core, in Depth – Part 1 and Part 2.

This is how much better the example above is when requests can be served concurrently:

The other big feature that was missing was to be able to send request from .NET to Node.js, previously all it was only possible to send a request from Node.js with an argument and get a response from .NET with some result.

For example:

connection.send('event.get', 'enceladus', events => {
    //events is a list of filtered events using the filter 'enceladus'
});

This was enough for simple applications but for more complex ones having the ability to have .NET send requests was super important.

To do this I had to change the format of the messages that were exchanged using the stdin and stdout streams.

Previously .NET's stdin stream would receive requests from Node, and responses to those requests were sent using its stdout stream.

To support duplex communication the messages included a type, which could be REQUEST of RESPONSE, and later on I added ERROR as well and also changed the API, in Node.js:

connection.send('requestType', 'optionalArgument', (err, optionalResponse) => {
    //err is the exception object if there's an exception in the .NET handler
});

//also added the ability to use promises:

try {
    const response = await connection.send('requestType', 'optionalArg');
}catch(err) {
    //handle err
}

//to handle request from .NET:

connection.on('requesType', optionalArgument => {
    //optionally return a response
});

And in .NET:

connection.On<T>("requestType", (T argument) => {

    //return optional response

});

//and to send:

connection.Send<T>("requestType", optionalArgument, (T optionalResponse) => {

    //use response

});

// there's also an async version:

var response = await connection.SendAsync("requestType", optionalArgument);

Proof: Porting a windows store application to Linux

When I first started with this idea I imagined a good proof that it would be viable would be to pick an application that was built using MVVM and be able to take the ViewModels, which are (should be) UI agnostic, and use them, unaltered, in an application using this approach.

Thankfully I had a game I built for the Windows Store around 2014 for which I still had the source code for. That game was named Memory Ace and you can still find it in the Windows Store here.

Turns out I was able to re-use all of the code to create the cross-platform version with no problems. Here it is running on Ubuntu:

I also was able to run it on Windows with no problems. I don't own a Mac so I could not try it there.

If you want to have a look at the source code, you can find it here. Also, the source for ElectronCGI is here for Node.js and here for .NET.

Also, here are some blog posts with extra information: ElectronCGI 1.0 – Cross-platform GUIs for .Net Core, ElectronCGI 1.0 – Cross-platform GUIs for .Net Core and ElectronCGI – Cross Platform .Net Core GUIs with Electron.

You can also see here how easy it is to setup a project with ElectronCGI (using an outdated version, but the process is identical).

So that's it. If I managed to grab your attention until now, can I kindly ask for your help?

I've been personally affected by the covid-19 pandemic. I was working as a contractor in a company that was badly affected (hospitality sector) and had to let everyone go. Me included.

I appreciate you might not be in a position to offer me a job, but any help is appreciated, for example if your company has open roles you can suggest me (I'm well versed in .NET. Node.js, React, Angular and several other technologies). Maybe there's even a referral program.

Or maybe you can add some endorsements on my LinkedIn profile.

Or if you know of any roles I could be a good fit for let me know, here's my twitter (my DMs are open).

Take care and stay safe.

Software Development as an Emergent System

Rui Figueiredo — Mon, 22 Oct 2018 21:57:15 +0000

Emergent systems are sometimes described as systems where the whole is greater than the sum of the parts.

I personally prefer to think of an emergent system as something that is complex to reason about as a whole but simple(r) to understand when you look at its individual components.

Probably one of the most given examples of an emergent system in nature is that of a flock of birds flying in perfect coordination.

If you try to model the birds' movement as a whole it becomes difficult and convoluted. The movement they make seems almost unpredictable.

But if instead you model this system by modeling one bird, then it becomes simple to understand.

This has been done exactly for this problem (modeling the behavior of a flock of birds) and the solution is called Boids.

In Boids, the complex behavior of a flock of birds is achieved by having each bird follow three simple rules.

The first one is separation, which states that each bird will try to keep a reasonable distance from other birds. The second rule is alignment which means that a bird will try to align itself with the birds in its vicinity. And finally, cohesion which means that each bird will try to move to the average position of other nearby birds.

It's impressive how lifelike a simulation looks using just these three simple rules. You can watch them in action in this video that does a very good job of explaining each rule in isolation.

But what does this have to do with software development?

Software is developed by individuals and any software system of reasonable size becomes hard to reason about as a whole.

Unfortunately it isn't possible to come up with simple rules that allow us to predict how a particular piece of software will evolve. However looking at software development from this perspective surfaces some aspects that might help determine if a software product will be a quality product, or if it will derail into something that no developer wants to work on.

One of the things that comes to mind are "initial conditions". In the Boids example this would be the birds' initial position in the simulation. Different starting positions might lead to one big flock or to multiple individual flocks.

In software there are several things that can be thought of as initial conditions, for example the type of technology used or even the project's initial structure.

I've worked on large projects where the default MVC structure (one folder for controllers, one folder for views and one folder for models) for a web application was blindly taken. Having a controller folder with hundreds of controllers quickly gets out of hand. At some point things become hard to name, to find, and a pain to work with.

An aspect that is implicit from the Boids' simulation is that every bird follows its rules perfectly. If they didn't the flock would break apart.

People can't be expected to follow rules perfectly, especially because there are no perfect rules in software development.

However, if we have some way to measure how things are going, like we can look at a flock and see where it's breaking apart, we can nudge things back to harmony.

But to do that we need to be able to measure the health of the system.

If you develop in .Net and you use Visual Studio you have access to a few Code Metrics, namely a Maintainability Index, Cyclomatic Complexity, Depth of Inheritance and Lines of Code.

Those can be useful (although the Maintainabilty Index may be a bit questionable). However, they are a bit limited and they can't be neither changed nor added to.

Recently I've discovered a tool that seems to have been made to solve this problem. It's called NDepend and it not only has these metrics and hundreds more, it also allows you do change them to your needs and even and add your own.

NDepend

NDepend is a paid extension for Visual Studio (with a trial period).

It analyzes your solution each time you build it and provides insights in the form of interactive graphs and a dashboard that shows passing/failing rules.

The rules in NDepend are similar to Linq queries (they are called CQLinq in NDepend where CQ stands for Code Query) and there are more than 200 available.

They are easy to read as well, for example this rule creates a warning for methods that have more than 10 lines of code:

// <Name>Keep methods short and concise</Name>
warn if count > 0 from method in Methods where method.NbLinesOfCode > 10 select method

This is what you'll see when there are methods that are picked up by this rule:

You can click any of the methods and NDepend will take you to where they are in the source code.

You can find the predefined and also your custom rules in NDepend's Rule Explorer Panel:

You've probably noticed that I've used select method in the rule definition. You can add additional information by returning an anonymous object with a method property and the extra information that we want to add. That extra information will be displayed in the UI.

For example, let's add the method's visibility and how many methods are called from the offending method:

// <Name>Keep methods short and concise</Name>
warn if count > 0 from method in Methods where method.NbLinesOfCode > 10 select new {
    method,
    method.Visibility,
    method.MethodsCalled
}

You can access this information when you click the rule in the Rule Explorer. You can even hover over each of the "returned" fields and get extra information:

Additionally to being able to customize how the rules are displayed you can define how they affect the project in terms of technical debt. Namely, through an arbitrary amount of time.

Let's say that we think that it would take on average 5 minutes to fix each line over 10 lines of code (by breaking the method in smaller methods, for example) in a method and we want that to be quantified.

We can include a Debt value in the rule's select statement that describes this:

// <Name>Keep methods short and concise</Name>
warnif count > 0 from method in Methods 
where method.NbLinesOfCode > 10 
select new { 
    method,
    method.Visibility,
    method.MethodsCalled,
    method.NbLinesOfCode,
    Debt = (5 * (method.NbLinesOfCode -10)).ToMinutes().ToDebt()
}

This will then add the the project's overall Debt that you can consult in NDepend's dashboard.

This is just a sample of what you can do with rules. I've only mentioned methods, but you can query assemblies, types, fields, code created by you (you can fine tune what this means in NDepend), third party code, etc. You can also cherry pick which rules you want considered in case you don't agree with some of them.

Also, you can integrate NDepend with a build server and have the build fail when you detect something you consider serious going on. This is achieved with a set of rules named Quality Gates that you can also customize and add/remove to.

Additionally to the rules you can produce graphs that provide visual insights over the code base.

For example, you can create a "Code Metric View" graph that can show you two dimensions simultaneously. For instance the number of lines of code (LoC) in a class and its cyclomatic complexity. The LoC is represented by how big an area the class takes and the complexity by the color:

You can see from the example above that the ManageController is the "worst" class in terms of cyclomatic complexity and it's the second largest in terms of lines of code.

NDepend has other graph types that you can use to better understand your project's dependencies, inheritance hierarchies and test code coverage just to name a few.

However, all of this only allows you to see one picture of how your project is doing in terms of quality right now.

My favorite feature of NDepend is that you can create a baseline and then see how the project evolves in relation to that baseline (you can configure this from the Dashboard).

For example, if you add or remove Debt or have more or less rules passing/failing this is highlighted and you can truly have a feeling of seeing the direction the project is going.

Conclusion

Software development is a complicated discipline, especially when you consider that it is performed by several people working together.

Comparing it to emergent systems is useful because it provides a perspective where we can think of software as something that evolves.

Being able to measure that evolution is then crucial if we want to be able to tell if the product we are building is holding up in terms of quality.

I also describe a tool named NDepend that serves exactly this purpose (and as far as I know has no competitors). It provides extensive metrics and allows for the creation of custom rules, all of this while supporting integration with a continuous integration workflow.

Everyone is watching what you do online. How user tracking with cookies works

Rui Figueiredo — Mon, 16 Jul 2018 18:20:18 +0000

Have you ever visited a website to check out something you want to buy only to be inundated with ads for that product in other websites?

I've been asked about this enough times that I thought it would be a good exercise to try and explain how this works.

This blog post is an explanation of how websites that seem totally unrelated to each other seem to know about what you are doing online.

It all starts with the Cookie

Before we describe what a cookie is (in the context of the web), let's start by describing why they are needed.

The web is mostly a disconnected system. That's an odd thing to say when the web is almost synonymous with being connected. What disconnected means in this context is that when you use your browser to open visit website, the browser connects to the server that hosts that website, gets the web page and disconnects. The web server only knows about you for that very brief period.

This is what allows one website to potentially serve millions of requests. It doesn't need to hold information about you in memory for very long.

Well, if that's the case how come we have websites that do "remember" us, where we can login and see data that is ours?

That's where cookies come in. A cookie is information stored in your computer about the website you visited.

When you visit a website your browser sends a HTTP request to a web server that is listening at the address of that website. The web server sends back an HTTP response with the contents of the website (HTML) and some "headers". A header is just a key value pair (for example: Server: TheServerName).

There's a header named Set-Cookie which when present in an HTTP response will make your browser create a cookie for the website you are visiting. It's actually a little bit more complex than this. The cookie is tied to the domain of the website, but for this discussion it's ok if you think that a cookie belongs to one website.

Also, a cookie is just a text file saved on your computer.

Here's an example of a Set-Cookie header from a response to myawesomewebsite.com:

Set-Cookie: the_cookie_name=the_cookie_value; path=/; expires=Tue, 06 Jun 2028 21:50:30 GMT; domain=.myawesomewebsite.com

After this response, every time you visit myawesomewebsite.com until 06 June 2028 your browser will send a Cookie header in the request like this:

Cookie: the_cookie_name=the_cookie_value

The easiest example to imagine of a website using this is a website that allows its users to pick a background color. In the response to the request to change the background color the web server will add a header like this: Set-Cookie: background_color=blue ....

Now every time you go to that website your browser will "tell" the website that you want the background to be blue, and the way it will do this is by including the Cookie header in the request: Cookie: background_color=blue.

The mechanism that allows you to log in to a website is very similar to this. When you visit a login page and enter the correct username and password the web server will send back a response with a Set-Cookie header whose contents will allow the web server to find your data. The value of this cookie is usually encrypted so that only the server can create it, and also only the server will be able to read it. This way a user can't just change the value and potentially sign in as another person.

Tracking users with cookies

I'm sure you've seen messages similar to this: "We use cookies to improve your experience on our site and to show you relevant advertising".

The way that cookies can be used to "show you relevant advertising" hinges on how they are treated by the browser. Every time your browser makes a request to a website for which it has a cookie, it will send that cookie.

Also, when the request originates form a website that is different from the website the cookie is for, the browser will add a header named Referer (yes, it's misspelled, but that's the way it is in the spec so we have to live with it). That header will contain the URL from the website from where the request originated.

This is simpler to understand with an example. Say you go to Wikipedia to read about the World Cup. While doing that you find something interesting in the references section that you want to explore. For example a page about the 2018 qualification that is hosted in another website that you have visited before (e.g. sofascore.com).

When you click on the link to sofascore.com your browser will add the Cookie header to the request for sofascore.com and it will also add a Referer header with the url from Wikipedia, for example: Referer: https://en.wikipedia.org/wiki/2018_FIFA_World_Cup.

How can this be exploited?

For simplicity imagine that the original website you've visited is named tracker. Now imagine tracker set a cookie for you with a value that is unique (will act as an identifier) and has a very long expiration date. Also, tracker has an agreement with loads of other websites, and those websites host a 1x1 pixel transparent image from tracker.

Each time you visit one of those websites your browser will make a request to tracker for the transparent image and send the cookie with your identifier and with the referer header. This way, not only does the tracker know who you are, it also knows which websites you are visiting.

The tracker can then sell your info to other websites that then can choose what they want to do with it. Mainly show you ads related to things they think you are interested in. It's as simple as that.

Also, this technique is known as web beacon and it's one of the most simple techniques. There are much more elaborate versions that will use a combination of features to "tag" you even if you delete your browser cookies.

In case you are wondering why a cookie is called a cookie maybe it's because it's like a fortune cookie? It has a message inside. I actually don't know, but if you do please leave a comment.

ORM-less Data Access in .Net Core

Rui Figueiredo — Fri, 01 Jun 2018 15:58:10 +0000

The use of Object Relational Mapper libraries (ORMs) is so prevalent today that it's uncommon to see anyone question their use.

There are good reasons for that. In the old days you'd see SQL code sprinkled everywhere. It was common to find examples where user input was concatenated directly with SQL statements opening the doors to SQL injection attacks (little Bobby Tables comes to mind).

Even though a lot of good came out of using ORMs, there's some less good things that came with it too. The first is performance, which is worse (sometimes much worse).

But apart from performance there are a set of other issues that although they are not disadvantages, they have a negative impact on the experience of using an ORM. They are all related to the fact that ORMs hide a lot of details about how the data is retrieved and saved. Frequently, people by not being aware of these details shoot themselves in the foot.

Just a few examples are the use of lazy loading even when that is arguably not a good idea (e.g. web applications) or N+1 problems stemming from the mechanism that triggers data being fetched (thinking specifically of Entity Framework here) not being obvious sometimes.

I'm not advocating that ORMs shouldn't be used, not at all. However, my perception is that nowadays most people working in the .Net ecosystem wouldn't be able to retrieve and create records in a database without using Entity Framework.

And that's unfortunate because it's not hard at all to go raw, and it might be quicker than to have to setup Entity Framework. I've been following that approach in small projects and I'm convinced that I can put something up faster without Entity Framework than with it (setting up EF can be a pain).

What I want to do in this blog post is show you how you can use the "raw" data access mechanisms (ADO.NET) available in .Net Core and also an alternative to Entity Framework named Dapper (which is used by Stack Overflow). Dapper is sometimes described as an ORM, but as we'll see it's more of an "object mapper".

CRUD with ADO.NET in .NET Core

To do data access without Entity Framework Core you need to master just three concepts. Those concepts are Connections, Commands and Data Readers.

A Connection object represents a connection to a database. You'll have to use the specific connection object for the database you want to interact with. For example for PostgreSQL we'd use NpgsqlConnection, for MySql MysqlConnection, for SQL Server SqlConnection. You get the idea. All these connection types implement the interface IDbConnection.

You need to install the right Nuget package for the database you want to use, for example for Postgres the package name is simply: Npgsql (An easy way to remember it, is N - for .Net and pgsql for PostGreSQL).

To create a connection object we need a connection string to the database we want to interact with. For example for a database named "example" with user "johnDoe" in Postgres we could create a connection this way:

var connection = new NpgsqlConnection("User ID=johnDoe;Password=thePassword;Host=localhost;Database=example;Port=5432");

A great resource to find information about how to create these connection strings is https://www.connectionstrings.com/.

After creating the connection object, to actually connect to the database we need to call Open on it:

connection.Open();

The connection objects are all IDisposable, so you should dispose of them. Because of this, usually the connection is created inside a using block:

using (var connection = new NpgsqlConnection("User ID=johnDoe;Password=thePassword;Host=localhost;Database=example;Port=5432"))
{
    connection.Open();
    //use the connection here
}

That's all you need to start.

Creating records

To create records we need to use a Command. A command is a container for all that is required to perform an operation in the database.

The easiest way to create a command is to ask the connection object for one:

using(var command = connection.CreateCommand()) 
{
    //use command here
}

You then specify the SQL you want to execute through the property CommandText and the values for the parameters in the SQL in the property Parameters. For example, if you want to add a record to a table named people with columns first_name, last_name, age it would look like this:

command.CommandText = "insert into people (first_name, last_name, age) values (@firstName, @lastName, @age)";
command.Parameters.AddWithValue("@firstName", "John");
command.Parameters.AddWithValue("@lastName", "Doe");
command.Parameters.AddWithValue("@age", 38);

You should use parameters because that prevents SQL injection attacks. If you don't and you create your SQL by using string-concatenation using data that was entered by the user you enable situations where a user can type something that will be interpreted as SQL.

The way a command is "executed" depends on the result you expect from it. For adding a record, and in case you don't care about any auto-generated column values (for example the new record's id) you can do this:

int numberOfUpdatedRows = command.ExecuteNonQuery();

This method returns the number of rows that were updated/created. Although it's not particularly useful on an insert statement, on an update that value might be useful.

Alternatively, if you want to insert and get the new record's id you can change the insert statement's SQL so that the new id is returned. The way you do this depends on which database you are using, for example in postgres the SQL would look like this insert into people (first_name, last_name, age) values (@firstName, @lastName, @age) returning id.

In sql server it would look like this insert into people (first_name, last_name, age) values (@firstName, @lastName, @age); select scope_identity().

To get the run the insert and get the new id you can use the ExecuteScalar method in the Command.

The ExecuteScalar method executes the SQL and returns the value (as type object) of the first column in the first row, for example in postgres:

command.CommandText = "insert into people (first_name, last_name, age) values (@firstName, @lastName, @age) returning id";
command.Parameters.AddWithValue("@firstName", "Jane");
command.Parameters.AddWithValue("@lastName", "Doe");
command.Parameters.AddWithValue("@age", 37);

var newId = (int)command.ExecuteScalar();

You might be thinking right now that these SQL statements involve a lot of typing. And on top of that, all of that is with no intellisense. Thankfully there are techniques around that. You can watch me creating an insert statement from scratch without actually having to manually type any column names.

Reads

To read data you simply need to write your SQL query and call the ExecuteReader method in the command object. That will return an instance of a DataReader which you can then use to retrieve the actual results of your query.

For example, if we want to retrieve all records in the people table:

command.CommandText = "select * from people";
DataReader reader = command.ExecuteReader();

Now, the way you get to the actual values is a little bit clunky. Definitely not as comfortable to do as with Entity Framework, but as I showed in the video on how to use Sublime to build the queries, you can also use the same techniques to create this code faster.

Here's how you could iterate over all results assuming that first_name and last_name are strings and age is an int.

command.CommandText = "select * from people";
using(DataReader reader = command.ExecuteReader()) 
{
    while(reader.Read())
    {
        string firstName = reader.GetString(reader.GetOrdinal("first_name"));
        string lastName = reader.GetString(reader.GetOrdinal("last_name"));
        int age = reader.GetInt32(reader.GetOrdinal("age"));

        //do something with firstName, lastName and age

    }
}

The GetString, GetIn32, GetBoolean, etc, methods expect a number that represents the column index. You can get that column index by calling reader.GetOrdinal("columnName").

Updates and Deletes

Updating and deleting records involves creating a command with the right SQL, and calling the ExecuteNonQuery in that command.

For example if we wanted to update all records on the people table that have the surname "Doe" to "Smith" we could do this

command.CommandText = "update people set last_name='Smith' where last_name='Doe'";
int numberOfAffectedRows = command.ExecuteNonQuery();

Deletions are very similar, for example, let's delete all records which have no last_name

command.CommandText = "delete from people where last_name is null";
int numberOfAffectedRows = command.ExecuteNonQuery();

Transactions

One thing that you get for free when using an ORM like Entity Framework is that when you persist your changes (i.e. call.SaveChanges()) that happens inside a transaction so that all the changes are persisted or none is.

Thankfully creating a transaction using ADO.NET is very simple. Here's an example where we add a new person, delete another and update another yet all inside a db transaction:

using (var transaction = connection.BeginTransaction())
{
    var insertCommand = connection.CreateCommand();
    insertCommand.CommandText = "insert into people (first_name) values (@first_name)";
    insertCommand.Parameters.AddWithValue("@first_name", "Jane Smith");
    insertCommand.ExecuteNonQuery();

    var deleteCommand = connection.CreateCommand();
    deleteCommand.CommandText = "delete from people where last_name is null";
    deleteCommand.ExecuteNonQuery();

    var updateCommand = connection.CreateCommand();
    updateCommand.CommandText = "update people set first_name='X' where first_name='Y'";
    updateCommand.ExecuteNonQuery();

    transaction.Commit();
}

The easiest way to create a transaction is to request one from the connection object. A transaction is created using an IsolationLevel. Although we didn't specify one here (the particular database's default will be used) you should check the list of available isolation levels and choose the one that is appropriate to your needs.

After having the transaction created we can do all operations as before and in the end we call .Commit() on the transaction. If anything goes wrong before .Commit() is called all the changes are rolled back.

Getting metadata from the database

This is an aside but it's something that is useful to know. Using ADO.NET it is possible to extract metadata about your database. For example all the column names and their types form a particular table.

The following snippet shows how you can get all the column names and data types form all the columns in a particular database table:

command.CommandText = "select * from people";
using(var reader = command.ExecuteReader()) 
{
    var columnSchema = reader.GetColumnSchema();
    foreach(var column in columnSchema)
    {
        Console.WriteLine($"{column.ColumnName} {column.DataTypeName}");
    }
}

Using Dapper

Alternatively to using just ADO.NET using Dapper is just as easy and with small differences in terms of performance.

In case you are unfamiliar with Dapper, it's a project from StackExchange and it powers the StackExchange familiy of websites (StackOverflow, SuperUser, AskUbuntu, etc).

To use Dapper you need to install a Nuget package conveniently named Dapper along with the specific Nuget package for the database you are targeting. For example for Postgres:

$ dotnet add package Npgsql
$ dotnet add package Dapper

Dapper adds a few extension methods to your connection object, namely Query<T> and Execute.

Query<T> allows you to run a query and map the results to the type you specify in the generic parameter. For example, this is how you can get all records in the people table:

using Dapper; //you need this to get the extension methods on the connection object
//...

using (var connection = new NpgsqlConnection("theConnectionString"))
{
    IEnumerable<Person> people = connection.Query<Person>("select * from people");                        
}

The Query<T> method always returns an IEnumerable<T> even if you only expect one record. For example you could do this to insert a new person and get the new Id:

int newId = connection.Query<int>("insert into people (first_name, last_name, age) values (@FirstName, @LastName, @Age) returning id", new {
    FirstName = "John",
    LastName = "Doe",
    Age = "40"
}).FirstOrDefault();

In the example above we are providing 2 parameters to the Query method, the first is the SQL statement and the second one is an anonymous object with property names that match the parameters in the SQL statement. You can also use an instance of a class (e.g. new Person { ... }) instead.

If you don't care about any results, for example you just want to delete a record, you can use the Execute method instead which will return the number of records affected in the database. For example if you want to delete all the records for which last_name is null:

var numberOfDeletedRecords = connection.Execute("delete from person where last_name is null");

This was just a gentle introduction to Dapper, the github page for the project is a good resource if you are interested in learning more.

I hope this blog post has given you enough information about how to go ORMless in .Net Core. Let me know your thoughts in the comments.

ASP.NET Core Development in Linux

Rui Figueiredo — Wed, 28 Mar 2018 13:45:04 +0000

When .Net Core came out there was a lot of excitement about the idea of developing .Net code outside of Windows.

The IDE of choice for .Net, Visual Studio, is and probably will always be, a Windows-only program.

There are alternatives though, and today it is possible to fully develop an ASP.NET Core application outside of Windows.

In this blog post we will be looking into that, specifically in Linux.

Please note that all of the tooling described here is cross-platform all of this will work on any platform that supports .Net Core.

The IDE

The full version of Visual Studio is only available in Windows. However, there's an alternative that has a somewhat confusing name: Visual Studio Code.

You might think that VS Code is an inferior version of the full Visual Studio product, but that is not the case.

VS Code is a different project altogether. It's built using a different set of technologies and this brings some features that full VS doesn't have, for example being easily extensible and customizable, and also less demanding in terms of resources.

VS Code offers the ability to extend its functionality by the installation of extensions. For C# you should install the C# extension. You don't need to go through the trouble of looking for it though. When you open a project that has C#, Visual Studio Code will prompt you to install the C# extension if you don't have it already.

In the rest of the blog post we will create a hypothetical ASP.NET Core web application in order to illustrate most of the required tasks when developing in .Net. Namely how to create projects and solutions, reference projects, install nuget packages, use dotnet tools (like entity framework) and run tests.

A Typical Project

The imaginary project we'll be creating is a reminder application. The idea here is to describe a project with sufficient complexity that would require multiple projects that reference each other, a solution file and tests.

The solution will contain an ASP.NET Core project that could, for example, be used to see past and future reminders. A Console application so that we can create reminders from the command line (and to illustrate how we can have a solution with more than one running project) and a class library that will contain all the common logic. We'll also add a test project to illustrate how unit tests can be run from the command line and inside VS Code.

Creating the projects and solution file

By this time you should have installed .Net Core and Visual Studio Code. You should be able to open a terminal window and type

$ dotnet

and get an output something similar to this:

Usage: dotnet [options]
Usage: dotnet [path-to-application]

Options:
-h|--help            Display help.
--version         Display version.

path-to-application:
The path to an application .dll file to execute.

One of the options of the dotnet command is "new". To check that and other options' help page just do dotnet [option] --help, in this case dotnet new --help.

That will display a list of possible project templates to choose from. For example to create an ASP.NET Core MVC project:

$ dotnet new mvc

If you run that as it is, a new MVC project will be created in the current folder, using the name of the current folder as the default namespace and project name.

You might not want that. In case you don't, you can use the --name and --output options. Using --name (or -n) you can specify the default namespace and with --output (or -o) you can specify the folder name to create and where to place the project files. If you use --name and don't specify --output, the output folder will implicitly take the same value as --name.

It is probably a good idea to define the project folder structure before we continue. Here it is:

   Reminders
   +---+ Reminders.Web
   +---+ Reminders.Cli
   +---+ Reminders.Common
   +---+ Reminders.Tests

First thing we need to do is to create a project folder named Reminders and then inside it do:

$ dotnet new mvc --name Reminders.Web
$ dotnet new console --name Reminders.Cli
$ dotnet new classlib --name Reminders.Common
$ dotnet new xunit --name Reminders.Tests

This will create 4 projects, one MVC, one console, a class library and a xUnit test project.

It's a good idea to create a solution file as well. Especially if eventually you want to open these projects in full Visual Studio, which will prompt you to create one until you eventually do it.

There are advantages in having a solution file other than avoiding full Visual Studio nagging you about creating the .sln file. If you have one you can just go to the Reminders folder and do a dotnet build or a dotnet restore and that build/restore all projects referenced in the .sln file.

Also, if you have a .sln file you can open all projects in VS Code by initiating VS Code in the folder where the .sln file is located.

Let's create our .sln file, name it Reminders.sln and add the four projects to it (inside the Reminders folder):

$ dotnet new sln --name Reminders
$ dotnet sln add Reminders.Web/Reminders.Web.csproj
$ dotnet sln add Reminders.Cli/Reminders.Cli.csproj
$ dotnet sln add Reminders.Common/Reminders.Common.csproj
$ dotnet sln add Reminders.Tests/Reminders.Tests.csproj

Adding project references

In full Visual Studio when you want to add a reference to a project you can just right click on references and select Add Reference, pick the project you want to add and you're done.

VS Code does not have that functionality. To do this we need to resort to the command line, but it's just as easy. For example, let's reference Reminders.Common in Reminders.Web, Reminders.Cli and Reminders.Tests.

Navigate to the folder where the .sln file is (Reminders) and type:

$ dotnet add Reminders.Web/Reminders.Web.csproj reference Reminders.Common/Reminders.Common.csproj
$ dotnet add Reminders.Cli/Reminders.Cli.csproj reference Reminders.Common/Reminders.Common.csproj
$ dotnet add Reminders.Tests/Reminders.Tests.csproj reference Reminders.Common/Reminders.Common.csproj

If you want to have a look at which other projects are referenced by a particular project you can either open the .csproj file and have a look at the ProjectReference entries or if you want to do it from the command line: dotnet list PathToCsProj reference.

You can also navigate to the folder where a particular project is and simply do dotnet add reference pathToOtherProject.csprj.

Picking the startup project

Open the solution in VS Code. The easiest way to do that is to navigate to the Reminders folder (where the .sln file is) and type code ..

When you do that you should get a message in VS Code: "Required assets to build and debug are missing from 'Reminders'. Add them?"

Click Yes.

That will create a .vscode folder.

In that folder there are two files: launch.json and tasks.json. launch.json configures what happens when you press F5 inside VS Code.

For me the project that will run is the Reminders.Cli console project:

{
// Use IntelliSense to find out which attributes exist for C# debugging
// Use hover for the description of the existing attributes
// For further information visit https://github.com/OmniSharp/omnisharp-vscode/blob/master/debugger-launchjson.md
"version": "0.2.0",
"configurations": [
        {
            "name": ".NET Core Launch (console)",
            "type": "coreclr",
            "request": "launch",
            "preLaunchTask": "build",
            // If you have changed target frameworks, make sure to update the program path.
            "program": "${workspaceFolder}/Reminders.Cli/bin/Debug/netcoreapp2.0/Reminders.Cli.dll",
            "args": [],
            "cwd": "${workspaceFolder}/Reminders.Cli",
            // For more information about the 'console' field, see https://github.com/OmniSharp/omnisharp-vscode/blob/master/debugger-launchjson.md#console-terminal-window
            "console": "internalConsole",
            "stopAtEntry": false,
            "internalConsoleOptions": "openOnSessionStart"
        },
        {
            "name": ".NET Core Attach",
            "type": "coreclr",
            "request": "attach",
            "processId": "${command:pickProcess}"
        }
    ]
}

Also, the tasks.json will only compile the Reminders.Cli project:

{
    "version": "2.0.0",
    "tasks": [
        {
            "label": "build",
            "command": "dotnet",
            "type": "process",
            "args": [
                "build",
                "${workspaceFolder}/Reminders.Cli/Reminders.Cli.csproj"
            ],
            "problemMatcher": "$msCompile"
        }
    ]
}

It is possible to customize these two files so that you can pick which project to run. In this case we want to be able to run either Reminders.Web or Reminders.Cli.

The first thing you should do is to remove the second item in tasks.json's args array: "${workspaceFolder}/Reminders.Cli/Reminders.Cli.csproj". We don't need it because we have the solution file (Reminders.sln). When dotnet build is executed in the folder that contains the sln file all the projects referenced in the solution get built.

After that we can now go to launch.json and click the "Add Configuration..." button in the bottom left corner.

Select ".Net: Launch a local .NET Core Web Application" and in the generated json change "program" and "cwd" (current working directory) to:

"program": "${workspaceRoot}/Reminders.Web/bin/Debug/netcoreapp2.0/Reminders.Web.dll",
"cwd": "${workspaceRoot}/Reminders.Web"

You can now go to the Debug "section" of Visual Studio Code and see the Web and Console app configuration there:

Note that you can change the configuration names if you like, just change the "name" property in launch.json.

Now when you press F5 inside VS Code the project that will run is the one that is selected in the Debug section of VS Code.

This only affects VS Code though, the dotnet run command is unaffected by these changes. In the terminal, if you navigate to the solution's folder and type dontet run you'll get an error (Couldn't find a project to run...). You need to use the --project argument and specify the path to the csproj file you want to run, e.g.: dotnet run --project Reminders.Web/Reminders.Web.csproj. Alternatively you can navigate to the project's folder and execute dontet run there.

Adding Classes and Interfaces

In full Visual Studio you have several templates available when adding new files to a project.

In VS Code the only out of the box option is to create a new file and use the VS Code snippets (e.g. typing class and then TAB).

Fortunately there's a VS Code extension named "C# Extensions" which adds two options to the context menu on VS Code's explorer view: "New C# Class" and "New C# Interface".

This extension also adds the ability to generate a class' constructor from properties, generate properties from the constructor and add read-only properties and initialize them (there are demos of this in the extension's page).

This section wouldn't be complete without mentioning a currently abandoned alternative named yeoman, specifically the generator-aspnet.

Yeoman is a nodejs application that allows you to install yeoman generators which generate code. For ASP.NET Core there was a particularly interesting generator named generator-aspnet.

That generator provided a set of project templates similar to what dotnet new now offers (mvc application, console app, etc). But that was not the most interesting thing about generator-aspnet. It was its set of subgenerators.

A subgenerator in yeoman is a small generator for creating just one file, for example a class, an interface or an empty html page. Here's an example of using generator-asp to create a new class: yo aspnet:class ClassName

Generator-asp had loads of these:

yo aspnet:angularcontroller [options] <name>
yo aspnet:angularcontrolleras [options] <name>
yo aspnet:angulardirective [options] <name>
yo aspnet:angularfactory [options] <name>
yo aspnet:angularmodule [options] <name>
yo aspnet:appsettings [options]
yo aspnet:bowerjson [options]
yo aspnet:class [options] <name>
yo aspnet:coffeescript [options] <name>
yo aspnet:dockerfile [options]
yo aspnet:gitignore [options]
yo aspnet:gruntfile [options]
yo aspnet:gulpfile [options] <name>
yo aspnet:htmlpage [options] <name>
yo aspnet:interface [options] <name>
yo aspnet:javascript [options] <name>
yo aspnet:json [options] <name>
yo aspnet:jsonschema [options] <name>
yo aspnet:middleware [options] <name>
yo aspnet:mvccontroller [options] <name>
yo aspnet:mvcview [options] <name>
yo aspnet:nuget [options]
yo aspnet:packagejson [options]
yo aspnet:program [options]
yo aspnet:startup [options]
yo aspnet:stylesheet [options] <name>
yo aspnet:stylesheetless [options] <name>
yo aspnet:stylesheetscss [options] <name>
yo aspnet:taghelper [options] <name>
yo aspnet:textfile [options] <name>
yo aspnet:tfignore [options]
yo aspnet:typescript [options] <name>
yo aspnet:typescriptconfig [options] <name>
yo aspnet:typescriptjsx [options] <name>
yo aspnet:usersecrets [options] <name>
yo aspnet:webapicontroller [options] <name>

Unfortunately they were removed in version 0.3.0. This list is from generator-asp@0.2.6 (latest version as of today, 25/2/2018 is 0.3.3).

From the talk on the generator-aspnet github issues, although never mentioned explicitly, it seems that dotnet new is the only way to go from now on. Unfortunately the only generation commands that are similar to these subgenerators in dotnet new are Nuget Config, Web Config, Razor Page, MVC ViewImports and MVC Viewstart (you get a list of them when you type dotnet new --help).

You can still install this specific version of generator-aspnet though (npm install generator-aspnet@0.2.6) and use it (some of the generators are still useful).

However, for creating classes the extension I mentioned previously is the easiest and most convenient to use. Let's use it to create a Reminder class inside the Reminder.Common project.

The Reminder class has an Id property, a Description, a Date and a boolean flag IsFinished to indicate the reminder was acknowledged.

First thing you need to do is install the "C# Extensions extension". Then right click on Reminders.Common in the explorer view and select New Class:

Name it Reminder.cs and create the three properties. In the end it should look like this:

using System;

namespace Reminders.Common
{
    public class Reminder
    {
        public int Id { get; set; }
        public string Description { get; set; }
        public DateTime Date { get; set; }
        public bool IsFinished { get; set; }
    }
}

Now put the cursor on the opening "{" after class Reminders and click the light bulb and select "Initialize ctor from properties":

You should end up with this:

using System;

namespace Reminders.Common
{
    public class Reminder
    {
        public Reminder(int id, string description, DateTime date, bool isFinished)
        {
            this.Id = id;
            this.Description = description;
            this.Date = date;
            this.IsFinished = isFinished;

        }
        public int Id { get; set; }
        public string Description { get; set; }
        public DateTime Date { get; set; }
        public bool IsFinished { get; set; }
    }
}

Although the constructor isn't really necessary (and it won't play well with Entity Framework if you are planning to use it) , it's just an example of some nice features you get from the extension. Also, when defining properties you can use the same snippets that are available in full Visual Studio (i.e. type prop and press TAB).

Razor views

Razor pages (.cshtml) is definitely an area where VS Code is lacking. There's no intellisense or even auto-indent.

There's an extension in the marketplace named ASP.NET Helper which will get you intellisense, however you have to import the namespaces for your models in _ViewImports.chtml (the extension's requirements are at the bottom of the extensions page). Also in its current version it only seems to work if the model you use is in the same project as the view.

This extension can be useful in some situations but it's very limited.

If this is a deal breaker for you here's the github issue for adding Razor support to the language service that VS Code relies on and which would provide intellisense similar to what is there for the full version of Visual Studio. If you upvote or leave a comment there it will bring visibility to this issue.

If you want a good experience while creating razor pages today and you don't mind paying a little bit of money you can try JetBrains Rider. Rider runs on Windows, Mac and Linux and is from the same people that created ReSharper. I tried it a good few months ago when it was in beta. At that time it required a fairly decent machine to run smoothly and it was lacking some features. I tried it again today while I'm writing this and it seems much much better. Razor support seems perfect.

Even though the experience with Razor in VS Code right now is not ideal, if you go with a front-end JavaScript framework (Angular, React, Vue, etc) instead of Razor, you'll find that VS Code is excellent. I've used mostly Angular and TypeScript and this is an area where VS Code is arguably better then the full version of Visual Studio.

If you are somewhat familiar with Angular and are planning to use ASP.NET Core as a Web Api for your front end check out my other article Angular and ASP.NET Core.

Nuget packages

In the full version of Visual Studio there's an option to Manage NuGet packages. You get an UI where you can search, update and install NuGet packages. In VS Code there's no support for NuGet out of the box.

Thankfully there's an extension you can install that will enable searching, adding and removing NuGet packages from VS Code. The extension is named NuGet Package Manager.

Alternatively you can use the command line. To add a package to a project the syntax is: dotnet add PathToCsproj package PackageName. Or, if you navigate to the project you want to add the NuGet package to, you can omit the path to the csproj file: dotnet add package PackageName.

Imagine you wanted to add the Microsoft.EntityFrameworkCore package to Reminders.Common class library. After navigating to it:

$ dotnet add package Microsoft.EntityFrameworkCore

Adding tools

An example of a CLI tool is for example Entity Framework's tool. That's what runs when you type in a project that has the EF tooling installed:

$ dotnet ef

And you should get this output:

             _/\__
       ---==/    \\
___  ___   |.    \|\
| __|| __|  |  )   \\\
| _| | _|   \_/ |  //|\\
|___||_|       /   \\\/\\

Entity Framework Core .NET Command Line Tools 2.0.1-rtm-125

Usage: dotnet ef [options] [command]

Unfortunately there's no automated way of configuring dotnet tooling. If you start with a project template that doesn't have the right setup you'll have to manually edit the .csproj file.

Not only that but you'll have to know the right package name and version.

This is not a problem specific to VS Code or Linux. This is a tooling problem. Here are the relevant issues in github: dotnet add package doesn't honor package type and VS 2017: PackageType=DotnetCliTool fails to install

There's a way to manually do it though. If you open a .csproj file you'll notice that it has ItemGroup sections. For example, for a new mvc project there's this:

<ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore.All" Version="2.0.0" />        
</ItemGroup>

For the tooling you should create another ItemGroup and inside it instead of PackageReference use DotNetCliToolReference (there may already be an ItemGroup with DotNetCliToolReference in your project, if that's the case just add to it).

Let's imagine we want to use the Entity Framework tooling in our web project. Navigate to Reminders.Web and open the csproj file.

Make note of the version of Microsoft.AspNetCore.All package. For example, let's imagine it's "2.0.0".

Create a new ItemGroup (or find the ItemGroup that already has DotNetCliToolReferences) and inside add the DotNetCliToolReference for the Microsoft.EntityFrameworkCore.Tools.DotNet with Version="2.0.0".

<ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore.All" Version="2.0.0" />            
</ItemGroup>
<ItemGroup>
    <DotNetCliToolReference Include="Microsoft.EntityFrameworkCore.Tools.DotNet" Version="2.0.0" /> <!-- ADD THIS -->
    <DotNetCliToolReference Include="Microsoft.VisualStudio.Web.CodeGeneration.Tools" Version="2.0.0" />        
</ItemGroup>

You should now be able to run dotnet ef in the Reminders.Web project.

The reason the version must match "Microsoft.AsNetCore.All" is because that package is a metapackage, i.e. a package that just references other packages. Some of those packages are Entity Framework packages which would cause compatibility issues with the tooling if the version does not match.

If we were installing Entity Framework Core tooling in a console project you wouldn't need to worry about the version. You would however have to add the package Microsoft.EntityFrameworkCore.Design additionally to manually adding the DotNetCliToolReference.

Running tests

One thing your development workflow probably involves is writing unit tests.

VS Code comes with out of the box support for running unit tests. For example if you open UnitTest1.cs you'll notice that over the test method there are two links, run test and debug test:

This will allow you to run a unit test at a time, there's no way inside VS Code of running all the tests in a project. You can however navigate to a test project in a terminal and type:

$ dotnet test

This is the output you should get if you do it for Reminders.Tests:

Starting test execution, please wait...
[xUnit.net 00:00:00.4275346]   Discovering: Reminders.Tests
[xUnit.net 00:00:00.5044862]   Discovered:  Reminders.Tests
[xUnit.net 00:00:00.5549595]   Starting:    Reminders.Tests
[xUnit.net 00:00:00.6980509]   Finished:    Reminders.Tests

Total tests: 1. Passed: 1. Failed: 0. Skipped: 0.
Test Run Successful.

Alternatively you can specify the path to the test project you want to run, for example dotnet test Reminders.Tests/Reminders.Tests.csproj.

It is also possible to specify which tests to run in the command line, for example say you only want to run the tests in class MyClassTests. Here's how you can do that:

$ dotnet test --filter MyClassTests

The filter functionality has several more options. The example above is short for dotnet test --filter FullyQualifiedName~MyClassTests (~ means contains).

There are other handy options, for example in xUnit it's possible to specify "traits" for tests, for example:

[Trait("TraitName", "TraitValue")]
public class UnitTest1
{
    [Fact]        
    public void Test1()
    {
        //...
    }
}

You can then run:

$ dotnet test --filter TraitName=TraitValue

And only tests that have that trait will be run. You can specify Trait on a method or class level.

Although trait is specific to xUnit, there are equivalent constructs on other testing frameworks. For example, mstest has TestCategory.

If you prefer to do everything inside VS Code you can create tasks for running tests. First go to tasks.json and make a copy of the build task, change its label to test (or whatever you want to call it) and update the args so that instead of build, test is executed with the path to the test project, for example for Reminders.Tests:

    {
        "label": "test",
        "command": "dotnet",
        "type": "process",
        "args": [
            "test",
            "${workspaceFolder}/Reminders.Tests/Reminders.Tests.csproj"
        ],
        "problemMatcher": "$msCompile"
    }

You can run tasks in VS Code by using the command palette (Ctrl + Shift + P) and typing "Run Task" and then selecting the task you want to run. You can even set a task as the "test task" and assign a keyboard shortcut for it (the same way you can do Ctrl + Shift + B to trigger a task configured as the "build task").

To set a task as the "test task" open the command palette and select "Tasks: Configure Default Test Task", choose the test task and you're done.

One last tip

Sometimes VS Code (actually Omnisharp) seems to go haywire, for example intellisense stops working. When that happens you can either reload VS Code by using the command palette and selecting "Reload Window" or the less aggressive option is to restart Omnisharp (the command name is: Omnisharp: Restart Omnisharp).

Hope this guide was helpful, let me know in the comments.

Angular and ASP.NET Core

Rui Figueiredo — Thu, 01 Mar 2018 23:01:07 +0000

The Angular CLI provides a way to develop front-end applications using angular that hides a lot of details. For example there's no requirement to understand how Webpack or SystemJS work.

In fact, if you don't know a little bit about Webpack, which is what is used to build the latest version of Angular applications, the CLI almost looks like magic. You just need to do a ng new and ng serve --open and you have a working Angular application open in your web browser.

The fact that the CLI hides all the plumbing might lead to questions like: "How do I use Angular with ASP.NET Core?".

I hope that by the end of this blog post it will be clear to you how you can answer that question (and not only with ASP.NET Core, with whichever technology you want to use your Angular app with).

You see, an angular app is an app in and of itself, it does need to be "served" somehow by a web server.

When you compile an angular application you are producing a set of JavaScript, CSS and one index.html file. That's it.

The default folder where those "artifacts" get copied to is yourApplicationFolder/dist. You can check it out by going to your Angular application and doing an ng build.

Go on, I'll wait.

When you do ng serve --open you are actually using a stand-alone web server (webpack-dev-server) to serve that index.html file in the dist folder.

The rest of this blog post will describe several approaches that you can take for using Angular with ASP.NET Core. The first is to have ASP.NET Core serve the Angular files.

The second approach is to have Angular and ASP.NET Core as different applications. There's an example of how to achieve this using Nginx where both Angular and ASP.NET Core are served using port 80 and in IIS where each application is served from its own port.

The final part of the post describes a setup that I consider ideal where you can use Angular's ng serve during development.

This post is quite long but the sections are fairly independent. If your are only interested in the last section and you are using Windows I recommend also reading the section on how to configure Angular in IIS.

Using ASP.NET Core to serve the Angular application

It can be argued that serving an Angular application "within" ASP.NET Core is wasteful in terms of resources. In the end the Angular application is just a set of static files, there's no need to have the request for those files go through the ASP.NET Core middleware pipeline.

There might be some good reasons for doing it though, also there's no harm in knowing how to do it and since it seems to be a common approach, being familiar with it might be useful.

One important thing to know in order to understand how we can serve an ASP.NET Core and Angular application together is to understand how a request is processed in ASP.NET Core.

When you run an ASP.NET Core application your request goes through a "pipeline" of middlewares. Every time a request comes in it goes through the middlewares in the order they are defined, and then in reverse order.

Every middleware has an opportunity to change the request or response two times, once before the other middlewares have been executed, and then after the other middlewares have executed. This allows for a middleware at the top of the pipeline to handle for example, a 401 response set by a middleware further down in the pipeline.

An example of this are the authentication middlewares that change a 401 response to a 302 redirect to a login page.

You can find the definition of this pipeline on the Startup.cs file, in the Configure method. For example, here's the pipeline that you get when you do a dotnet new mvc:

public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
    if (env.IsDevelopment())
    {
        app.UseDeveloperExceptionPage();
    }
    else
    {
        app.UseExceptionHandler("/Home/Error");
    }

    app.UseStaticFiles();

    app.UseMvc(routes =>
    {
        routes.MapRoute(
            name: "default",
            template: "{controller=Home}/{action=Index}/{id?}");
    });
}

Every time a request comes in to this ASP.NET Core application it can go through at most three middlewares. First the DeveloperExceptionPage/ExceptionHandler middleware depending if the ASP.NET Core application is running in development mode or not. Then the StaticFiles middleware and then finally the Mvc middleware.

The middleware that is key here is StaticFiles. This is the middleware that serves files contained in the wwwroot folder, i.e. if a request comes in for index.html and there's an index.html file at wwwroot/index.html then that file is sent to the client. StaticFiles middleware won't call the middlewares below it after this (in this case it would be Mvc).

You can probably already see how this could work with an Angular application. Just put it under wwwroot.

That's absolutely correct, however there's a detail about StaticFiles that is important to know. StaticFiles won't try to do any guesses for you, i.e. if your request is for /, StaticFiles won't look for /index.html. It will just assume that this request isn't supposed to be handled by it and it will call the next middleware in the pipeline, in this case Mvc.

For this approach to work you need another middleware named DefaultFiles which must come before StaticFiles in the pipeline:

//...
app.UseDefaultFiles();
app.UseStaticFiles();
//...

DefaultFiles will cause cause StaticFiles to look for index.html if the url ends with /.

Now the only thing left to do is to configure your Angular CLI to compile to your ASP.NET Core application's wwwroot folder.

If you look in your Angular's application folder you'll find a .angular-cli.json file. In that file look for the outDir property:

...
"apps": [
{
    ...
    "outDir": "dist",
...

Change it from "dist" to the path of your ASP.NET Core's wwwroot folder. Run ng build in your Angular application and now if you run your ASP.NET Core web application you should see your Angular application in the browser.

A nice development workflow is to run the Angular CLI build in watch mode: In a console window do ng build --watch or ng build -w if you want to save a few key strokes, and leave it running. Now every time you make a change in your Angular application you can just refresh the browser and see the change (you also need to have your ASP.NET Core application running).

There is one thing missing from this approach, though. Deep-linking support, i.e. if your Angular application uses routing and you send a user a url with a valid Angular route (e.g http://yourapplication.com/products/5) the receiving user won't be able to open it. Trying to get to that route will result in a 404 Not Found response.

That's because the request will go all the way through your ASP.NET Core application's pipeline and when it reaches the MVC middleware it won't know what to do with it and will set the response's status code to 404 Page Not Found.

What we can do is at the top of the pipeline we look for a 404 response that is about to be sent and change its path to our Angular application's index.html file (that way what gets served is the Angular application which will know what to do with the url in terms of routing). After this we make the request go through the pipeline again:

//add this at the start of Configure
app.Use(async (HttpContext context, Func<Task> next) =>
{
    await next.Invoke();

    if (context.Response.StatusCode == 404)
    {
        context.Request.Path = new PathString("/index.html");
        await next.Invoke();
    }
});

That fixes deep links but introduces a new problem. What if your web api (that you've implemented in your ASP.NET Core application) needs to send a 404 response. That's something more than reasonable to do. Instead of a 404, the service call will receive a 200 response with index.html.

The solution here is to look at the url and decide if it's intended for the web api or an Angular route. Usually a call to the web api will have /api in the url. That's a simple test to perform and it will solve this problem. Here's the revised version of a custom middleware that solves this problem:

//add this at the start of Configure
app.Use(async (HttpContext context, Func<Task> next) =>
{
    await next.Invoke();

    if (context.Response.StatusCode == 404 && !context.Request.Path.Value.Contains("/api")))
    {
        context.Request.Path = new PathString("/index.html");
        await next.Invoke();
    }
});

One last note about this approach. I've seen examples where the Angular application is in the same Visual Studio solution as the ASP.NET application. Visual Studio (not VS Code) will try to compile the typescript files. If you are using ng build -w you'll want Visual Studio to leave your Typescript files alone. To do that open your project's .csproj and add in any PropertyGroup:

<TypescriptCompileBlocked>true</TypescriptCompileBlocked>

Nginx

Nginx is a web server that can act as a reverse proxy for ASP.NET Core applications and which is also very good at serving static content.

The setup for having an Angular application work with ASP.NET Core is much simpler in Nginx. You just need a configuration similar to this:

server {
    listen 80;        

    location / {
        root /pathToYourAngularApplication/dist;
        index index.html;
        try_files $uri $uri/ /index.html;
    }

    location /api/ {
        proxy_pass http://localhost:5000;
    }
}

This is how a typical Nginx configuration file looks like. If you are not familiar with Nginx and ASP.NET Core I recommend my blog post: HTTPS in ASP.NET Core from Scratch. It has a section with instructions on how to install and setup websites using Nginx.

This configuration allows us to have both the Angular and ASP.NET Core application on port 80. Let's look at the important parts in it.

The listen 80 statement establishes that Nginx will respond to requests coming in on port 80.

The location blocks are where we are going to define how our two applications will be served (Angular and ASP.NET). Each time a request comes in, Nginx will look at the URL and try to find the location block that best matches it. In this case the location blocks urls act like a "prefix match", i.e., the first block will match every URL (every url that starts with a /). The second location block matches URLs that start with /api/.

Nginx picks the most "specific" location block, so even though a request for /api/users would match both location blocks, since the second one (/api/) is more specific, it will be the one that would be used to handle the request.

In the first location block (/):

root /pathToYourAngularApplication/dist sets the path where static content will be looked for as the location where your compiled Angular application files are (dist is the CLI's default output folder).

index index.html specifies which file should be served for URLs that end in /.

try_files $uri $uri/ /index.html can be read this way: check if there's a file that matches the normalized URL (e.g. http://www.yourwebsite.com/assets/image.jpg -> /assets/image.jpg), if that file does not exist try the normalized URL plus a / (e.g. http://www.yourwebsite.com/documents -> /documents/ -> /documents/index.html because of the index rule). If all of that fails serve the file /index.html.

Serving /index.html if no match is found is what enables us to use deep linking. For example a URL such as http://www.yourwebsite.com/documents with no math in the file system will be served with the Angular application's index.html. Index.html will load all the necessary files for the Angular application to run, specifically the routing module. The routing module will then look at the url, and according to the routes defined in the angular app will decide which component to load.

Finally, the last location block. It instructs Nginx to forward the requests that start with /api/ to a webserver that is listening on port 5000 on localhost. That will be your ASP.NET Core's application.

One note about the Nginx's syntax for proxy_pass. It matters a lot if the URL for the application has a / at the end or not. The url in proxy_pass is treated differently if it has what is described in Nginx's documentation as a "optional URI" (optional URI isn't a great name, since in the end a URL is a URI).

An example of a URL with an optional URI is: http://localhost:5000/optionalURI/. If the location's path is /api/, then a request for http://yourwebsite.com/api/users will be forwarded to your ASP.NET Core's application as http://localhost:5000/optionalURI/users.

That's why not adding the / at the end in proxy_pass is so important, because if you do (e.g.: proxy_pass http://localhost:5000/;) it falls into the "optional URI" category (it will be interpreted as an empty optional URI), and a request for http://yourwebsite.com/api/users will be seen in your ASP.NET Core's application as a request for http://localhost:5000/users.

If you don't add the / at the end (e.g.: proxy_pass http://localhost:5000;) then a request for http://yourwebsite.com/api/users will be seen in the ASP.NET Core application as a request for http://localhost:5000/api/users which is probably what you want.

If you need a more complete example that explains how you can make this work outside a development-time scenario (i.e. have your ASP.NET Core application auto start and remain online even if there's an exception) check out HTTPS in ASP.NET Core from Scratch where there's an example describing how you can use Supervisor to keep the ASP.NET application running even in the event of errors (by auto restarting it).

IIS

With IIS it becomes very cumbersome to have a configuration similar to what we can do with Nginx where both the Angular and ASP.NET Core applications are served on port 80.

To understand why it makes it easier if we understand the IIS concepts of Website and Application. When you create a website you define (among other settings) the port (e.g. 80) where it will be served from. A website can then have several applications "inside" it, all of which will share the website configuration (and therefore be served on the same port).

We could for example put our Angular application inside the "Default Web Site" and the ASP.NET Core one as an IIS Application under it, and call it for example "api".

If the "Default Web Site" responds at http://localhost, then the ASP.NET Core application could be at http://localhost/api. Which seems to be exactly what we want. However, the requests for http://localhost/api would be seen in ASP.NET Core without the api in the url.

As far as I know there's no way to change this behavior.

This means your ASP.NET Core application will behave differently when running inside IIS vs when executed directly (either in Visual Studio or with dotnet run).

To make matters worse an ASP.NET Core application needs to be published (dotnet publish) for it to work in IIS. It's not like a non-Core ASP.NET application where you can just point an IIS Application to the folder that contains the ASP.NET application's files .

So when using IIS the reasonable options are to either have ASP.NET Core serve the angular application as it was described in the first section of this article or have two separate Websites.

Lets walk-though the process of creating two separate websites. First a website for the Angular project and then for ASP.NET Core.

Angular in IIS

We'll be adding a Website named MyNgWebSite on port 80. That means that if you have a "Default Web Site", which in all likelihood you'll have, you need to stop it or change its bindings since the default for it is port 80.

But before we get there we need to create an application pool for our Angular application. Right click on Application Pools in IIS Manager:

The Application Pool for an Angular application does not require Managed Code (we only need to serve static files). We should choose "No Managed Code" in the .NET CLR Version:

We can now add a new IIS web site and set the new application pool we created as its application pool:

The physical path should be set to where your Angular project is being compiled to, usually this is the dist folder.

If you were to try to access http://localhost right now (and assuming that you stopped the "Default Web Site" or used a different port than 80) you would get a permissions error. That's because when you create an application pool a "virtual" user is created. That user is a local user and must have permissions to access the folder that contains the files you are trying to serve.

That user's name is IIS AppPool\ApplicationPoolName, in this example it's IIS AppPool\ApplicationPoolForAngular.

Go to the folder that contains the compiled Angular project, right click on it and select properties, go to the security tab, click edit, then add and finally add the application pool user:

We should now be able to access your Angular application if you go to http://localhost.

We still need to do one more thing though. Enable deep-linking support.

If you have routes in your Angular application these won't work if someone tries to access them from "outside" the Angular app. What this means is that if navigating to http://localhost/documents is valid inside the Angular application and you send that url to someone else, when that someone else clicks the link they will be greeted with a 404 page from IIS.

That's because there is no documents folder nor index file inside it for IIS to serve. We need to tell IIS that it must serve the file index.html when someone tries to access a URL that does not exists.

We are going to use the same mechanism used for having a custom 404 page, but instead of a 404 page we'll serve the Angular application.

To achieve this we need to create a web.config file and put it in the src folder of the Angular application with this inside:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <httpErrors errorMode="Custom" existingResponse="Replace">
            <remove statusCode="404"/>
            <error statusCode="404" responseMode="ExecuteURL" path="/index.html"/>
        </httpErrors>
    </system.webServer>
</configuration>

A very quick explanation of what's going on. We are using httpErrors with an errorMode="Custom" and existingResponse="Replace". This instructs IIS to replace the default error pages with the one we are about to specify.

remove statusCode="404" will remove any custom settings for 404 pages if they already exist.

error statusCode="404" responseMode="ExecuteURL" path="/index.html" will configure IIS to execute the /index.html url if there's a 404 error. This will effectively serve your Angular application and won't change the URL seen by the client.

Now we need to edit the .angular-cli.json file so that web.config gets copied to the output folder as an asset when the application is compiled. The assets section is under "app", here's an example:

{
"$schema": "./node_modules/@angular/cli/lib/config/schema.json",
"project": {
    "name": "your-app"
},
"apps": [
    {
    "root": "src",
    "outDir": "dist",
    "assets": [
        "assets",
        "favicon.ico", 
        "web.config"
    ],
    "index": "index.html",
...

ASP.NET Core in IIS

The process for the configuring an ASP.NET Core application in IIS is similar, although we need to select a different port.

But before you start you need to make sure you have the ASP.NET Core Module for IIS installed. It might already be installed if you installed the .Net Core SDK, however the best way to make sure is to go to IIS Manager and see if it's in the modules' list:

If you don't have it you can find more information about it here and a direct link to download it here.

This module takes care of starting and keeping an ASP.NET Core application running.

Before we create the website in IIS we need the published version of the ASP.NET Core application. You can do that in the command line with dotnet publish or, in full Visual Studio, right click on the project and select Publish, then click publish to folder.

Create a new Website and point it to the ASP.NET Core project published folder, give it a different port number (for example 8080) and create an Application Pool for it.

An application pool for an ASP.NET Core application is also unmanaged (No Managed Code). Although this might seem odd, it's because IIS is actually just acting as a reverse proxy.

Before we're able to run the ASP.NET Project using IIS we need to changed the published folder's permissions so that the Application Pool user can access it. If you don't you'll get this moderately unhelpful error message:

HTTP Error 500.19 - Internal Server Error
The requested page cannot be accessed because the related configuration data for the page is invalid.

If you look at the Config Error section you'll see "Cannot read configuration file due to insufficient permissions", which pretty much says it all.

Go to the published folder and add the application pool user to the list of users with permissions over that folder.

Your ASP.NET Core application should now be available on the port you've selected when you created the website in IIS. However, if you try to call it from the Angular application you'll get this error "Failed to load ... No 'Access-Control-Allow-Origin' header is present on the requested resource...". Here's an example of how that would look like in the developer tools console tab:

That's because even though both our our Angular and ASP.NET Core applications are on the same domain, they are in different ports, and that's enough to qualify the request as a Cross Origin Resource Sharing (CORS) request in all browsers except IE.

We need to enable CORS on the ASP.NET Core application. To do that we need to add the package Microsoft.AspNetCore.Cors and in ConfigureServices(IServiceCollection services... method in Startup.cs add services.AddCors():

public void ConfigureServices(IServiceCollection services)
{
    //...
    services.AddCors();
    //...
}

And in the Configure method we need to create a "policy" that says that we are expecting requests from http://localhost. We should do that before the MVC middleware:

public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
    //...
    app.UseCors(builder => builder.WithOrigins("http://localhost"));
    app.UseMvc();
}

You should be good to go. Your Angular and ASP.NET Core should both be working now.

Platform Agnostic Development Setup

Both Angular and ASP.NET Core applications provide ways to detect if they are running in development or production mode. That can be leveraged to create a setup that works both in Linux, Windows or Mac.

The easiest way to run an Angular application is to use run ng serve. That spins up a webpack development server that serves the Angular application on port 4200 by default.

This also has the advantage of having hot module replacing, which means you can see your changes to the Angular application as soon as you make then without even having to refresh the browser.

So ideally we want to run the Angular application this way.

For the ASP.NET Core application we want to run it without having to publish it which you would have to if it is being served by IIS.

This is the ideal development scenario, ng serve for Angular and dotnet run or running the ASP.NET Core from Visual Studio without having to publish it.

In this ideal scenario when developing we could have the Angular application running on port 4200 (through ng serve) and the ASP.NET Core application running on port 5000. When in production the Angular application would typically be served from port 80 and the ASP.NET Core application for port 8080 for example (or from a different server on port 80).

On the ASP.NET Core side of things we'd have to configure CORS to accept requests from port 4200 when in development and from port 80 when in production. In Startup.cs that would look like this:

public void ConfigureServices(IServiceCollection services)
{
    services.AddCors();
    //...        
}

// This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
    //...
    if (env.IsDevelopment())
    {
        //...
        app.UseCors(builder => builder.WithOrigins("http://localhost:4200"));
    }else 
    {
        app.UseCors(builder => builder.WithOrigins("http://localhost"));
    }

    app.UseMvc();
}

That takes care of the ASP.NET Core application.

For Angular we need to leverage the environemnt.ts and environemnt.prod.ts files. You can find then under a folder name environemnts under the src folder on an Angular project.

What you put on environment.ts will be available when you run in development mode (the default) and the values in environment.prod.ts will be used when in production. To compile the Angular project with the environment set to production use the --env=prod flag (e.g. ng build --env=prod).

Here's a simple example of how the environment files could be configured to support our hypothetical scenario, environment.ts:

export const environment = {
    production: false,
    apiBaseUrl: "http://localhost:4200/"
};

environment.prod.ts:

export const environment = {
    production: true,
    apiBaseUrl: "http://localhost/"
};

In your Angular services, to get to the environment values you just need to import the environment (always environment and not environment.prod):

import { Injectable } from '@angular/core';
import { HttpClient } from '@angular/common/http';
import { environment } from '../environments/environment';

@Injectable()
export class MyServiceService {

    constructor(private httpClient: HttpClient) { }

    getStuff(){
        return this.httpClient.get(`${environment.apiBaseUrl}/api/suff`);
    }  
}

This approach would work even if you host on Nginx or IIS so probably the best option if you need/want to support having developers using different platforms of if you just want to compare performance between them.

Things you wanted to know about storing passwords but were afraid to ask

Rui Figueiredo — Mon, 04 Dec 2017 22:32:38 +0000

Security breaches are very common. To make matters worse, when it comes to users' passwords it is frequent that no reasonable precautions were taken to ensure that they can't be easily extracted from the breached data.

People tend to use the same password, or simple variations of it in multiple sites. This makes it easier for one to remember a password, but it also means that when that password is exposed, an attacker can potentially get access to the other websites where that password was used.

The thing is, there's no reason for this to be so common nowadays. Even though the science around cryptography is fairly complicated, the algorithms are easy to use and are readily available.

This blog post is an attempt at explaining what good qualities a stored password should have and which algorithms can be used to get there.

So what qualities should a stored password have? First, only the user should know it. There cannot be a way of getting to the password after it is stored.

That rules out saving the password in clear text, but it also rules out encrypting it.

This might sound like common sense, but unfortunately there's been examples of fairly prominent online business that would just email you the password in clear text if you use their "Forgot password functionality".

The example I'm thinking of right now is Tesco, one of the biggest supermarket chains in the UK and Ireland. A few years ago, when pressed about it, they said that they were encrypting the passwords. This is not a good idea since if there's a security breach and the cryptographic key that was used to encrypt the passwords were to be discovered, the attackers would have access to all of the users' passwords.

So if encrypting the passwords is not a good idea, what alternative can we use?

Hashing

Maybe we could hash the passwords. But first, what is hashing?

The idea behind hashing is that you can take any input and produce an output that has a specific size and from which you cannot "extract" original input.

For example here's the output of the SHA1 hashing algorithm in base64 for the string "hello world":

22596363b3de40b06f981fb85d82312e8c0ed511

And here's the SHA1 hash for "hello world!":

f951b101989b2c3b7471710b4e78fc4dbdfa0ca6

And here's the SHA1 hash for a video file with 700Mb:

1b2803f08a8f2ba251a557cd61f1a38b07427011

Two very similar inputs produce completely different hashes, and irrespectively of the size of the input the output always has the same size.

Hashing algorithms are typically used for detecting duplicate data. Imagine you are storing video files and you want to determine if a certain video file has already been stored.

If each time you need to store a new video file you first compute its hash, you can then compare the video file's hash with previously stored hashes. If there is already a hash that is the same as the video file you are trying to save that means it's a duplicate, if not, you can save the video file with the hash. The advantage of doing this is that it is much faster to compare two hashes than two video files.

Another common use case for hashing is to detect if data has been corrupted while being transmitted.

For example, if a file needs to be transmitted through a medium that is not very reliable, before the file is sent a hash of it is created. Then, the file is sent together with the hash.

On the receiving end the hash is computed again and compared with the hash that was transmitted. If they don't match it means the file was corrupted.

Some examples of hashing algorithms are MD5, SHA1, SHA256 and SHA512 just to name some common ones.

The qualities that the output of these algorithms have seem like they are very appropriate for storing passwords. Given a password, compute it's hash and store that.

When the user tries to login, compute the hash from the password the user entered in the login form. If that hash matches the stored one, then it's the right password.

Unfortunately, there's a catch here. Hashing is supposed to be very fast. In its original use cases (e.g. duplicate detection, validating that data has not been changed/corrupted) it needs to be very fast.

That makes it a bad match for storing passwords because if it's fast to compute that means many hashes can be generated in a short period of time. Which means that if you want to figure out what password was used to generate a particular hash, you can try many candidate passwords in a short period of time to see if any produces a hash that matches the one you are looking for.

Because most passwords people pick are somewhat predictable (e.g. words that you can find on a dictionary or simple variations of them), if an attacker manages to get access to a password's hash, most of the time finding that password becomes an exercise of computing hashes of words in a dictionary until finding a match.

A way to speed up the process of getting a password from a hash is to use data from previous breaches, namely the most commonly used passwords. By pre-computing the most common passwords it is possible to just do a lookup search from the hash to the password. If an attacker manages to get access to a website's database where the hashes are stored, it is very likely that this process will reveal a significant number of passwords.

This method of looking up passwords by using pre-computed password hashes is called a rainbow table attack. You don't even need to generate the hashes yourself, there are pre-computed hashes readily available online for purchase. They are not too hard to find.

HMAC

If we can't stop people from using the same passwords all the time is there something we can do to make hashing more secure?

What if we just appended a random string to the passwords?

We can store the random string with the hash of the password + random string combination. When the user tries to sign in, we append the random string with the password the user entered and recompute the hash. If they match the password is correct.

For example, if the password is "cutecats", instead of computing the hash of "cutecats", we can compute the hash of "cutecatsm@O88q39!Q^x" and store HASH("cutecatsm@O88q39!Q^x") and m@O88q39!Q^x.

It is very unlikely you'll find cutecatsm@O88q39!Q^x in a rainbow table.

An effective way to counteract a rainbow table attack is to generate a random string per password. If you consider the alternative of just having a random string for all passwords an attacker could still pre-compute all the hashes once and use them to try to get to the passwords.

If there's a random string per password it means that the attacker can't reuse any previously computed hashes.

The random string appended to the password is commonly know as salt.

Although we could just use an hashing algorithm to perform these operations, for example SHA256(password + salt), there are reasons for not doing it. Imagine that the password is "cutecats" and the salt is "1salt". Now imagine that the password is "cutecats1" and the salt is "salt". These two combinations will generate the same hash. This is suboptimal.

There's another algorithm that combines two inputs and generates an output with the same properties of the previously mentioned hashing algorithms. It's named HMAC which stands for Hash-based Message Authentication Code. Using HMAC, "cutecats", "1salt" and "cutecats1" and "salt" generate different outputs.

Before we go into what HMAC is, lets talk about MAC (Message Authentication Code) and what it's used for.

Imagine you want to communicate with someone over the internet and although you don't mind if someone is listening to your conversation, you want to make sure that the conversation is not tampered with (i.e. no one can send a message as one of the two people participating in the conversation). A hash of the message wouldn't do the trick here. A hash only guarantees that the message wasn't corrupted, if a bad actor has access to the message and can alter it, that means that he can compute a new hash for the altered message. To the receiver of the message it would look like everything was ok.

A Message Authentication Code addresses this issue by using a secret that is known only by the parties that take place in the communication. Using Alice and Bob as an example, if Alice wants to send a message to Bob she'll compute MAC(secret, messageToBob) and send that together with messageToBob. Bob, when he receives the message will repeat the process and compare the MAC code he generated with the one he received from Alice. If they match Bob can be confident that not only wasn't the message altered, but it was sent by Alice, since she's the only other person who could've generated a valid MAC.

In case you are wondering how Alice and Bob can agree on a secret even when there might be people listening to their communications have a look at Brief(ish) explanation of how https works, where you'll find an example of the Diffie–Hellman algorithm that is used exactly for this.

That is MAC. What about HMAC (Hash-based Message Authentication Code)? It's just an implementation of MAC that allows us pick from several different hashing algorithms to compute the MAC, for example there's HMAC-SHA256 where the hashing algorithm used is SHA256, of HMAC-SHA1 where the hashing algorithm is SHA-1, etc.

How can we then use HMAC to store passwords? We could, for example, pick SHA256 and compute HMAC-SHA256(password, salt) and store that together with the salt.

Solved? Unfortunately, no.

Although rainbow table attacks wouldn't work here, today's hardware allows us to generate these hashes so quickly that it becomes feasible to just use passwords from a list of commonly used password or even do a brute force attack, i.e. for example compute HMAC-SHA256(testPassword, salt) where testPassword is "a", then "aa", then "ab", up to all combinations of characters, numbers and symbols until a certain length.

To give you an idea, a high end graphics card that you can go buy today, for example a NVIDIA GTX 1080, can do ~4450 million SHA256 hashes per second. It is possible to have several of these cards in a single computer, so that number can be made much much higher.

Key Derivation

So if HMAC does not work, then what?

The idea of using a salt and a password is a good one, the only reason that it doesn't work is because computing HMAC is so fast that it is possible to guess the password in a reasonable amount of time. To do this the attacker would only need the password's hash and the salt (which would happen in the case of a breach) and good enough hardware that was capable of generating many hashes per second.

Maybe there's a way to make the process more computationally expensive?

What if we do HMAC(password, salt)=hash1 and then HMAC(password, hash1)=hash2 and repeat this a large number of times, for example 100.000. That would significantly slow down the process. Trying out passwords one by one, or a brute force approach be not be reasonable anymore.

There are a few algorithms for doing just that, and they do a little more than just repeat HMAC. They are called Key Derivation function algorithms.

The idea behind key derivation is that given a key you generate one or more keys from that original key that have certain properties. For example, the AES-256 encryption algorithm requires a key with 256 bits.

It is not reasonable to ask a user to generate an exactly 256 bit key by hand. That key would be extremely hard to memorize.

That's one of the uses of key derivation. It's a way of going from a key, for example the password the user thinks up, to another key that has a specific size.

But apart from that these key derivation have the property of, with the right configuration, being very computationally expensive.

For example PBKFD2 (Password-Based Key Derivation Function 2) can be configured to use a password, a salt, a particular HMAC algorithm and the number of iterations the HMAC should be applied. There is also bcrypt and scrypt that can also be used in a similar fashion.

As an example here's how using PBKDF2 in dotnet core would look like, with a 128 bit salt, 100.000 iterations and an output key size of 128 bits (16 bytes).

const int saltSize = 128 / 8;
const int outputKeySize = 128 / 8;
const int numberOfIterations = 100000;

var password = "cutecats";

using (var rng = RandomNumberGenerator.Create())
{
    var salt = new byte[saltSize];
    rng.GetBytes(salt);

    var derivedKey = KeyDerivation.Pbkdf2(password, salt, KeyDerivationPrf.HMACSHA256, numberOfIterations, outputKeySize);

    Console.WriteLine($"The salt in base64: {Convert.ToBase64String(salt)}");
    Console.WriteLine($"The 'hash/derived key' in base64: {Convert.ToBase64String(derivedKey)}");
}

To run this example you need to install the Microsoft.AspNetCore.Cryptography.KeyDerivation nuget package.

If you are wondering why I picked a 128 bit salt size or a 128 bit output key I don't have a good answer for that, but a good rule of thumb is that it should be less than what the hashing algorithm produces, for example we used SHA256 which produces a 256 bit output or 32 bytes.

By the way, "cutecats" is a horrible password, you should use a password manager like LastPass, 1Password or similar to generate passwords for you. That's because even if a password is stored correctly, if it's something like "cutecats" or for example a word you'd find in a dictionary then, even if it takes 1 second to try each password, an attacker might find it just by guessing and having a little patience.

So that's it, I hope I've helped explain why we should use passwords with a reasonable length, combinations of several numbers, symbols and characters and why they should not be stored in a way that can be easily "reversed" back to the the original password.

The Road to Modern JavaScript

Rui Figueiredo — Wed, 16 Aug 2017 12:01:48 +0000

When I recently decided to learn webpack I realized just how many new things were added to the JavaScript ecosystem in the last couple of years. Things that you need to know if you want to feel comfortable with all the new frameworks and tools like Angular, React, Gulp, Webpack, etc.

The goal for this blog post is to walk you through the major developments in the language that lead to what is considered modern JavaScript development. It also has examples that are illustrative of how modern tools and features work.

JavaScript development has changed immensely in the last two decades. When JavaScript was first introduced in 1995 one of the major goals was that it should be easy for beginners. It had requirements like being embeddable directly in HTML. It was supposed to be the "glue" between Java applets.

We all know that it evolved in a very different direction. All of that was motivated by JavaScript taking an ever more prominent role in web development, and that clashed with some of those earlier goals.

Scope and naming

In the 90s it was common to find this in a .html file:

<input type="button" value="Save" onclick="save();"/>
<script>
  function save() {
    //...
  }
</script>

Script tags with large chunks of code intermingled with HTML, plus inline event handlers. All of this quickly made the code hard to read and maintain.

Another thing that caused problems was it was really easy to get into a situation where you would accidentally redefine a function because you named it the same way as a previous one.

For example if there were two .js files that define a save function, the second one would override the first. This is perfectly valid in JavaScript, so there would be no errors or warning messages.

The solution for this problem was to try to mimic the namespace functionality that exists in other programming languages. We started doing things like:

var MyNamespace = (function() {
  function save(){
    //...
  }

  return {
    save: save
  };
})()

And then instead of just calling save() we'd call MyNamespace.save().

This takes advantage of the fact that in JavaScript new scopes are only created by functions. This became so popular that IIFE became a common "word" (iffy) between JavaScript developers. It means Immediately-invoked function expression. A really simple example is:

(function() { 
    //whatever variables and functions you declare here won't be "visible" outside the function
})()

It was now possible to have more complex applications, and to reuse parts of the code because the naming of functions was not an issue.

We also started making our JavaScript "unobtrusive", meaning that we didn't mix it with HMTL, and we made more object-oriented.

Too many files to load

As these new practices made writing more complex JavaScript more manageable we started to get into situations where we had a lot of it. That JavaScript had to be loaded to the browser, and as good practices dictate, it had to be separated over several files with meaningful names.

Well, there's a limit on how many concurrent GET requests a browser can do, and they are not many.

We've started using tools to bundle all our JavaScript. Bundling means that all the JavaScript code is concatenated into a single file. My first experience with bundling was with ASP.NET. With ASP.NET it's actually .Net code that bundles the JavaScript files.

This only worked in .Net so alternatives were required in order for this technique to be used with other technologies.

At some point in time someone decided that it would be a good idea to have JavaScript run outside of the browser. Node.js was created. Node leverages the open-source V8 JavaScript engine created by Google. What's so appealing about Node is that you can create C++ Addons that can be invoked through JavaScript running in Node, which basically means you don't have any of the limitations that you have running inside a browser (it is possible to access the filesystem, etc).

A lot of tools started showing up that were created using Node. Specifically for doing bundling the most popular ones were Grunt and Gulp.

In reality Grunt and Gulp are task runners, meaning that they run tasks, and bundling is just one of those possible tasks. Another example that also goes hand in hand with bundling is minification (or "uglification" outside the .Net world). It's the process of making the JavaScript as small as possible by renaming the variable and function names to single letters, and also removing all whitespace and comments.

Here's an example of how a gulp configuration file that creates a bundle looks like:

var gulp = require('gulp');
var concat = require('gulp-concat');

gulp.task('default', function(){
  gulp.src(['player.js', 'game.js'])
      .pipe(concat('bundle.js'))
      .pipe(gulp.dest("."));
});

When you run this task with gulp it creates a bundle with player.js and game.js (in that order) named bundle.js. If you are interested in learning Gulp I recommend: Automate Your Tasks Easily with Gulp.js.

Modules

Even though bundling solves the issue of the limited number of GET requests that browsers can perform simultaneously, it requires that the JavaScript files are added to the bundle in a particular order if they have dependencies on each other. It's also easy to end up in a situation where there is JavaScript code that never gets executed inside the bundle. Over time bundles become hard to manage.

JavaScript modules solve this problem. The idea behind using modules is that it is possible to have dependencies stated explicitly. For example imagine you are creating a JavaScript game and you have a game.js file. That file uses code from another file named player.js. We can explicitly say that game.js depends on player.js.

There are a few different module "formats". The most common ones are commonjs which is the one used in Node.js, there's also Asynchronous Module Definition (AMD)](https://github.com/amdjs/amdjs-api/wiki/AMD), and ES6 modules.

Let's imagine a simple scenario with game.js and player.js and describe them with these three module formats. Game has a start method that calls Player's getName method.

In all these module formats each JavaScript file is a module, so in this case we would have two modules, game and player.

CommonJS

With commonjs the player.js file would look like this:

var privateVar; //if this is not "exported" it won't be available outside player.js

function getName() {
  //...
}

module.exports.getName = getName;

And game.js:

var player = require('./player.js');

function start(){
  var playerName = player.getName();
  //...
}

It's through module.exports that we expose what's inside the module to whomever requests it. In this case the only thing that was "exported" was the getName function.

In commonjs to get the exported parts of another module we use the require function. You might have notice the ./ in the require statement in game.js. In this case it would mean that both files are in the same folder, however the way a module's file is found can become complicated. I'd recommend reading Node.js documentation on how to get to the exact filename when require is used.

Asynchronous Module Definition

The AMD syntax is a little bit different, it consists in using a define function where a module's dependencies are listed in an array, and then supplying a function where each of the arguments will be a dependency in the order they are listed in the array.

With AMD the player.js would look like this:

define([], function(){
  var privateVar; //not accessible outside the module

  function getName() {
    //...
  }
  return {
    getName: getName
  };
})

And game.js:

define(['./player'], function(player) {
  function start(){
    var playerName = player.getName();
    //...
  }
});

Here's a good resource to learn more about AMD.

ES6 Modules

The ECMAScript 6 standard which is the new specification for JavaScript (the new version of JavaScript if you will) introduced modules.

With ES6 modules the player.js file would look like this:

var privateVar;

function getName(){
  //...
}

export { getName };

And game.js would look like this:

import * as player from './player.js'

function start() {
  var playerName = player.getName();
  //...
}

Module Loaders

If you were to just load game.js or player.js as they are defined in the examples above they wouldn't work (you would get errors stating that require/define/import are not defined).

For them to work they need to be loaded through a module loader. A module loader is a JavaScript library that runs in the browser and which is capable of interpreting one (or several) module formats.

There are several popular module loaders. The most popular ones is probably SystemJS.

SystemJS supports several module formats. You can specify which one you are using through configuration options.

To use them you need to specify which module is the "entry point". You can think of the entry point as the main module, in our example that would be game.

Here's how we could use SystemJS to load the CommonJS example above:

<script src="system.js"></script>
<script>
  SystemJS.config({
    meta: {
      format: "cjs" //use commonjs module format
    }
  });

  SystemJS.import('game.js');
</script>

When you do this SystemJS will load game.js inspect it and realize that it needs to fetch player.js. It will then load the JavaScript from player.js and then game.js in the browser.

You can find a good introduction to JavaScript modules and module loaders in this pluralsight course: JavaScript Module Fundamentals.

JavaScript build process

Although client-side module loaders enable the use of modules, if there are a lot of them, we will again get into the issue of browsers having a limited number of GET request that can be performed simultaneously.

There's no reason not to do the module's loader "work" beforehand as a build step, and as a result produce a bundle. An example of a tool that does this is browserify.

Browserify gets its name from the idea of enabling the use of modules in the browser the same way they are used in Node.js. It's a "browserification" of Node.js modules (which use the commonjs format).

To create a bundle with browserify we just need to specify what is the main module. Browserify will figure out what other modules that module depends on, and which other modules those modules depend on and so on.

In our example we could create a bundle simply by doing this:

$ browserify game.js --outfile bundle.js

We then just need to include our bundle in our web page and we are good to go.

Transpilation

One thing that JavaScript is known for is being lax with regards to types. In JavaScript you don't need to specify which type a variable is, what is the return type of a function or what are the types of its parameters.

This made creating tools to aid the developers difficult. Some IDE's would provide some intellisense information (e.g. Visual Studio) but the experience was never perfect.

TypeScript is a language that is a superset of JavaScript and that allows for type information to be added.

To use TypeScript you need to compile it to JavaScript. This process of compiling a language to another language is what transpilation is.

Here's how a function definition with TypeScript looks like:

function getPlayer(id: number) : IPlayer {
  //...
}

Here we are saying that the getPlayer function expects a parameter named id that is a number and returns an IPlayer. In TypeScript you can define interfaces, for example IPlayer could be:

interface IPlayer {
  id: number;
  name: string;
}

When you compile this TypeScript code, the interface has no effect on the output, but during development type you get intellisense when you have an instance of IPlayer. Also, you will also get an error if you pass for example a string as an argument to getPlayer (e.g. getPlayer("abc")), you will also get intellisense in regards to the function parameters and their type, in this case you'd get intellisense for id of type number.

TypeScript was by no means the first language to come by that transpiles to JavaScript. The first that became really popular for a while was CoffeeScript, however (at least from my perception) it seems to be fading away.

Because it enables a better development experience, TypeScript is probably responsible for enabling ever more complex projects to be done in JavaScript. Also, because having build steps for JavaScript is so common now, having one more for transpilation adds very little friction.

Although TypeScript is probably the most popular language that transpiles to JavaScript, it should be mentioned that just writing ES6 code, the new version of JavaScript, is also very popular. Since not all features from ES6 are supported by current browsers, ES6 code is also transpilled to the current version of JavaScript. The tool that enables this is Babel.

Build tools on steroids

Imagine using JavaScript to load images or CSS instead of doing it in HTML. That's what build tools like Webpack enable.

If this is the first time you've heard about this you might be thinking about how this can be a good idea. It turns out that it enables scenarios that solve some common problems in web development. The same way we now have modules in JavaScript, we can apply the same solution to CSS where if we import CSS through JavaScript, that CSS might be scoped so that it does not interact with any other CSS in the page.

Images in CSS can automatically be converted to base64 and embedded inside the CSS itself if they are below a certain threshold size.

These are just some examples of what Webpack enables. If you spend some time becoming familiar with it you'll recognize that the new version of Angular relies heavily on this type of functionality.

Conclusion

In this post I tried to describe how I perceived JavaScript to evolve into what it is today. At the beginning JavaScript was a simple language, it sill is, but it didn't had this buzzing ecosystem around it. Most of that ecosystem was enabled by addressing problems that were a consequence of how JavaScript was being used. With the amount of shareable work that was done in Node.js and with ways to use it in a similar fashion in the browser (Browserify) the JavaScript ecosystem grew immensely. It continues to evolve with tools like Webpack that facilitate scenarios and practices that enable ever more complexity in a manageable way.

Rethinking email confirmation

Rui Figueiredo — Tue, 08 Aug 2017 22:27:09 +0000

We are very accustomed to how user registration works online. Enter your email, possibly pick a username, enter password, re-enter the password, get a confirmation email, click on a link in that email and voilÃ , account created.

However, there a are a few things that can go wrong.

Imagine this scenario. You want to create a new account with an hypothetical website named TheWebsite. You go through all the normal process but you mistype your email. Instead of john.smith@email.com you type jon.smith@email.com. But not all is bad, you typed your username correctly: johnsmith.

You wait for the confirmation email, but it doesn't arrive. It's 6pm, time to go home. You'll check it tomorrow, surely it will have arrived by then.

Tomorrow comes, and you forget that you didn't confirm your email, so you try to login, johnsmith + password. And it works, happy days.

Years pass, and you've been using TheWebsite happily. All is well until one day try to log in and you get an invalid login error. You try again, and again you get an error.

This can't be right you think. I always use johnsmith and the same password. You click the reset password link, but no reset email ever arrives.

This is a tragedy, all those pictures, and other stuff you've uploaded to TheWebsite. You can't access them anymore.

What happened?

Well, turns out that the person that actually owned the jon.smith@email.com was a kid that was curious and clicked the email from TheService asking him to verify his email address.

He himself forgot about this until a couple of years later when he heard about TheWebsite from some friends, and decided to try it. He tried to create an account and got an "account already exists" error. He used the password reset functionality and that's that. He now owns the original John Smith's account.

Although this story might seem convoluted it does happen. Brock Allen when talking about ASP.NET Identity Core in NDC described a situation where this had happened to someone from his family.

So what can be done?

A possible solution would be to ask for the password again when clicking the email confirmation link. This way Jon Smith wouldn't have been able to confirm John Smith's email, because he wouldn't know the password.

This introduces a new problem though. What about if between choosing a password an receiving the confirmation email you forget the password?

Well, then bye bye ability to create an account with that email. And goodbye cool username you managed to pick, won't be able to use those again.

Is there no hope?

There is. Here's a radical idea: don't let the user pick anything until s/he confirms the email address.

The process would be:

Click Sign Up
Enter email
Tell user to check email and click on the email confirmation link
User clicks on email confirmation link and is taken to a page where he enters
- username (if required, some websites only require an email)
- password
- retype password
Done

It's like saying I'll only talk to you when you confirm you own that email address. After that you can tell me what you want.

Trying to create an actual implementation

Last week I wrote a step by step guide on how to create a web application from scratch that is able to deal with user registration using email confirmation.

It was while researching ASP.NET Identity Core that I saw that NDC video with Brock Allen and learned about this problem.

I decided to try to implement this approach using ASP.NET Identity, but unfortunately couldn't get to a reliable implementation.

A scenario where the user picks an invalid password or a username that is already taken will leave the account for the user's email in limbo.

Here's a snippet that demonstrates the problem when using ASP.NET Identity:

[HttpPost]
public async Task<IActionResult> VerifyEmail(string id, string token, string username, string password, string repassword)
{
    var user = await _userManager.FindByIdAsync(id);

    var confirmEmailResult = await _userManager.ConfirmEmailAsync(user, token);
    if (!confirmEmailResult.Succeeded)
        throw new InvalidOperationException("Invalid token");      

    var addPasswordResult = await _userManager.AddPasswordAsync(user, password);
    if (!addPasswordResult.Succeeded)
    {
        addPasswordResult.Errors.ToList().ForEach(error => ModelState.AddModelError(string.Empty, error.Description));
        return View(); //goes back to the form where the user chooses the username and password
    } 
    //...

If the user mistypes the passwords the email gets confirmed but the user has no password set (it is possible to create users without password in ASP.NET Identity). The token for confirming the email cannot be used again, so the user won't ever be able to set the password.

One could think that we should only confirm the email token after setting the user's password, unfortunately that does not work because the token becomes invalid (won't match the user) if you change any of the user's properties.

Also, we can't validate the email and then redirect the user to another page to pick a username and password. That other page could be abused by another party to take over the account simply by navigating to it before the user did.

So it seems there's no reliable way of doing this with ASP.NET Identity, at least without creating custom email confirmation tokens that are still valid if we set the user's password.

The idea is still valid though, however it's hard not to feel that we are fighting the framework while trying to do it.

Brief(ish) explanation of how https works

Rui Figueiredo — Mon, 03 Jul 2017 11:22:24 +0000

When learning about how to use OpenSSL to create self-signed certs, it became clear to me that most of the information available online assumes you know what certificates are, and how they fit in HTTPS.

I decided to have a go at explaining HTTPS by explaining how communications are made secure, namely how Diffie-Hellman key exchange and digital certificates work. Here's my take on it.

HTTPS allows us to communicate securely over an insecure channel (for example over an open wifi network in an airport) where it is easy for someone to listen to all network traffic. The communications with HTTPS are secure because they are encrypted.

The communication is encrypted using symmetric key encryption. This particular type of encryption requires that both the sender and receiver have the same key (i.e. the same key is used to encrypt and decrypt the data).

But if the sender and the receiver need to send the key to each other through the insecure channel, doesn't this make this whole exercise pointless?

This can only work if the sender and receiver have a way of sending a secret key to each other even with someone listening to the communication (and without that someone being able to figure out what the secret key is).

This is where the Diffie Hellman key exchange comes into play.

Diffie-Hellman

Diffie-Hellman allows two parties to create a shared secret key without revealing it, while doing it "in the open", i.e. even if someone has access to all the information that the two parties exchange, they still won't be able to discover what the secret key is.

It sounds like magic, right? No, it's just math.

Here's a simple example of how it works. First two numbers are picked, they are usually called p and g. p is short for prime, g is short for generator.

A prime is a natural number greater than 1 that has no divisors other than 1 and itself.

A generator g for prime number p is a number such that g^x mod p for x between 1 and p-1 will "generate" all the numbers from 1 to p-1.

mod (short for modulo) is a function that returns the remainder of an exact division, for example 5 mod 3 is 2, because the exact division of 5 by 3 is 1 and the remainder of that is 2 (3 * 1 + 2 = 5).

Now that we know what mod is, here's an example of a generator for a prime. 3 is a generator for 7 because:

- 3^1 mod 7 = 3
- 3^2 mod 7 = 2
- 3^3 mod 7 = 6
- 3^4 mod 7 = 4
- 3^5 mod 7 = 5
- 3^6 mod 7 = 1

If we order the results we get: 1, 2, 3, 4, 5, 6, we can see that they are all the numbers from 1 to p-1, which makes g = 3 a generator for p = 7.

We know have all the pieces to perform Diffie-Hellman "by hand".

As an example, imagine Alice and Bob want to write letters to each other. Unfortunately someone is reading their correspondence, so they decide to use Diffie-Hellman to be able to communicate secretly.

Alice picks g and p, for example g = 3, and p = 7 and sends a letter with them to Bob.
Alice also picks a secret number less than p, for example 2.
Once Bob receives p and g, he picks a secret number less than p, for example 5.

Bob then computes this: g^secretBob mod p, in this case 3^5 mod 7 = 5. Bob sends this number to Alice, let's call this number numberBobSent.

At the same time Alice computes g^secretAlice mod p, in this case 3^2 mod 7 = 2 and sends it to Bob, let's call this number numberAliceSent.

Alice receives Bob's letter with number 5. She then computes numberBobSent^secretAlice mod p, in this case 5^2 mod 7 = 4.

Bob receives Alice's letter and does the same: numberAliceSent^secretBob mod p, in this case 2^5 mod 7 = 4.

Their shared secret is 4. Whoever read the letters could read p, g, numberAliceSent and numberBobSent. However, they don't know neither Alice nor Bob's secret numbers nor their shared secret. Alice and Bob can now use their shared secret to encrypt and decrypt their correspondence.

This works because for large primes it is prohibitively costly to figure out the shared secret with the information that an eavesdropper has access to (p, g, and numberBobSent and numberAliceSent).

Digital Certificates

We now know that web servers and browser clients can make use of Diffie-Hellman to establish a secret key to encrypt communications even when someone has access to all information that is exchanged.

This solves the problem of secure communications over an insecure medium, right? Well, not really.

Going back to Alice and Bob's example, imagine that whoever is reading their correspondence, let's say Eve, takes a more proactive role. Eve will grab Alice's letters and replace them with her own, and will do the same with Bob's.

She will be able to establish a secret key with both Alice and Bob. Alice will think she's corresponding with Bob, and Bob with Alice, while they are both corresponding with Eve.

She then just has to receive Alice's letters, decrypt them with the key she established with Alice, read them, encrypt them with the key she established with Bob and send them to him.

There's no way for Alice and Bob to know that their communication is not private.

This is what is called "A man in the middle attack".

So how can we solve this problem?

That's where digital certificates come into play, but before we need to talk about asymmetric key encryption.

Asymmetric Key Encryption

With asymmetric keys, there are two keys, a private and a public one. Both can encrypt and decrypt, but if you encrypt with one of them you can only decrypt with the other.

The private key is never supposed to be sent, while the public key can be freely sent to anyone. Because anything encrypted by the private key can only be decrypted with the public key, if someone sends you something encrypted and you can decrypt it with a public key, you know that only the holder of the private key could've encrypted what was sent.

A certificate is just a public key bundled together with information about the holder of the private key (e.g. the subject name which usually contains the website domain, e.g. blinkingcaret.com).

What makes a certificate trustworthy is that it is digitally signed by a certificate authority (an example would be Symantec, formerly Verisign). But first, what is a digital signature?

Digital Signature

The goal of a digital signature is to prove the identity of the sender of the information and that the information was not changed along the way.

A digital signature is performed in two steps. First a hash of what is to be signed is created (a hash function transforms an arbitrary amount of data into a fixed amount). Then it is encrypted using a private key. The encrypted hash is bundled together with the information. The encrypted hash is the signature.

When someone receives a piece of information with the encrypted hash they can create a hash of the information themselves (using the same type of hashing function, e.g. md5), use the public key to decrypt the signature and compare the two hashes. If they match it means that the information not only was not changed, it was also created by whomever the private key belongs to.

In case you are wondering why we don't just encrypt all the information instead of using a hash function, that's because the hash is much smaller and still has all the properties we need to guarantee that the information was not tampered with.

Certificate Authority

A certificate authority is an entity that signs certificates. To sign a certificate, a certificate authority needs a private key, and with that a public key, which also comes in a certificate.

The certificate authorities' certificates are special though. They are self-signed, which means they are signed with the private key that pairs with the public key in the certificate. Also, your computer's operating system came installed with a several certificate authorities' certificates.

When someone gives us a certificate, if that certificate was signed by a certificate authority we trust (any of the certificate authorities for which we have a certificate installed in our computer) we trust the certificate (that's when you get the green address bar your browser).

So, coming back to Alice and Bob's example, we should change the story here because the metaphor with the letters breaks a bit.

Imagine that Bob and Alice would deliver the letters to each other by hand.

Also, imagine that Alice and Bob had never met, they don't know how the other one looks like.

Although more complicated, a man in the middle attack is still possible in this scenario. Imagine someone, let's say Joe would make himself pass by Bob and meet Alice, and then someone else, for example Jane, would pass herself by Alice and meet Bob. John and Jane could coordinate between themselves so that even if Alice and Bob tried to use Diffie-Hellman they would end up talking to Jane and John without knowing it.

However, imagine now that Bob and Alice have id cards, issued by a government, and that are extremely difficult to forge.

Alice can ask Bob for his id card. The card has Bob's name and his picture, and is issued by a government that Alice knows is trustworthy, Alice is satisfied that Bob is whom he says he is. Bob can do the same in regards to Alice.

In HTTPS the id card is the certificate and the issuer is the certificate authority. Also, in HTTPS normally only the web server provides the certificate. However, although uncommon authentication of users with certificates is also possible.

I hope this gives you a better idea of how all these pieces fit together to enable HTTPS. Be sure to check my blog for more related content, for example how to create your own certificate authority, install it and create signed certificates using OpenSSL.

ORMs, Lazy Loading and Web Applications

Rui Figueiredo — Wed, 14 Jun 2017 16:30:41 +0000

What do these three things have in common?

They are frequently used together. Is that a good idea though? No, not at all, and that's what I'm going to convince you of in this blog post.

But first lets talk little bit about what an ORM is, what it's for, and what is lazy loading in the context of ORMs.

ORM is an acronym for "Object Relational Mapper". Two popular examples of ORMs are Entity Framework and Hibernate.

In relational databases (like Sql Server, MySql, etc) the data is represented using tables, columns and rows. There is also the concept of a constraint like primary and foreign keys, indexes and many other things that have no direct equivalent in an object oriented programming language.

Before ORMs were popular if you needed to read data from a database you would write a SQL query and that SQL code would live somewhere together with your source code.

ORMs changed that by providing us with a way to declaratively specify how the data in a database maps to object oriented constructs. For example, how a table maps to a class and how a column maps to property.

By having this intermediate layer it was now possible to write code that looks like it's only manipulating objects, but that under the covers, is converted to SQL queries that are transparently sent to the database.

This is great because it simplifies how we interact with data. Previously we'd have to write SQL statements, which effectively were strings in our code. Concatenating SQL with user input was very common, and that is basically what enables SQL injection. SQL injection happens when a user writes SQL statements instead of normal input in a textbox that end up being executed in your database.

So that is roughly what an ORM is about.

What about lazy loading?

To really understand lazy loading we need to talk a little bit more about a concept in ORMs called Context (or Session). Whenever data is loaded and converted to objects in memory by an ORM and these objects are stored in a Context.

Usually an interaction that involves fetching data from the database starts by creating a Context and then performing the operation that triggers the fetching of the data. Here's an example using Entity Framework that fetches all the Costumers from the database:

var myContext = new MyContext();
var allCostumers = myContext.Customers.ToList();
//...

After accessing the Customers property in the context and calling .ToList() on it (which is the operation that triggers the fetching of the data from the database), the customer data is stored in the context itself (for example in Entity Framework you would be able to access it in myContext.Customers.Local). This is so that if you make changes to a customer the ORM can figure out what changed and generate the appropriate SQL statements.

The context also has the ability to give you a slightly altered version of a Customer. And this is particularly relevant when Lazy Loading is enabled. Imagine the Customer table is represented in the database as having many orders. The corresponding Customer class could look like this:

public class Customer 
{
    public int CustomerId {get; set;}   

    //...

    public virtual ICollection<Order> Orders { get; set; }
}

Notice the virtual keyword in the Orders' collection. It just means that you can create a subclass of customer and override what that property does. And that is precisely what the context will do if you enable lazy loading.

With lazy loading enabled the context will keep track if that property was accessed or not. When it is accessed for the first time the context will transparently fetch the associated data (in this case the customer's orders).

This allows us to write code like this:

var myContext = new MyContext();
var johnDoe = myContext.Customers.Single(customer => customer.Name == "John Doe");

foreach(var order in johnDoe.Orders)
{
    //do something with John Doe's order
}

This all looks very well. It is easy to read and understand, however it could be made more efficient. However, it is not obvious why unless you are familiar with how the particular ORM that you are using works.

In this case this is Entity Framework where the data will be fetched on calling .Single(...) and when enumerating over the Orders. So in this example there are two database calls.

It is very easy to make this much worse and create what is called the N+1 problem. Here's an example:

var myContext = new MyContext();
var thisYearCustomers = myContext.Customers.Where(customer => customer.JoinYear == 2017);

foreach(var customer in thisYearsCustomers)
{
    foreach(var order in customer.Orders)
    {
        //do something with the customer's order
    }       
}

This will trigger a database request when the the customers are enumerated (foreach over customers) and then for each customer's orders. Hence the N+1, or 1+N if you prefer.

In case you're thinking this that lazy loading is terrible and never makes sense, in this situation you're probably right. A situation that leads to a N+1 problem is likely always a mistake. However that doesn't mean that lazy loading is never useful.

Imagine a desktop application where the context is created and lives through several user interactions. For example the user opens a customer screen, looks at it and then decides to look at that customer's orders.

In this scenario lazy loading is very convenient and makes sense.

When it does not is in a web application. That's because the context will not exist during more than one user interaction. It's just not possible.

In a web application the user's actions result in an HTTP request being sent from the user's browser to the server. The server then does all the required processing for that request and sends a response back to the user. And then this process repeats for every user action.

Between requests the server forgets about the user, so if a context is created in response to a user's action it will be gone after the response is sent. That is just the stateless nature of the web.

The only thing you can achieve using lazy loading in a web application is extra database calls you could avoid. That's because if user asks for the orders of a particular customer, the code that runs in the server will have to load the customer and the orders all during the handling of the user's request, it can either do it in a single database call or two. Lazy loading just makes it really easy to end up in the two database call scenario without realizing it.

For completeness I'll just mention what happens if you have lazy loading disabled in most ORMs. In the example above the Orders property would be null. You would have to instruct the ORM to fetch it together with the Customers all in one go. This is called eager loading:

var myContext = new MyContext();
var johnDoe = myContext.Customers.Include(customer => customer.Orders).Single(customer => customer.Name == "John Doe");

foreach(var order in johnDoe.Orders) //no extra db access
{
    //do something with John Doe's order
}

There's nothing to be gained by using lazy loading in web applications, however it's so much more common to see it being used in "the wild" than not. Even if we use the argument that it might be more convenient at times, the likelihood of having serious performance problems (like the N+1) offsets any possible benefits.