DEV Community: Mike Samuel

A web security story from 2008: silently securing JSON.parse

Mike Samuel — Thu, 06 Apr 2023 18:58:41 +0000

My 8 year old is doing a report for school on cyber security so I thought I'd dig up an old report for a break-the-web level security vulnerability that we silently fixed and which, as far as I know, has never been disclosed.

Back in 2008, JSON.parse was not part of the JavaScript language. It was a separate library downloaded from json.org that used JavaScript eval to unpack data.

// In the third stage we use the eval function to compile the text into a
// JavaScript structure. The "{" operator is subject to a syntactic ambiguity
// in JavaScript: it can begin a block or an object literal. We wrap the text
// in parens to eliminate the ambiguity.

                j = eval("(" + text + ")");

That code from a modern version of json2.js wraps the JSON source text in parentheses, because {} is a statement block in JavaScript but ({}) is an object constructor.

But a side-effect of using eval is that, if the source text is something like doVeryBadThings() then the JavaScript engine will happily do those very bad things, a classic arbitrary code execution vulnerability.

In computer security, arbitrary code execution (ACE) is an attacker's ability to run any commands or code of the attacker's choice on a target machine or in a target process.

Luckily, json2.js did a bunch of regular expression checks to make sure that text contained valid JSON, allowing commands that construct a value but not more powerful commands.

Unluckily, JSON is not a subset of JavaScript in the semantic sense. There are important differences.

To understand those differences, let's begin at the end with a change to the JavaScript language definition that I argued for based on this research.

Here's the changed specification text. It mirrors some explanatory text from Unicode §6.2.

7.1 Unicode Format-Control Characters
The Unicode format-control characters (i.e., the characters in category “Cf” in the Unicode Character Database such as LEFT-TO-RIGHT MARK or RIGHT-TO-LEFT MARK) are control codes used to control the formatting of a range of text in the absence of higher-level protocols for this (such as mark-up languages).
It is useful to allow these in source text to facilitate editing and display.
The format control characters maybe used in identifiers, within comments, and within string literals and regular expression literals.

And to the right of that is some deleted specification text.

7/2/2008 Deleted: anywhere in the source text of an ECMAScript program. These characters are removed from the source text before applying the lexical grammar. Since these characters are removed before processing string and regular expression literals, one must use a Unicode escape sequence (see 7.6) to include a Unicode format-control character inside a string or regular expression literal

JavaScript allows these control flow characters, known as [Cf], in identifiers because they're important for proper presentation of identifiers, especially those in cursive writing systems like عنصر in the Perso-Arabic alphabet. But they're a bit of a blind spot for many developers.

They were "removed from the source text before applying the lexical grammar." That means before the JavaScript parser has broken the source code into tokens, so

before it pairs quotation marks (") that start and end quoted string values, and
before it pairs delimiters like /* and */ or groups sequences like // and += that have no analogue in JSON.

JSON's specification does not have an equivalent clause. json.org simply says:

A string is a sequence of zero or more Unicode characters, wrapped in double quotes, using backslash escapes.

I realized that, by putting a [Cf] character between a backslash and a quote character I could get JavaScript's eval to find a different end of string than the JSON grammar would as expressed in the regular expressions that vet the JSON text. And that would allow me to sneak code into a JSON string that eval would execute.

I sent an email to the JSON maintainer, Douglas Crockford, with a proof of concept:

Transcript:

From Mike Samuel
to Douglas, Kevin

On firefox 2, the below alerts "hello world" after "created string about to parse" using a version of http://www.JSON.org/json2.js downloaded earlier today.

<html>
<head>
<script src=json2.js></script>
</head>
<body onload="
  var s = '&quot;\\\u200D\\&quot;, alert(\'hello world\') //&quot;\n';
  alert('created string about to parse');
  JSON.parse(s);
">

</body>
</html>

From Douglas Crockford
to me

I don't understand what is happening here.
Can you please explain the attack to me?

I didn't do a good job explaining it the first time though, but here is my second attempt:

The proof of concept was, where | stands in for Unicode's zero-width joiner character (U+200D):

"\|\", alert('hello world') //"

So a valid JSON parser would see just a single quoted string that contained two escaped characters, one of which was an escaped quote. And the regular expressions that json2.js used to approximate had the same interpretation.

So json2.js's security checks let that string through to JavaScript's eval.

But JavaScript's tokenizer saw something different because the control character is stripped out before tokenization:

"\\", alert('hello world') //"

There, the two backslashes (which previously had a [Cf] character between them) combine to form one escaped backslash.
The previously escaped double quote now ends the string. As the email explains (with bullets added for clarity):

Firefox sees ...

a string literal containing only a backslash followed by

a comma followed by

a call to alert followed by

a line comment.

(That line comment hides the final double quote, so that eval doesn't stop with a syntax error.)

Douglas realized he needed to change the regular expression used by json2.js.

So I think I have to do the same search for restricted characters that I do for ADsafe:
cx = /[\u0000-\u001f\u007f-\u009f\u00ad\u0600-\u0604\u070f\u17b4\u17b5\u200c-\u200f\u2028-\u202f\u2060-\u206f\ufeff\ufff0-\uffff]/,
Bother.

(Bother, indeed.)

But then he realized that we didn't want attackers to be able to reverse engineer the vulnerability from that targeted change, so he unilaterally changed the definition of JSON on the fly.

If I put cx in json.js and announce them[sic] problem, it will be giving a pretty clear signal to the miscreants about how to exploit but. But if instead, the fix is that I replace the regexp
/\\./
with one that only matches \ followed by a printable ASCII character, then the the text will be rejected. My fix will be less informative.

So that's how an arbitrary code execution that affected almost all of the JavaScript code using JSON in 2008 was silently closed with, afaik, no-one the wiser.

An alternative to "Distinguishing an Interpreter from a Compiler"

Mike Samuel — Fri, 24 Mar 2023 20:15:40 +0000

This is a response to Distinguishing an Interpreter from a Compiler in which I propose a capture-the-flag approach to distinguishing between compilers and interpreters.

any language can be implemented as an interpreter or a compiler

Laurence Tratt notes as he tries to define crisply what the difference is. And he quotes Mario Wolcko's observation.

For example, suppose the source program reads a number and then has a loop whose trip count is this number. The time it takes to compile this program will be independent of the number, in contrast to the time it takes to interpret the program.

This gets at the heart of the difference between static (considering all possible runs of a program) and dynamic (focused on one run of a program with known inputs) analysis.

Tratt defines compilation in terms of partially evaluating program elements; trading off work before runtime for work done at runtime. And this ahead of time work can be amortized over many program runs or a single run on a large input. This he boils this down to some big-O notation tests; crisp and mathematical.

I dislike tying compilation to performance though; I'm skeptical of focusing on one goal, because compilers, imho, can improve programs along multiple dimensions.

So I'm going to try to broaden this definition to account for other kinds of improvements which means I can't lean on the nice big-O related math and need a different technique.

But first, let's get out of the way things that won't help us put a toolchain on the compiled/interpreted spectrum.

Semantics should not differ between compiler and interpreter

When writing a compiler for a language that previously only had an interpreter available, one tries to preserve the semantics of programs.

So we can't rely on obvious differences in semantics. That said, many PLs have reflective affordances and underspecified behaviour that often will differ. I'll show some ways to abuse introspection and debugging to test my definition.

Performance is more an indicator of toolchain maturity, not whether compilation happened

In 2004, when Gmail launched, JavaScript engines were relatively immature. Gmail relied heavily on a JavaScript compiler to reduce the network bandwidth required to load what was then the single largest JavaScript program, and to make Gmail perform acceptably.

Today, un-pre-compiled JavaScript easily out-performs 2004's pre-compiled websites, because modern interpreters (albeit ones with just-in-time compilers) are much better than the JScript 5 engine used in IE6. Google and Mozilla invested an enormous amount of engineering in their engines.

So what does a good compiler do?

A well compiled program needs less to do the same job.

less time or
fewer system resources or
fewer instructions or
a smaller runtime or
less risk of leaking private information or
less of something else 🤷

and it does that based on considering what all possible runs of the program need and removing things they don't.

Maybe they don't need to perform that expensive operation inside a loop, and instead it can be done once outside.

Maybe they can replace that loop with a single vectorized instruction.

Maybe they don't need that function at all. It's in a library that other programs may use but not this one.

I propose that a tool-chain is compiler-y to the degree it lowers the requirements for a program to run.

A practical test for compilation

How does this awfully fuzzy definition let us distinguish between a program that is running after having been compiled, and one where the interpreter is considering instructions for the first time upon reaching them?

Capture the flag (CTF) contests are a long tradition in computer security:

an exercise in which "flags" are secretly hidden in purposefully-vulnerable programs

If we can embed flags in a program that, per the program's semantics, shouldn't be needed, and then skilled flag-capturers fail to find them, that's a strong signal that a compiler identified them as unnecessary and got to them first.

Below I work through some programs in various languages doing roughly the same for each:

I write a program that mentions a string value, "FLAG" but whose semantics do not depend on it.
I run the tool-chain for the program.
I try to find the value "FLAG" somewhere in an intermediate representation or at runtime in the program.

(Caveat: I'm not particularly good at capture the flag. I was always better at writing the challenges, but I know the basics, so please bear with me)

Capture the flag against a C compiler

#include <stdio.h>
#define PRINT_FLAG 0

int main(int argc, char** argv) {
  char* flag = "FLAG";
  if (PRINT_FLAG) {
    printf("%s\n", flag);
  }
  char *other = "OTHER";
  printf("%s\n", other);
  return 0;
}

We can see that the level of compilation, the -O flag to gcc matters.

$ gcc -O0 ctf.c
$ ./a.out
OTHER
$ strings -a ./a.out
FLAG
OTHER
$ gcc -O1 ctf.c
$ ./a.out
OTHER
$ strings -a ./a.out
OTHER

I compiled the above C program two ways and each time it had the same behaviour, but looking in the binary with strings showed that, when doing any kind of optimizations (-O1 or higher), the binary has the string "OTHER" which is needed, but does not have the string "FLAG" which is not.

By embedding a secret, and looking at the program from the outside, we can distinguish between a run of gcc -O0 that intentionally does no optimizations and one which does. (That may seem surprising. More on that below)

Capture the flag against a JavaScript engine with and without pre-compilation

We can use a debugger to get a sense of what is available. If a program is pre-compiled, the debugger simply won't have access to flags that were removed. For example, running the C progam above over GDB will simply skip over char* flag = "FLAG"; as if it never existed.

Here's our sample JS:

<script>
  function createClosure() {
      var flag = "FLAG";
      // Technically, this function closes over flag.
      return () => undefined;
  }

  var closure = createClosure();
</script>

It calls a function to create a closure that closes over var flag but which never uses it. Since this is in a <script> tag, the var closure is attached to the window object so this whole program isn't a no-op.

If I run this program through ClosureCompiler, a JS code minifier, I get

Note that the "compiled code" on the right-hand-side is similar but lacks comments and the line var flag = 'FLAG'.

Note: that this is not technically a semantics preserving transformation because it affects the semantics of calling createClosure.toString(); stringifying a JS function value dumps its source code. See the "hide source" directive proposal for more details. This requirement does not prevent a compiling JS engine from ignoring the instruction though.

If I load that same HTML uncompiled in Chrome's debugger, I can see that Chrome does not eliminate the unneeded flag variable.

The flag variable is available to the debugger which helpfully tells me that it's executing that line.

You can see after I step to the next line, the debugger shows next to the previous line that the value "FLAG" was bound to the name flag.

So V8 is not compiling out things that gcc -O1 does. V8 (Chrome's JS engine) does a lot of other just in time compilation, but not this particular one. It may be that V8 does fewer optimizations, specifically when you invoke the debugger, but that is still a choice to be less compiled when being interactively debugged. Quite possibly, V8 runs this interpreted because it is not a performance hotspot.

So our definition based on needing less also works when compilation is an optional step for a toolchain.

Capture the flag against Python

Is Python compiled? CPython spits out .pyc files for each .py fail it loads, so it seems like this is an example of compilation at load time.

import dis

def f():
    flag = 'INITIAL_VALUE'
    if False:
        flag = 'FLAG'
    print('%s' % flag)

dis.dis(f)

This program has a function that assigns a value to a local variable, then does not assign the 'FLAG' string to that variable, before printing the value stored in that variable.

The function is needed because another module could load this module and call the function so it can't be entirely eliminated.

And we can use Python's internal disassembler (module dis) to look at that function.

$ python3 ctf.py
  3           0 RESUME                   0

  4           2 LOAD_CONST               1 ('INITIAL_VALUE')
              4 STORE_FAST               0 (flag)

  5           6 NOP

  7           8 LOAD_GLOBAL              1 (NULL + print)
             20 LOAD_CONST               4 ('%s')
             22 LOAD_FAST                0 (flag)
             24 BINARY_OP                6 (%)
             28 PRECALL                  1
             32 CALL                     1
             42 POP_TOP
             44 LOAD_CONST               0 (None)
             46 RETURN_VALUE
$ python3 -m py_compile /tmp/ctf.py
$ strings /tmp/__pycache__/ctf.cpython-311.pyc
INITIAL_VALUEF
FLAGz
print)
flags
ctf.py
disr
<module>r

So I ran the program. Notice the 6 NOP instruction. That's where I would expect a LOAD CONST ... ('FLAG') if it had not compiled out the second assignment.
It looks like Python's internal compilation removes branches like if False:; python3 wins the capture the flag.

Then I asked python3 to generate a .pyc file with python -m py_compile. The strings in that file do contain FLAGz. PEP-3147 notes that with .pyc files, "the compilation phase can be bypassed." Perhaps it's the case that not all of the CPython compiler's transformations are reflected in the pyc file. I'm not sure whether that qualifies as a capture the flag win or not.

Capture the flag against Java

public class Ctf {
    public static void main(String... argv) {
        String flag = "INITIAL";
        if (false) {
            flag = "FLAG";
        }
        System.err.println(flag);
    }
}

Our code here is similar to our Python example: we initialize a variable, then because of if (false) we don't change it to "FLAG". Finally we print it, so that we need the initial value but not the one we want to capture.

If I compile and disassemble the .class file, I see that Java wins.

$ javac Ctf.java
$ javap -c -constants -p Ctf
Compiled from "Ctf.java"
public class Ctf {
  public Ctf();
    Code:
       0: aload_0
       1: invokespecial #1                  // Method java/lang/Object."<init>":()V
       4: return

  public static void main(java.lang.String...);
    Code:
       0: ldc           #7                  // String INITIAL
       2: astore_1
       3: getstatic     #9                  // Field java/lang/System.err:Ljava/io/PrintStream;
       6: aload_1
       7: invokevirtual #15                 // Method java/io/PrintStream.println:(Ljava/lang/String;)V
      10: return
}
$ strings Ctf.class
java/lang/Object
<init>
INITIAL
java/lang/System
Ljava/io/PrintStream;
java/io/PrintStream
println
(Ljava/lang/String;)V
Code
LineNumberTable
main
([Ljava/lang/String;)V
SourceFile
Ctf.java

The .class file does not contain instructions to fetch (ldc aka load constant) any second value from the class's constant pool, and strings shows that the constant pool contains "INITIAL" but not "FLAG".

The Java tool-chain wins the capture the flag, without even getting into its second just-in-time compilation phase.

Conclusions

A compiler is a needs reduction machine, a program transformation that allows a program to do the same with less; a tool-chain is compiler-y to the degree it lowers the requirements for a program to run.

This definition seems vaguer than Tratt's big-O notation based definition, but recognizes that some compilers aim to provide benefits other than runtime performance.

It may seem vague but it is testable. By understanding the goals of a compiler, we can adversarially test whether it achieves those goals.

To show that, I took one example of less, it doesn't need a string value that appears in the program source but which doesn't affect the program's semantics, and showed how to test whether something is or is not compiled by playing capture the flag against these toolchains:

gcc -O0, C in "just generate instructions no fancy stuff" mode
gcc -O1 through -O3, C with optimizations
JavaScript pre-compiled using a code minifier
JavaScript not pre-compiled
Python running in CPython3 with its load-time compilation and its cached source files
Java compiled via javac

And I outlined a number of techniques for testing whether a tool-chain compiles according to this definition:

Using the UNIX strings tool to look at tool-chain artifacts, like the a.out executable built by gcc
Using a debugger to see if instructions that initialize an unneeded variable are executed
Using disassemblers like javap and Python's in-program dis module to look for instructions and static value storage

The results above largely mirror programmer's understandings of their toolchains.

Java is compiled ahead of time, just not to a platform-specific instruction set.
Python compiles at module load time in a similar way.
JavaScript can be pre-compiled to simpler JavaScript if you so wish. Modern JavaScript engines do a lot of binary instruction generation internally, but it's not a tool-chain centred around compilation at the end of the day.

There are some surprises. gcc -O0 fails the capture-the-flag, so according to this definition is not compiled. This is a flaw. In this case gcc removes the need for a large runtime, so still falls into the "does the same with less" paradigm but this CTF exercise does not capture that.

Boolean coercion pitfalls (with examples)

Mike Samuel — Mon, 27 Feb 2023 19:50:02 +0000

It's not uncommon for programming language designers to let people write conditions where the type is not boolean, by automatically coercing non-boolean, but truthy values:

    ┏━━━━━━━━━ not boolean
    ▼
if (x) { 
    body(x);
}        ▲
         ┗━━━━ known to be usable here

This lets developers run body only if x is a kind of value that you can do things with.

It's tempting, as a language designer, when you see developers writing the same kinds of test conditions over and over to want to simplify things.

-if (pointer != null) { ... }
+if (pointer) { ... }
-if (!myList.empty) { ... }
+if (myList) { ... }
-if (result.isSuccess) { ... }
+if (result) { ... }
-if (inputText != "") { ... }
+if (inputText) { ... }
-if (count != 0) { ... }
+if (count) { ... }

All else being equal, a language that encourages shorter code is better.

But, the principle of least surprise says we ought craft semantics that have a clear meaning, even when developers are fuzzy on the precise semantics of a phrase.

Below are examples of different ways that silent conversion, specifically of if and loop condition results to boolean values, can violate that principle.

Special "nothing here" values like NULL are exactly the kind of things we might want to filter out. So maybe these two should be equivalent.

// For any pointer, p
if (p != NULL) { ... }
if (p)         { ... }

// In C++, for example, NULL is falsey, 
// and all other pointer values are truthy.

But it's easy to confuse containers with their content. Developers may be surprised when the container and content coerce to different truth values.

// What if we're pointing to a boolean.
bool b = false;
bool* p = &b;

if (p) {
  std::cout <<
    "a non-null pointer to false is truthy\n";
}

In languages that don't have null pointers, Option and Result types often serve a similar function: a box of zero or one value.

(* This is not actually valid OCaml
 * since OCaml doesn't do coercion. 
 *
 * But imagine Some _ is truthy and
 *             None   is falsey
 *)
let x = Some(false) in
if (x) then
  ...

Note that Swift allows its optional types in a condition but only via a binding condition which makes it clear that the container is what is being checked.

// Swift
let x: Bool? = false

// When x is not `nil`, unpacks it into b
if let b = x {
  print("b is \(b)")
}

// if x { ... }   // Does not type-check

Consider programs that work with data languages like JSON and YAML that use syntax typing; the textual value determines its type.
For example, YAML, a common configuration language, treats many un-quoted strings as booleans:

y|Y|yes|Yes|YES|n|N|no|No|NO
|true|True|TRUE|false|False|FALSE
|on|On|ON|off|Off|OFF

Developers who work with these languages are used to thinking of words as having truth values.

There's potential for confusion when a programming language assigns different boolean valence to a string value than the language of the string's content.

This happens in many programming languages: if (someContainer) executes the body if someContainer is not empty.

For example, Python says:

Any object can be tested for truth value, for use in an if or while condition, or …

By default, an object is considered true unless … Here are most of the built-in objects considered false:

…

empty sequences and collections: '', (), [], {}, set(), range(0)

So Python treats the string '' as falsey but others are truthy, including strings like 'false' and 'no'.

Here's some JavaScript, which has similar string truthiness.

// Unpack a value in a data language
// that the program does not control.
let request = JSON.parse(
  `
  {
    "deleteAccount": "no",
    "userConfirmationCheckbox": "off"
  }
  `
);
// But we forgot to convert those fields
// to booleans using the request language's
// conventions.
// So JavaScript's conventions prevail.
if (
    request.deleteAccount &&
    request.userConfirmationCheckbox
) {
  performIrreversibleDelete();
}

A programming language should support developers in checking assumptions about values that come from outside the program. Assigning arbitrary truthiness to string values makes it harder to find and check these assumptions.

It's easy to confuse a zero-argument function with its result.
When first-class function values have a boolean sense, this can lead to confusion.

# Python
class Animal:
    def is_goldfish_compatible(self):
        return True
class Goldfish(Animal): pass
class Dog(Animal): pass
class Cat(Animal):
    # overrides a predicate from SuperType
    def is_goldfish_compatible(self):
        return False

goldfish = Goldfish()
cat = Cat()
dog = Dog()

def play_together(a, b):
    print("%r plays with %r" % (a, b))

for animal in [dog, cat]:
    if animal.is_goldfish_compatible :
        #                           ▲
        # Pay attention here ━━━━━━━┛
        play_together(animal, goldfish)

It's easy to confuse the method call:

animal.is_goldfish_compatible()

with a read of a bound method:

animal.is_goldfish_compatible

Especially since other classes define similarly named is_* boolean attributes.

Unit and void are common names for special values produced by functions that are meant to be called for their side effect.

They make for clearer code. If a function's author doesn't intend to provide a value to the caller, they can simply not return anything.

These special values should probably not be silently treated as having a boolean valence, but in some languages they are.

This JavaScript is fine.

// A lambda that returns true
let shouldDoTheVeryImportantThing =
  () => true;

if (shouldDoTheVeryImportantThing()) {
  doIt();
}

That lambda returns true when called, but maybe I'm debugging the program, so I add some logging. I need a block to put a logging statement in, so I wrap it in {...}.

let shouldDoTheVeryImportantThing =
  () => {
    console.log("Called should...");
    true
  };

if (shouldDoTheVeryImportantThing()) {
  doIt();
}

When I added a block around the lambda body, I forgot to add a return before the true. Now it returns the special void-like value undefined, which coerces to false in a condition.

The second version logs, but silently fails to call doIt().

Automatic coercion results from a genuine desire by language designers to help developers craft more succinct and readable programs.

But when the semantics are not carefully tailored, this can lead to confusion.

Be especially careful around:

generic boxes and collections-of-one that might wrap booleans, including pointer types, Option and Result types,
catch-all types like string and byte[] that may layer semantics in another language which is at odds with the coerced semantics,
producers of values like first-class function values, thunks, and promises which might have a different boolean valence from the value they produce, and
special placeholder values like void or undefined which, when coerced to booleans silently, mask a failure to account for a missing value.

Before designing coercion semantics, maybe ask yourself, could a proposed coercion semantics mask missing steps that the author should have performed?

Might a different mechanism (like Swift's binding condition above) suffice?

Maybe you think that the rules you've chosen are obvious and that developers will have them clearly in mind when reading & writing conditions. "On the arbitrariness of truthi(ness)" explains how different PL designers, who probably each thought that their notions of truthiness were clear, came to very different decisions.

Thanks for reading and happy language designing.

Hygiene is not just for macros

Mike Samuel — Thu, 09 Feb 2023 19:13:11 +0000

JavaScript has no macro system but has some surprising variable scoping problems. Why?
What is macro hygiene and how is it a lens through which we can understand these problems and how to avoid them in future language designs?

Quick, what does this JavaScript do?

try {
  throw 'thrown';
} catch (e) {
  if (true) {
    var e = 'assigned in block';
  }
  console.log('In catch:', e);
}
console.log('Afterwards:', e);

We throw a string, which is caught and stored as e in catch (e).

Then, there's an if that declares another var e and initializes it to 'assigned in block'.

So what gets logged?

In catch: assigned in block
Afterwards: undefined

In JavaScript, var declarations are hoisted. Every var declaration is effectively pulled to the top of the containing function or module.

So the above is equivalent to

var e; // ◀━━━━━━━━━━━━━━━━━━━━━━━━━━┓
try {                        //      ┃
  throw 'thrown';            //      ┃
} catch (e) {                //      ┃
  if (true) {                //      ┃
    e = 'assigned in block'; // var ━┛
  }
  console.log('In catch:', e);
}
console.log('Afterwards:', e);

The var e is hoisted to the top.

Makes sense. var declarations are not block scoped; it doesn't only affect the {…} block that it appears in. (In JavaScript, if you want block scoping you use let or const declarations instead)

But notice also that the e = 'assigned in block' was left behind. Moving the initializer would cause problems; what if it was a complex expression, and in a less predictable if statement? We might execute code out of order or that we shouldn't have executed at all.

But because catch (e) introduces another variable e, the e = 'assigned in block' assigns a different variable than was intended.
Then when console.log('In catch', e) happens, instead of logging 'thrown', it logs the 'assigned in block' value.

Finally, since the var e was never actually assigned, the last line logs Afterwards: undefined.

Why did this happen?
Could it have been avoided?

The above program is not a good program, but it's actual meaning is likely to surprise even people who have made an effort to learn JavaScript's rules.

The problem here is that the JavaScript language treats names as text. This problem shows up often in languages with macro systems, ways of generating parts of the program by running code while a program is compiling or loading.

The Hygienic macro wikipedia page notes (emphasis added):

Hygienic macros are macros whose expansion is guaranteed not to cause the accidental capture of identifiers.

What happened above seems very similar. The identifier e in e = 'assigned in block' was accidentally captured by the declaration in catch (e).

A typical way of dealing with this in languages with macros is to have a name resolution step where textual identifiers are matched up with declarations to remove ambiguity. After that happens, it's safe to move uses of names around and introduce new declarations; you won't accidentally break the relationship between declarations and uses that are apparent in the source code.

For our code above, that renaming stage might add serial numbers to variable declarations.

try {
  throw 'thrown';
} catch (e_0) {
  if (true) {
    var e_1 = 'assigned in block';
  }
  console.log('In catch:', e);
}
console.log('Afterwards:', e);

Here we've replaced the different textual names like e with abstract names like e_0 to make them unambiguous.

Then we can hoist declarations to the appropriate place, taking care that, when we split var e_1 = ..., to use e_1 in both the original position and the hoisted position.

var e_1; // ◀━━━━━━━━━━━━━━━━━━━━━━━━━━┓
try {                          //      ┃
  throw 'thrown';              //      ┃
} catch (e_0) {                //      ┃
  if (true) {                  //      ┃
    e_1 = 'assigned in block'; // var ━┛
  }
  console.log('In catch:', e);
}
console.log('Afterwards:', e);

Now that our declarations are unambiguous, and in their final places, we can resolve references by matching the remaining textual names with declarations.

var e_1; // ◀━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
try {                            //     ┃
  throw 'thrown';                //     ┃
} catch (e_0) { // ◀━━━━━━━━━━━━━━━━━━┓ ┃
  if (true) {                    //   ┃ ┃
    e_1 = 'assigned in block';   //   ┃ ┃
  }                              //   ┃ ┃
  console.log('In catch:', e_0); // ◀━┛ ┃
}                                //     ┃
console.log('Afterwards:', e_1); // ◀━━━┛

This modified program outputs the below, which is probably closer what the original author^† meant and what a code reviewer would expect it to do.

In catch: thrown
Afterwards: assigned in block

† - if the author wasn't intentionally writing silly programs to make a point.

So why wasn't JavaScript designed that way?

Well, JavaScript was famously created in 10 days so there was not much time to think through var hoisting corner-cases.

But also, the language re-used certain abstractions. JavaScript Objects are bags of named properties that inherit names from a prototype. Objects are re-used to represent environment records, bags of named variables that inherit names from an outer record.

But is there anything about that re-use that prevents the kind of hygiene that came in handy above?

Let's consider a case where JavaScript blurs the distinction between environment records and Objects.

let o = { x: 'property of o' };
with (o) {
  var x = 'initial value for var x';
}
console.log('o.x after with:', o.x);

Some readers may not be familiar with JavaScript's deprecated with statement. It brings an object's properties into scope, so that instead of saying o.x, within with (o) you can just say x.

Here, we create an object with a property named x and we use with to bring its properties into scope. Inside that scope, we have a var declaration of the same textual name.

So what happens? If you're using JS engine that supports with (it runs in non-strict mode), then you get:

o.x after with: initial value for var x

To see why this happens, consider that program with the var x hoisted.

var x; // ◀━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
let o = { x: 'property of o' };  //      ┃
with (o) {                       //      ┃
  x = 'initial value for var x'; // var ━┛
}
console.log('o.x after when:', o.x);

Now, it's clear that x = ... assigns to o.x because of the with (o).

So this is another identifier capture problem; the with (o) captured the identifier x.

Could we fix this with our resolution phase?

It's definitely complicated by with. with is a dynamic construct. We don't know which names with brings into scope until we know which properties o has which we might not know until we run the program.

Consider a slightly more involved use of with.

let o = { /* initially empty object */ };
var x = 'global   x';
var y = 'global   y';
with (o) {
  // Repeatedly log x and y, add a property to o.
  // Some of these property names are the same as the
  // variables defined outside.
  for (let propertyName of ['x', 'y', 'z']) {
    console.log(`x=${x}, y=${y}, o=${JSON.stringify(o)}`);
    o[propertyName] = `property ${propertyName}`;
  }
}

That program outputs the below. Note that references to ${x} and ${y} depend on which properties we've added to o; how they resolve changes from one iteration of the loop to the next.

x=global   x, y=global   y, o={}
x=property x, y=global   y, o={"x":"property x"}
x=property x, y=property y, o={"x":"property x","y":"property y"}

Could JavaScript have been created with hygienic var and the with statement?

It turns out, yes.

Any time a free name appears in a with, you have to rewrite it to either lookup from the object or resolve.

So the program above is equivalent to the one below. But the program below has unambiguous names, and still exhibits dynamic🤔 scoping:

let o_0 = { /* initially empty object */ };
var x_1 = 'global   x';
var y_2 = 'global   y';
{ // with erased to a block
  const withed_object_3 = o_0;
  function readName_4(propertyName_5, readFree_6) {
    if (propertyName_5 in withed_object_3) {
      return withed_object_3[propertyName_5]
    } else {
      return readFree_6();
    }
  }
  for (let propertyName_4 of ['x', 'y', 'z']) {
    readName_4('console', () => globalThis.console)
      .log(`x=${
        readName_4('x', () => x_1)
      }, y=${
        readName_4('y', () => y_2)
      }, o=${
        readName_4('JSON', () => globalThis.JSON)
          .stringify(readName_4('o', () => o_0))
      }`);
    readName_4('o', () => o_0)[propertyName_4] =
      `property ${propertyName_4}`;
  }
}

This doesn't look pretty, but that's because we've moved with machinery from the JavaScript engine into the program via program rewriting. Lots of compilers and language runtimes do things like this; some short language constructs are expressed internally in terms of more verbose ones.

The important thing is that we've allowed even a program that makes use of very dynamic features to run in a hygienic manner.

We could have had consistently intuitive scoping in JavaScript if only we'd had a bit more time.

This matters because every JavaScript tool, from Babel to VSCode, needs to deal with these corner cases, or unintentionally change the meanings of programs, and because the JavaScript language designers have to think through these corner cases, and may have to reject otherwise excellent language changes that are affected by these.

Let's not make the same mistake in the next language.