DEV Community: Abdullah Ali

The Holey Array Problem

Abdullah Ali — Mon, 23 Jul 2018 20:42:39 +0000

One of the "features" of JavaScript I hate the most is "holey" arrays. If you're unsure what that is, consider the following:

const array = [1, 2, 3];

That is what is called a "packed" array. Elements are contiguous and the array is made up of one element type: number.

On the C++ side: when using V8 (aka Node.js), and under the hood, this array is actually stored as PACKED_SMI_ELEMENTS, which is a way to store small integers in memory, and arguably the most efficient out of the myriad of ways V8 will store your arrays.

Now consider this innocuous line of code:

array.push(3.14); // push a floating point number to the array.

On the C++ side: your array has just been transformed from PACKED_SMI_ELEMENTS (integers) to PACKED_DOUBLE_ELEMENTS(doubles) in memory. It became slightly different, but is still tightly packed and performant. This transformation is irreversible.

Nothing has changed on the JavaScript side.

Moving on to the next step:

array.push('Hello world!'); // push a string to the array

On the C++ side: your array has just been irreversibly transformed again. This time from PACKED_DOUBLE_ELEMENTS to PACKED_ELEMENTS. A PACKED_ELEMENTS array can hold any JavaScript value; but has to sacrifice much more memory space to represent itself compared to a SMI or Double array.

Now let us proceed to the next line of code:

console.log(array.length); // 5
array[9] = true;
console.log(array.length); // 10

This is allowed in JavaScript, right? You may assign to an arbitrary index in the array, and your array will be padded. So what happens on the C++ side?

On the C++ side: your array has been irreversibly transformed yet again, this time to HOLEY_ELEMENTS. Much, much slower to work with; and you've just made the V8's JIT (Just-In-time compiler) optimisations much harder, as it will be unable to optimise your program to a large extent.

It's worthy of note that calling new Array(n) or Array(n) will always create this type of array and slow down your code.

But why stop here? Let me introduce Satan's special data structure:

array[999] = 'HAIL SATAN! ♥'

On the C++ side: your array has just transformed from HOLEY_ELEMENTS to DICTIONARY_ELEMENTS, and you've summoned a demon that can no longer be banished.

Let me quote the V8 source code directly:
  // The "slow" kind.
  DICTIONARY_ELEMENTS,

From JavaScript's point of view: your array just became a dictionary, or in other words: a plain object. The literal worst case scenario of JavaScript arrays.

Why this is dangerous:

Such an operation will silently succeed and never throw an error.
Any form of loop-based enumeration or attempt at serialisation will most likely crash your server.
The array's keys will silently be converted to strings.
The array will still serialise to an array, not an object. (JSON.stringify will try padding all empty indices using nulls)
Array.isArray(array) will return true for DICTIONARY_ELEMENTS arrays.

If you try to call JSON.stringify on the array above, this is what you'll get:

[1,2,3,3.14,"Hello world!",null,null,null,null,true,null,null,null,null,null,null,null,null,null,null,null,null,null,...,null,null,null,null,"HAIL SATAN! ♥"]

How this can be used against you:

Consider the following example of a REST API using express to manipulate a todo list:

// Naïve example of holey array potential vulnerability

class Todos {

  constructor(username, items) {
    this.username = username;
    this.items = items || Todos.load(username);
  }

  // add a new todo
  add(todo) {
    this.items.push(todo);
    return this.items.length - 1;
  }

  // update an existing todo
  update(index, todo) {
    // index is expected to be an integer
    // we're making the mistake of accepting an arbitrary/unbounded index here though
    // this operation will succeed silently, and node won't throw any errors with a huge index.
    // e.g. if an attacker passes 10000000, the program won't crash or show signs of instability, the array will silently become "DICTIONARY_ELEMENTS".
    this.items[index] = todo;
    return index;
  }

  remove(index) {
    return this.items.splice(index, 1);
  }

  // another common case:
  // you're keeping a list of todos and want to give the user the ability to reorder items.
  swap(i1, i2) {
    const temp = this.items[i1];
    this.items[i1] = this.items[i2];
    this.items[i2] = temp;
  }

  // load a list of the user's previously saved todos
  // we’re not using a database for simplicity’s sake
  static load(username) {
    const userPath = path.join('data', this.username + '.json');
    if (fs.existsSync(userPath) {
      return JSON.parse(fs.readFileSync(userPath, 'utf8'));
    }
    return [];
  }

  // this saves the array back to disk as JSON when the request is ending
  // holey/dictionary arrays with absurd indices will pad empty ranges with `null`.
  // this could result a multi-gigabyte file if passed a holey/dictionary array with a big enough (sparse) index in them. Most likely we’ll run out of memory first because the resulting string will be too big.
  save() {
    fs.writeFileSync(path.join('data', this.username + '.json'), JSON.stringify(this.items));
  }

}

app.use((req, res, next) => {
  // initialise/load previous todos
  req.todos = req.todos || new Todos(req.session.username);
  next();
});

// add new todo
app.post('/todos/new', (req, res, next) => {
  if (req.body.payload)
    res.json({ index: req.todos.add(req.body.payload) });
  else
    res.status(500).json({ error: 'empty input' });
});

/// update existing todo (vulnerable to unbound indices!)
app.post('/todos/:idx/update', (req, res, next) => {
  if (req.body.payload)
    res.json(req.todos.update(parseInt(req.params.idx, 10), req.body.payload));
  else
    res.status(500).json({ error: 'empty input' });
});

…

// save current todo list after request
// a better idea is to override res.end() via a thunk though.
app.use((req, res, next) => {
  next();
  req.todos.save();
});

Here’s an example malicious request: POST /todos/10000000/update payload="hi"

You now have an invisible problem (10000000 element dictionary array) in memory, when the request ends it will try to write out a huge JSON file, or your server will run out of memory trying to serialise the array to a string.

For further reading on V8 internals:

https://v8project.blogspot.com/2017/09/elements-kinds-in-v8.html
https://v8project.blogspot.com/2017/08/fast-properties.html

Introducing Nexus.js: A multi-threaded JavaScript run-time

Abdullah Ali — Sat, 09 Dec 2017 08:44:25 +0000

First of all, I recommend reading the old series if you're not familiar with this project. If you don't want to read all of that, no worries! I got you covered!

Now that that's out of the way, let us begin.

Last year I started implementing Nexus.js, a multi-threaded server-side JavaScript run-time based on WebKit/JavaScriptCore. I dropped it for a while, due to circumstances out of my control that I'm not going to discuss here, but here we are again: I couldn't keep myself from working on it for long.

So let us start by discussing the architecture of Nexus and how things work:

Nexus.js Runtime

Event Loop

There is no event loop.
There is a thread-pool with a (lock-free) task queue.
Every time you call setTimeout or setImmediate or create a promise, a task is queued to the task queue.
Every time a task is scheduled, the first available thread will pick the task and execute it.
Promises resolve on all CPU cores. A call to Promise.all() will resolve the promises in parallel.

ES6

async/await is supported, and encouraged.
for await(...) is supported.
Destructuring is supported.
async try/catch/finally is supported.

Modules

CommonJS is not supported. (require(...) and module.exports)
All modules use the ES6 import/export syntax.
Dynamic importing is possible through import('file-or-package').then(...).
import.meta is supported. For example: import.meta.filename and import.meta.dirname to name a few.
Bonus feature: you can import URLs directly from the source code.

Example:

import { h } from 'https://unpkg.com/preact/dist/preact.esm.js';

EventEmitter

Nexus implements a promise-based EventEmitter class.
Event handlers will be queued on all threads, and will execute in parallel.
The result of EventEmitter.emit(...) is a promise that will resolve to an array of all values returned by the event handlers.

Example:

class EmitterTest extends Nexus.EventEmitter {
  constructor() {
    super();
    for(let i = 0; i < 4; i++)
      this.on('test', value => { console.log(`fired test ${i}!`); console.inspect(value); });
    for(let i = 0; i < 4; i++)
      this.on('returns-a-value', v => `${v + i}`);
  }
}

const test = new EmitterTest();

async function start() {
  await test.emit('test', { payload: 'test 1' });
  console.log('first test done!');
  await test.emit('test', { payload: 'test 2' });
  console.log('second test done!');
  const values = await test.emit('returns-a-value', 10);
  console.log('third test done, returned values are:'); console.inspect(values);
}

start().catch(console.error);

I/O

All input/output is done through three primitives: the Device, the Filter and the Stream.
All input/output primitives implement the EventEmitter class.
To use a device, you construct a ReadableStream or WritableStream on top of it.
To manipulate data, you add filters to your ReadableStream or WritableStream.
Lastly, you use source.pipe(...destinationStreams) and then await source.resume() to process the data.
All input/output operations are done using ArrayBuffer objects.
Filters implement process(buffer) to handle the data.

Example: (Converts UTF8 to UTF16 with 4 separate outputs files)

async function start() {
  const startTime = Date.now();
  try {
    const device = new Nexus.IO.FilePushDevice('enwik8');
    const stream = new Nexus.IO.ReadableStream(device);

    stream.pushFilter(new Nexus.IO.EncodingConversionFilter("UTF-8", "UTF-16LE"));

    const wstreams = [0,1,2,3]
      .map(i => new Nexus.IO.WritableStream(new Nexus.IO.FileSinkDevice('enwik16-' + i)));

    console.log('piping...');

    stream.pipe(...wstreams);

    console.log('streaming...');

    await stream.resume();

    await stream.close();

    await Promise.all(wstreams.map(stream => stream.close()));

    console.log(`finished in ${(Date.now() * startTime) / 1000} seconds!`);
  } catch (e) {
    console.error('An error occurred: ', e);
  }
}

start().catch(console.error);

TCP/UDP

Nexus.js provides an Acceptor class, responsible for binding addresses/ports and listening for connections.
Every time a connection is received, the connection event is fired and provided with a Socket device.
Each Socket instance is a bidirectional I/O device.
You may use ReadableStream and WritableStream to manipulate the Socket.

Very basic example: (Writes 'Hello world!' to clients)

const acceptor = new Nexus.Net.TCP.Acceptor();
let count = 0;

acceptor.on('connection', (socket, endpoint) => {
  const connId = count++;
  console.log(`connection #${connId} from ${endpoint.address}:${endpoint.port}`);
  const rstream = new Nexus.IO.ReadableStream(socket);
  const wstream = new Nexus.IO.WritableStream(socket);
  const buffer = new Uint8Array(13);
  const message = 'Hello World!\n';
  for(let i = 0; i < 13; i++)
    buffer[i] = message.charCodeAt(i);
  rstream.pushFilter(new Nexus.IO.UTF8StringFilter());
  rstream.on('data', buffer => console.log(`got message: ${buffer}`));
  rstream.resume().catch(e => console.log(`client #${connId} at ${endpoint.address}:${endpoint.port} disconnected!`));
  console.log(`sending greeting to #${connId}!`);
  wstream.write(buffer);
});

acceptor.bind('127.0.0.1', 10000);
acceptor.listen();

console.log('server ready');

HTTP

Nexus provides a Nexus.Net.HTTP.Server class that essentially inherits TCPAcceptor.
Same basic interface.
When the server finishes parsing/validating the basic HTTP headers of an incoming connection, the connection event is fired with the connection and peer information.
Every Connection instance has a request and a response. Those are input/output devices.
You may construct ReadableStream and WritableStream to manipulate the request/response.
If you pipe to a Response object, the streams enter chunked-encoding mode. Otherwise, you may use response.write() to write a regular string payload.

Complex example: (Baseline HTTP server with Chunked encoding, details omitted)


....


/**
 * Creates an input stream from a path.
 * @param path
 * @returns {Promise<ReadableStream>}
 */
async function createInputStream(path) {
  if (path.startsWith('/')) // If it starts with '/', omit it.
    path = path.substr(1);
  if (path.startsWith('.')) // If it starts with '.', reject it.
    throw new NotFoundError(path);
  if (path === '/' || !path) // If it's empty, set to index.html.
    path = 'index.html';
  /**
   * `import.meta.dirname` and `import.meta.filename` replace the old CommonJS `__dirname` and `__filename`.
   */
  const filePath = Nexus.FileSystem.join(import.meta.dirname, 'server_root', path);
  try {
    // Stat the target path.
    const {type} = await Nexus.FileSystem.stat(filePath);
    if (type === Nexus.FileSystem.FileType.Directory) // If it's a directory, return its 'index.html'
      return createInputStream(Nexus.FileSystem.join(filePath, 'index.html'));
    else if (type === Nexus.FileSystem.FileType.Unknown || type === Nexus.FileSystem.FileType.NotFound)
      // If it's not found, throw NotFound.
      throw new NotFoundError(path);
  } catch(e) {
    if (e.code)
      throw e;
    throw new NotFoundError(path);
  }
  try {
    // First, we create a device.
    const fileDevice = new Nexus.IO.FilePushDevice(filePath);
    // Then we return a new ReadableStream created using our source device.
    return new Nexus.IO.ReadableStream(fileDevice);
  } catch(e) {
    throw new InternalServerError(e.message);
  }
}

/**
 * Connections counter.
 */
let connections = 0;

/**
 * Create a new HTTP server.
 * @type {Nexus.Net.HTTP.Server}
 */
const server = new Nexus.Net.HTTP.Server();

// A server error means an error occurred while the server was listening to connections.
// We can mostly ignore such errors, we display them anyway.
server.on('error', e => {
  console.error(FgRed + Bright + 'Server Error: ' + e.message + '\n' + e.stack, Reset);
});

/**
 * Listen to connections.
 */
server.on('connection', async (connection, peer) => {
  // Start with a connection ID of 0, increment with every new connection.
  const connId = connections++;
  // Record the start time for this connection.
  const startTime = Date.now();
  // Destructuring is supported, why not use it?
  const { request, response } = connection;
  // Parse the URL parts.
  const { path } = parseURL(request.url);
  // Here we'll store any errors that occur during the connection.
  const errors = [];
  // inStream is our ReadableStream file source, outStream is our response (device) wrapped in a WritableStream.
  let inStream, outStream;
  try {
    // Log the request.
    console.log(`> #${FgCyan + connId + Reset} ${Bright + peer.address}:${peer.port + Reset} ${
      FgGreen + request.method + Reset} "${FgYellow}${path}${Reset}"`, Reset);
    // Set the 'Server' header.
    response.set('Server', `nexus.js/0.1.1`);
    // Create our input stream.
    inStream = await createInputStream(path);
    // Create our output stream.
    outStream = new Nexus.IO.WritableStream(response);
    // Hook all `error` events, add any errors to our `errors` array.
    inStream.on('error', e => { errors.push(e); });
    request.on('error', e => { errors.push(e); });
    response.on('error', e => { errors.push(e); });
    outStream.on('error', e => { errors.push(e); });
    // Set content type and request status.
    response
      .set('Content-Type', mimeType(path))
      .status(200);
    // Hook input to output(s).
    const disconnect = inStream.pipe(outStream);
    try {
      // Resume our file stream, this causes the stream to switch to HTTP chunked encoding.
      // This will return a promise that will only resolve after the last byte (HTTP chunk) is written.
      await inStream.resume();
    } catch (e) {
      // Capture any errors that happen during the streaming.
      errors.push(e);
    }
    // Disconnect all the callbacks created by `.pipe()`.
    return disconnect();
  } catch(e) {
    // If an error occurred, push it to the array.
    errors.push(e);
    // Set the content type, status, and write a basic message.
    response
      .set('Content-Type', 'text/plain')
      .status(e.code || 500)
      .send(e.message || 'An error has occurred.');
  } finally {
    // Close the streams manually. This is important because we may run out of file handles otherwise.
    if (inStream)
      await inStream.close();
    if (outStream)
      await outStream.close();
    // Close the connection, has no real effect with keep-alive connections.
    await connection.close();
    // Grab the response's status.
    let status = response.status();
    // Determine what colour to output to the terminal.
    const statusColors = {
      '200': Bright + FgGreen, // Green for 200 (OK),
      '404': Bright + FgYellow, // Yellow for 404 (Not Found)
      '500': Bright + FgRed // Red for 500 (Internal Server Error)
    };
    let statusColor = statusColors[status];
    if (statusColor)
      status = statusColor + status + Reset;
    // Log the connection (and time to complete) to the console.
    console.log(`< #${FgCyan + connId + Reset} ${Bright + peer.address}:${peer.port + Reset} ${
      FgGreen + request.method + Reset} "${FgYellow}${path}${Reset}" ${status} ${(Date.now() * startTime)}ms` +
      (errors.length ? " " + FgRed + Bright + errors.map(error => error.message).join(', ') + Reset : Reset));
  }
});

/**
 * IP and port to listen on.
 */
const ip = '0.0.0.0', port = 3000;
/**
 * Whether or not to set the `reuse` flag. (optional, default=false)
 */
const portReuse = true;
/**
 * Maximum allowed concurrent connections. Default is 128 on my system. (optional, system specific)
 * @type {number}
 */
const maxConcurrentConnections = 1000;
/**
 * Bind the selected address and port.
 */
server.bind(ip, port, portReuse);
/**
 * Start listening to requests.
 */
server.listen(maxConcurrentConnections);
/**
 * Happy streaming!
 */
console.log(FgGreen + `Nexus.js HTTP server listening at ${ip}:${port}` + Reset);

Benchmark

I think I've covered almost everything I've implemented so far. So now, let's talk about performance.

Here's the current benchmark for the above HTTP server, with 100 concurrent connections and a total of 10,000 requests:

This is ApacheBench, Version 2.3 <$Revision: 1796539 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient).....done


Server Software:        nexus.js/0.1.1
Server Hostname:        localhost
Server Port:            3000

Document Path:          /
Document Length:        8673 bytes

Concurrency Level:      100
Time taken for tests:   9.991 seconds
Complete requests:      10000
Failed requests:        0
Total transferred:      87880000 bytes
HTML transferred:       86730000 bytes
Requests per second:    1000.94 [#/sec] (mean)
Time per request:       99.906 [ms] (mean)
Time per request:       0.999 [ms] (mean, across all concurrent requests)
Transfer rate:          8590.14 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:     6   99  36.6     84     464
Waiting:        5   99  36.4     84     463
Total:          6  100  36.6     84     464

Percentage of the requests served within a certain time (ms)
  50%     84
  66%     97
  75%    105
  80%    112
  90%    134
  95%    188
  98%    233
  99%    238
 100%    464 (longest request)

That's 1,000 requests per second! On an old i7 that's running both the benchmark software, an IDE consuming 5GB of RAM, and the server itself!

voodooattack@voodooattack:~$ cat /proc/cpuinfo 
processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 60
model name  : Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
stepping    : 3
microcode   : 0x22
cpu MHz     : 3392.093
cache size  : 8192 KB
physical id : 0
siblings    : 8
core id     : 0
cpu cores   : 4
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts
bugs        :
bogomips    : 6784.18
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:

And here are the results in graph form:

I tried testing with 1,000 concurrent requests, but ApacheBench times out due to that many sockets being open. I tried httperf and here are the results:

voodooattack@voodooattack:~$ httperf --port=3000 --num-conns=10000 --rate=1000
httperf --client=0/1 --server=localhost --port=3000 --uri=/ --rate=1000 --send-buffer=4096 --recv-buffer=16384 --num-conns=10000 --num-calls=1
httperf: warning: open file limit > FD_SETSIZE; limiting max. # of open files to FD_SETSIZE
Maximum connect burst length: 262

Total: connections 9779 requests 9779 replies 9779 test-duration 10.029 s

Connection rate: 975.1 conn/s (1.0 ms/conn, <=1022 concurrent connections)
Connection time [ms]: min 0.5 avg 337.9 max 7191.8 median 79.5 stddev 848.1
Connection time [ms]: connect 207.3
Connection length [replies/conn]: 1.000

Request rate: 975.1 req/s (1.0 ms/req)
Request size [B]: 62.0

Reply rate [replies/s]: min 903.5 avg 974.6 max 1045.7 stddev 100.5 (2 samples)
Reply time [ms]: response 129.5 transfer 1.1
Reply size [B]: header 89.0 content 8660.0 footer 2.0 (total 8751.0)
Reply status: 1xx=0 2xx=9779 3xx=0 4xx=0 5xx=0

CPU time [s]: user 0.35 system 9.67 (user 3.5% system 96.4% total 99.9%)
Net I/O: 8389.9 KB/s (68.7*10^6 bps)

Errors: total 221 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 221 addrunavail 0 ftab-full 0 other 0

As you can see, it still works; albeit with some connections timing out due to the stress. I'm still working on figuring out what's causing this issue.

The source code to the project is available at GitHub, feel free to check it out.

Happy hacking, and until next time!

Edit:

Here is a pre-alpha release in case you wish to do your own benchmarking. Please note that this was only tested on two machines, both were running Ubuntu 17.10:

https://github.com/voodooattack/nexusjs/releases/tag/4dd3419

CloudPress — Part 2: CMX, the user-friendly variant of JSX!

Abdullah Ali — Wed, 15 Nov 2017 07:17:50 +0000

In my previous article, I talked about CloudPress, a new Content Management System that I’ve been working on for the past year. I talked about the plugin architecture and how the system works. Today, I’ll be introducing you to a couple of new technologies that I’ve implemented in the past week.

CMX: User-friendly standardised markup, for a better future

CMX is a spin-off of JSX. Their syntax is essentially the same, with a single key difference: in CMX, values are evaluated as JSON data; and not JavaScript expressions.

An example CMX page would look like this:

<Document>
  <div className="container">
    <style dangerouslySetInnerHTML={{ __html: `
      /* multi-line styles, CMX supports template strings! */
      body { background-color: #eee; }
    ` }} />
    <img src="img/logo.png" style={{ border: '1px solid' }} />
    {/* this is a comment*/}
    <CustomComponent cssBreakpoints={[320, 768, 1224]} trueProp customConfig={{
        testProp: 10,
        object: { string: "test" }
      }}></CustomComponent>
  </div>
</Document>

Which would output the HTML below:

<span><div class="container"><style>
  /* multi-line styles, CMX supports template strings! */
  body { background-color: #eee; }
</style><img src="img/logo.png" style="border:1px solid"/></div></span>

If you’re not familiar with JSX, you might be wondering, Document and CustomComponent are not standard HTML elements. So what are they?

And my answer would be that they are custom Components. They are React components that are registered with the system by plugins. They act exactly like shortcodes in WordPress. Although they fit in with HTML more naturally in my opinion.

Components have access to the system through GraphQL, and they can query data or perform actions by calling mutations. This gives plugin developers a very powerful tool for interacting with the system. They are also isomorphic and should render on the server exactly as they render in the browser.

One more notable (albeit perhaps too technical) feature of CMX is that it allows regular expression literals to be passed along in props.

GraphQLDatabaseLoader

Another technology I’ve finished implementing yesterday is GraphQLDatabaseLoader, which is a caching loader built on top of TypeORM that folds a set of different database queries into a singular query.

If you look at Facebook’s data-loader library, you’ll see a glaring problem: it’s too generic. As you see in the example below:

const userLoader = new DataLoader(keys => myBatchGetUsers(keys));
userLoader.load(1)
  .then(user => userLoader.load(user.invitedByID))
  .then(invitedBy => console.log(`User 1 was invited by ${invitedBy}`));

It can only load items by IDs, which is well and good, but it limits us severely in our use case.

For example: if you use it with GraphQL and a SQL database (which is a situation that many will find themselves in), it does not allow you to optimise your queries as much as you’d normally be able to. (Using something like join monster for example)

Using the the GraphQLResolveInfo parameter provided to the GraphQL resolvers, one could use it to query precisely the required data. Nothing more, and nothing less. A SELECT statement could select precisely what was queried and return it. Yet with Facebook’s DataLoader, you simply can’t make use of that information, because of caching (think, what would happen if an incoming request wanted a field and it was not present in the cache) and a thousand other reasons.

Another limitation is this: what if the query wanted to lookup something by a field other than the ID? A slug perhaps?

It wasn’t acceptable to me, and I had to stop and think hard on this subject, before I implemented my own solution.

GraphQLDatabaseLoader is database and GraphQL-aware. It will fold all database requests received from all sources (think: GraphQL resolvers, koa middleware, whatever) during a single event loop cycle into a single database request, and cache the results on top of that.

For example, with a query such as this one:

query {
  user1: node(id: "VXNlcjox") {
    __typename
    id
    ...UserFragment
  }
  hello_world: node(id: "UGFnZTox") {
    __typename
    id
    ...PageFragment
  }
  test2: lookupPageBySlug(slug: "test2") {
    __typename
    id
    content
    ...PageFragment
  }
}
fragment PageFragment on Page {
  title
  excerpt
  slug
  author {
   name
  }
}
fragment UserFragment on User {
  name
  username
  email
}

The 3 different GraphQL queries will result in exactly 1 database query, one which will SELECT exactly the required fields:

SELECT
        "Page0"."id" AS "Page0_id",
        "Page0"."title" AS "Page0_title",
        "Page0"."slug" AS "Page0_slug",
        "Page0"."content" AS "Page0_content",
        "Page0"."excerpt" AS "Page0_excerpt",
        "Page0_author"."name" AS "Page0_author_name",
        "User1"."id" AS "User1_id",
        "User1"."name" AS "User1_name",
        "User1"."username" AS "User1_username",
        "User1"."email" AS "User1_email",
        "Page2"."id" AS "Page2_id",
        "Page2"."title" AS "Page2_title",
        "Page2"."slug" AS "Page2_slug",
        "Page2"."excerpt" AS "Page2_excerpt",
        "Page2_author"."name" AS "Page2_author_name",
        "74d5c2aed587be81c9d67117dc60afd8" AS "Page0_KEY",
        "bdeac7ffad7e49ac60b1ab6c123e4f85" AS "User1_KEY",
        "d81c9566475e497a46b39e00d0826e3c" AS "Page2_KEY" 
    FROM
        "page" "Page",
        "user" "User",
        "page" "Page" 
    LEFT JOIN
        "page" "Page0" 
            ON (
                "Page0"."slug"=$1
            )  
    LEFT JOIN
        "user" "Page0_author" 
            ON "Page0_author"."id"="Page0"."authorId"  
    LEFT JOIN
        "user" "User1" 
            ON (
                "User1"."id"=$2
            )  
    LEFT JOIN
        "page" "Page2" 
            ON (
                "Page2"."id"=$3
            )  
    LEFT JOIN
        "user" "Page2_author" 
            ON "Page2_author"."id"="Page2"."authorId"

And return the results:

{
  "data": {
    "user1": {
      "__typename": "User",
      "id": "VXNlcjox",
      "name": "Abdullah",
      "username": "voodooattack",
      "email": "voodooattack@hotmail.com"
    },
    "hello_world": {
      "__typename": "Page",
      "id": "UGFnZTox",
      "title": "Welcome to CloudPress!",
      "excerpt": "test",
      "slug": "hello-world",
      "author": {
        "name": "Abdullah"
      }
    },
    "test2": {
      "__typename": "Page",
      "id": "UGFnZToy",
      "content": "<Document>\n  <div className=\"container\">\n    <style dangerouslySetInnerHTML={{ __html: `\n      /* multi-line styles, CMX supports template strings! */\n      body { background-color: #eee; }\n    ` }} />\n    <img src=\"img/logo.png\" style={{ border: '1px solid' }} />\n    {/* this is a comment*/}\n    <CustomComponent cssBreakpoints={[320, 768, 1224]} trueProp customConfig={{\n        testProp: 10,\n        object: { string: \"test\" }\n      }}></CustomComponent>\n  </div>\n</Document>",
      "title": "test 2",
      "excerpt": "",
      "slug": "test2",
      "author": {
        "name": "Abdullah"
      }
    }
  }
}

The part I want you to notice is this part of the query:

"74d5c2aed587be81c9d67117dc60afd8" AS "Page0_KEY",
"bdeac7ffad7e49ac60b1ab6c123e4f85" AS "User1_KEY",
"d81c9566475e497a46b39e00d0826e3c" AS "Page2_KEY"

Those are the hashes used for cache-busting. Each query is hashed and assigned a key in the loader’s cache, like so:

/**
 * Load a model from the database.
 * @param where Query conditions.
 * @param {GraphQLResolveInfo} info GraphQL resolver information argument.
 * @param {IModelInfo} modelInfo The model type to load.
 * @returns {Promise<T>}
 */
async load<T>(where: any, info: GraphQLResolveInfo, modelInfo: IModelInfo): Promise<T|undefined> {
  const fields = graphqlFields(info);
  const hash = crypto.createHash('md5');
  const key = hash.update(JSON.stringify({ where, fields })).digest().toString('hex');
  if (key in this._cache)
    return this._cache[key];
  ...

If the query hash is found in the cache table, the cached value is returned.

And I almost forgot to mention that each HTTP request gets its own GraphQLDatabaseLoader, so no collisions or leaks occur between user sessions.
That is all for now!

Edit: This is now available as a stand-alone package on npm: typeorm-loader

In this series, I’ll hopefully discuss more of the technical aspects of the project and the challenges I face. I’ll also try and post regular updates, future plans, and repeatedly and shamelessly beseech people to contribute to the project.
If you’re interested in contributing (I really could use the help), don’t hesitate to contact me here or on Twitter.

Until next time!

CloudPress — Part 1: How to reinvent a better wheel!

Abdullah Ali — Wed, 15 Nov 2017 07:01:54 +0000

Foreword:

I've decided to move my articles from Medium to dev.to. I just discovered this platform and immediately fell in love with it. I believe that this is a better medium (pun intended) to publish articles in the tech industry. (It's also not blocked in my country, so there is that burden off my shoulder.)

You can find the old article here.

CloudPress is a new content management system that I’ve been working on intermittently for the past year or so. It went through multiple evolutions so far, the biggest of which was the migration from plain JavaScript to TypeScript, which took about a month considering that — as it stands — the project is approximately 56,000 lines of TypeScript code. (Not counting comments and miscellaneous files.)

You might be saying “Hey, look, another foolish attempt at a CMS, let’s see how long this one lasts.” Or you might be asking, “why do we need yet another CMS at all?” But hear me out please. Okay?

Over the past decade, we’ve made amazing and huge strides in technology. We now have React.js and another hundred virtual DOM libraries out there that make it possible to run isomorphic/universal JavaScript on the client and the server. We have GraphQL, an amazingly strict API for data loading and processing. We have CSS modules. We have all kinds of new toys, but where does the web stand?

We have PHP running 82.9% of the entire web according to this source.

We have WordPress leading the CMS with a 58.9% market share.

Even Facebook itself (the inventors of React and GraphQL) are still using PHP.

So, who’s using the amazing technologies we’ve seen coming out in the past few years?

Few, fragmented projects. For instance: there’s Vulcan.js attempting to bring GraphQL and isomorphic server rendering to the Meteor flagship and offering CMS-like ease of development, if not outright CMS functionality out of the box.
There are quite a number of emerging content management systems built with Node.js and those cool technologies. Although all of them are in the early stages of development and lack maturity in my opinion; and some are more opinionated than I’d like.

But the problem I saw. The problem that still plagues me, is that all of that is meaningless to the average Joe. There’s a real barrier between your end user and the cool technologies we developers can easily deploy.

We have a load of rocket parts but no rocket scientist is building the rocket. Meanwhile, the end user is forced to use dated tech and with limited choices at that.

Unless, of course, they dished out enough money to develop a custom solution from scratch. Which is quite the undertaking, actually, considering the average wages of node developers nowadays.

So I sat down and thought: I have multiple project ideas that share a common denominator: they all require a CMS that scales, and one that’s obscenely dynamic and versatile.

Something infinitely extensible.

And just like that, a plan came to mind; and after looking at my options I decided that I would build one from scratch to suit my needs.
I worked for a while as a WordPress developer in the past, and I really liked some things about WordPress’ design philosophy. Namely how the filter and action systems make it very extensible and straightforward. So I decided to start by emulating that in JavaScript with node.

Now let’s get technical.

The current system is an amalgamation of the WordPress way of doing things and my own vision on the subject. Instead of a global filter/action system, CloudPress is component-based. Meaning that all plugins inherit a base class: Channel.

A Channel is an event-based object that supports filters and actions. If you’re not familiar with the filter system in WordPress: it’s a system where a single value (or what’s called a Payload in CloudPress) is forwarded through a chain of middleware. Each middleware (handler) can make modifications to the value or overwrite it altogether, then call the next handler in line.

As a matter of fact, since the middleware in CloudPress is essentially an async function, it can call the rest of the chain first, then modify the value. The system is versatile like that.

Here’s an example of how a plugin (here the Renderer plugin, responsible for rendering the page) might apply filters:

/**
   * Renders and returns the HTML for a page.
   * @param renderInfo Information for the renderer.
   * @param renderContext Context data.
   * @returns {string}
   */
  async render(renderInfo: RenderInfo, renderContext: RenderContext) {
    const filterContext = { renderInfo, renderContext };
    const meta = await this.applyFilter('meta', { ...renderInfo.metaTags }, filterContext);
    const { graphql, bundle } = this.imports as any;
    const scripts: string[] = [], stylesheets: string[] = [];
    const bundles = new Set(renderInfo.bundles);
    if (bundles) {
      for (let name of bundles) {
        const item: IBundle = await bundle.regenerate(name);
        if (item.script)
          scripts.push(item.script);
        if (item.stylesheet)
          stylesheets.push(item.stylesheet);
      }
    }
    /**
     * The `browser-state` filter can be used to alter the state object on the client.
     */
    const state: any = await this.applyFilter('browser-state', {
      graphql: { endpoint: await graphql.endpoint() },
      initialState: {}
    }, filterContext);
    const component = React.createElement(
      ApolloProvider, renderContext, React.createElement(HtmlContainer, {
        title: renderInfo.title || await this.engine.configManager.readKey('site.title',
          'Welcome to cloudpress!'),
        meta,
        state,
        scripts: await this.applyFilter('page-scripts', scripts, filterContext),
        stylesheets: await this.applyFilter('page-stylesheets', stylesheets, filterContext)
      } as any, React.createElement(renderInfo.component, renderInfo.props))
    );
    try {
      await getDataFromTree(component);
    } catch(e) {
      if (e.queryErrors) {
        for(let error of e.queryErrors) {
          await this.log([error.message, error.stack], Severity.Error);
        }
      } else
        await this.log([e.message, e.stack], Severity.Error);
    }
    state.initialState = renderContext.store.getState();
    cleanupApolloState(state.initialState.apollo);
    /**
     * Plugins can use the `stream` filter to alter the final HTML stream.
     */
    return this.applyFilter('stream', ReactDOM.renderToNodeStream(component), filterContext);
  }

And here’s how the browser plugin adds the viewport meta tag:

  await rendererPlugin.useFilter('meta', async (payload: Payload<any>, next: Function) => {
    // TODO: Make this configurable!
    payload.value.viewport = "width=device-width, initial-scale=1, maximum-scale=1";
    return await next();
  });

In addition to the payload’s value, the middleware can access payload.arguments to access the named arguments for the original function. This allows CloudPress plugins to modify each other’s behaviour quite easily.

Another thing to note here is how plugin inter-dependencies are handled. Each plugin offers a factory as its main module’s export.

import { IEngine, IPlugin, IPluginImports, IPluginFactory } from '@cloudpress/interfaces-core';
import { BrowserPlugin } from './browser';

const pkg = require('../package.json');

export default class BrowserFactory implements IPluginFactory {

  get name(): string { return 'Browser'; }
  get version(): string { return pkg.version; }
  get provides(): string { return 'browser'; }
  get consumes(): string[] { return ['bundle', 'router', 'redux', 'renderer', 'subscriptions']; }

  async createInstance(engine: IEngine, imports: IPluginImports): Promise<IPlugin> {
    return new BrowserPlugin(engine, imports);
  }

}

The factory lets the system know of that plugin’s requirements and what service it provides, and the system will instantiate the plugin with its imported dependencies ready and activated. For instance, in the case of the renderer plugin, it depends on bundle, graphql and redux services. It provides the renderer service which is used in turn by the router service to serve requests. In short, a plugin can provide a single service, and may consume any number of services.

What’s more (and was not shown here) is that the Channel base-class inherits yet another. It inherits a special promise-based EventEmitter that’s completely asynchronous. Which means that it will execute all event handlers in parallel and await any promises returned from them before returning. This provides functionality akin to WordPress’ actions.

And just like filters, you can broadcast and subscribe to events on any object that inherits Channel.

  async installLoadingScreen() {
    this.imports.server.on('install-middleware', async (app: Koa) => {
      await this.log('installing bundle routes');
      const outputDir = path.join(__dirname, '../assets');
      const endpoint = '/plugin-bootstrap/assets/';
      app.use(async (ctx: Koa.Context, next: Function) => {
        if (ctx.path.startsWith(endpoint)) {
          const filePath = ctx.path.substring(endpoint.length);
          return await send(ctx, filePath, { root: outputDir });
        } else
          return await next();
      });
      app.use(async (ctx: Koa.Context, next: Function) => {
        if (!this._ready && ctx.path == '/' && ctx.method == 'GET')
          await send(ctx, 'loader.html', { root: outputDir });
        else
          return await next();
      });
    });
  }

This is how all system components communicate and extend each other. At this point in time, there are 18 plugins that I’ve implemented or am in the process of implementing. The installer works. The database connection works (you can use any database that TypeORM supports), and I’m in the process of implementing the front-end, dashboard, and authentication modules.

The project is currently licensed under GPL v3 (I’m a fan of GPL), but I might switch or dual license it under MIT as well.

Until next time!

Part 2