DEV Community: Simon Knott

Iterate quickly using the new --only-changed option

Simon Knott — Wed, 21 Aug 2024 07:14:15 +0000

Playwright v1.46 ships with a nifty new feature for local development.
By specifying the --only-changed option, Playwright looks at uncommited changes and runs all affected test files.

If you've used Jest before, this is pretty similar to Jest's --onlyChanged flag.

Let's see it in action. If you prefer video, watch the Demo in the release video. In this example, we have a repository without uncommitted changes:

// utils.ts
export const inputPlaceholder = 'What needs to be done?'
export const todoItemTestID = 'todo-item'
export const todo = 'buy milk'

// test.spec.ts
import { test, expect } from '@playwright/test'
import { inputPlaceholder, todo, todoItemTestID } from './utils.ts'

test('adding a todo', async ({ page }) => {
    await page.goto('https://demo.playwright.dev/todomvc')

    const inputField = page.getByPlaceholder(inputPlaceholder)
    await inputField.fill(todo);
    await inputField.press('Enter');

    await expect(inputField).toBeEmpty();
    await expect(page.getByTestId(todoItemTestID)).toHaveText(todo);
})

If we run a test with --only-changed, we see that no tests are run:

$ npx playwright test --only-changed
Error: No tests found

Now let's make a change to one of the files, but not commit it yet:

// utils.ts
export const inputPlaceholder = 'What needs to be done?'
export const todoItemTestID = 'todo-item'
- export const todo = 'buy milk'
+ export const todo = 'make cappucino'

❯ npx playwright test

Running 1 test using 1 worker

  ✓  1 [chromium] › test.spec.ts:4:5 › adding a todo (426ms)

  1 passed (837ms)

Playwright found changes in utils.ts, so it executed all test files that depend on utils.ts. Wonderful!

By default, Playwright will compare against HEAD to determine changed files.
But you can also specify a different Git revision to compare against: Using --only-changed=main on a feature branch executes all tests that are affected by changes on the feature branch. And --only-changed=<commit sha> executes all changes between HEAD and the specified commit.

How do I use this?

--only-changed is another tool in the toolbelt to help you iterate quickly. Here's five ideas for how you can integrate it into your workflow:

1. Local development: Use --only-changed=main to quickly run affected tests as you make changes. This is especially useful when you're working on a feature and want to see the impact of your changes.

2. pre-commit hook: Running with --only-changed in a pre-commit hook prevents failing tests from being commited. Here's an example hook you can use:

npx playwright test --only-changed

3. pre-push hook: As the name says, a pre-push hook runs before pushing. This prevents failing tests from invoking a costly CI run. Here's an example implementation:

while read local_ref local_sha remote_ref remote_sha
do
    # deleting branch, nothing to check
    if [ "$local_sha" = '0000000000000000000000000000000000000000' ]; then continue; fi

    echo "Running Playwright for HEAD...$remote_sha"
    npx playwright test --only-changed="$remote_sha"
done

4. Faster feedback in CI: Prior to a full run, use --only-changed to get faster feedback on new failures. This is especially useful in large test suites, where running all tests can take a long time. Here's how that'd look in GitHub Actions:

...
    - name: Install Playwright Browsers
      run: npx playwright install --with-deps
+   - name: Run changed Playwright tests
+     run: npx playwright test --only-changed=${{ github.base_ref }}
    - name: Run Playwright tests
      run: npx playwright test
...

5. Bonus idea: If you're interested in --only-changed, you might also be interested in Playwright's Watch mode that's available in the UI and in the VS Code Extension.

That's it! Let us know what you think and how you're using --only-changed. Either in the comments here, in the Community Discord, or on LinkedIn! And have a wonderful day.

Quirrel is acquired! And I am joining Netlify

Simon Knott — Tue, 01 Feb 2022 15:59:26 +0000

I am very happy to announce that Quirrel was acquired by Netlify, and I am joining as a software engineer. This is incredibly exciting for me, and I can’t wait to show you the fruits of this acquisition.

Before I reflect on the journey, what this means for me, and how thankful I am for everything and everyone; let’s get the business things out of the way:

What does this mean for current Quirrel users?

quirrel.dev, the hosted Quirrel service, will continue operating through the end of July 2022. As of today, we will not be taking new sign-ups.

Since Quirrel is (and will stay) open source, existing customers will be able to switch to self-hosted instances, and we (Netlify + me) will do the best we can to help make this transition process smooth for you.

If you’re a customer, please reach out via info@quirrel.dev so we can coordinate on this.

Taking the experience from building Quirrel, Netlify today launches Scheduled Functions to Beta via Netlify Labs. It’s inspired by Quirrel’s CronJob feature, deeply integrated within Netlify, and a joy to use. Please try it out and let me know what you think. :)

If you have more questions about this, check out the FAQ and feel free to reach out .

With that out of the way, let’s get to more details on this announcement!

When I started working on Blitz.js in February 2020, I’d never have imagined how that journey would unfold.

I got into web development in 2017, and got my feet wet in entrepreneurship by working on my first product. Economically speaking, it was a total failure (turns out, sales teams are important sometimes!). But it ignited my love for web development and - in particular - developer tooling, and led me to join Brandon Bayer + extended community’s quest to build a better way for fullstack React applications, better known as Blitz.js.

Eventually, every fullstack application grows to need some sort of a scheduling system. But there was no such thing for serverless deployments! I quickly realised this was a gap to fill. During a holiday in Croatia, an idea of how to build this crossed my mind. I shared an API mock-up on Twitter, and was surprised by how much it resonated. Not huge, but certainly bigger than expected. On the train back from that holiday, I started hacking together a first prototype. (Please don’t tell anybody it was faked and used repeater.dev under the hood...). That’s how everything started.

Fast-forward to now. You’ve read it in the title: Netlify brought me on to bring my experience with function scheduling into the Netlify product, and they acquired the (by now, significantly more mature) Quirrel project along the way. I’m proud to share the first fruits of that cooperation today. Netlify Scheduled Functions (also known as cron jobs, for you Linux peeps out there) goes to public beta, and, if people like it, there’s more to come.

Being at Netlify now, I get to not only build a better way for fullstack React but to build a better web. To me, this is the most wonderful professional challenge, and I am super humbled to work alongside wonderful colleagues (thank you to the Netlify team including Eduardo and Ivan 💙).

When I look back, I am more than happy to see how everything unfolded. I cannot thank anybody enough who’s been part of it, this journey would’ve been much different without you.

Quirrel Newsletter #5 - Pricing RFC; Some Minor Updates

Simon Knott — Tue, 26 Jan 2021 15:50:19 +0000

It's the Quirrel Newsletter, this time with some minor updates.

Before we get to those, I'd like to ask you a small favour: If you have a minute, please take a look at this RFC. It's about changes to quirrel.dev pricing, and I'd love to get your feedback to make sure they're in-line with community consent. (don't worry, it won't become more expensive ☺️)

Alright, on to the new stuff:
Quirrel now has a Blitz Recipe! blitz install quirrel will set up everything you need to get started with it.
Also, the Quirrel UI now allows you to store connection details as a bookmark, making it a lot more comfortable to use it for production monitoring.
For some more minor changes, check out the update thread.

That's it for this issue, thanks for reading! 🐿

Best Wishes, Stay Safe
Simon

Quirrel hits v1 🥳

Simon Knott — Mon, 04 Jan 2021 13:51:52 +0000

A happy new year everyone! 🎉

I'm proud to announce: Quirrel goes to v1.
The release comes with improvements all over the board.
Let's go through it!

TL;DR: There's improved cron jobs, clients for 5 new frameworks and the Quirrel UI now also can be used for monitoring deployments. Migrating to v1

Improved Cron Jobs

Cron Jobs have been a huge PITA:

I don't understand how and where I can register "good old cron jobs"?
~ every Quirrel user, ever

With v1, they finally get a well-deserved make-over:

// pages/api/myCronJobRoute.ts
import { CronJob } from "quirrel/next";

export default CronJob(
  "api/myCronJobRoute",
  "5 4 * * *", // every day at 04:05
  async () => {
    // do something
  }
);

That's it, that's everything you need to do.
Your local Quirrel instance will automatically pick it up:

This is implemented using filesystem watchers,
similarly to how tsc --watch works.

For Deployments, running quirrel ci once during build will register all detected Cron Jobs with your production Quirrel instance.
It will also remove obsolete ones.

Support for new Frameworks

After seeing tremendous interest in Quirrel among Next.js and Blitz.js users,
I'm adding support for 5 new frameworks:

Nuxt.js (quirrel/nuxt)
Redwood (quirrel/redwood)
Vercel Serverless Functions (quirrel/vercel)
Express.js (quirrel/express)
Connect (quirrel/connect)

If you have another framework in mind that you'd like a Quirrel client for, let me know! :)

Pre-v1, you had to install both quirrel and @quirrel/next.
To ease maintenance, they were consolidated, so @quirrel/next becomes quirrel/next (notice the missing @).

Until now, Next.js users could omit the leading api/ in their Queue's routes.
Going forward, that won't be possible anymore, to match behaviour with the other framework client.

Quirrel UI for Production

You've asked for it, so here it is: The Quirrel UI now supports connecting to non-local environments.
The connection dialog can be opened using a new dropdown in the top left:

It enables you to monitor & administrate any Quirrel deployment,
which has been a heavily requested feature.

I'm thinking about adding a "Open in UI" button to the Quirrel Dashboard.
Let me know if that'd be helpful for you.

Why this release was chosen for v1

The previous state of Cron Jobs really nagged on me.
Although they played an integral part to my initial vision for Quirrel,
I struggled to find an easy-to-use design for a long time.¹
Over the course of mutliple brainstorming sessions with some of you (thank you ❤️), we managed to find a good solution.
With those improved Cron Jobs shipping in v1, I'm finally content with Quirrel's feature set.

Moving to v1 also means "going stable".
This is a big step, because we have to maintain that API for the foreseeable future.
Judging by the feedback I received, the current API is well-designed and fit for its task,
giving me the confidence to commit to it.

Thank you.

Since Quirrel's inception in Q3 last year, I've received amazing support from all of you. Thank you ❤️
If you've put in the time to brainstorm with me: Thank you x2 ❤️ Your input has been incredibly valuable for Quirrel.

As you know, I want to create the best developer experience possible.
If there's anything that detracts from it, please let me know! :D

To migrate your application to v1, follow this Guide: Migrating to v1

I'm looking forward to your feedback!

All the best for 2021,
Simon

CronJobs come with their special set of problems, primarily caused by serverless environments running forever. HMU if you want to read/hear more about that :D ↩

Quirrel Newsletter #3 - Incident Dashboards, Cron RFC

Simon Knott — Thu, 10 Dec 2020 09:40:21 +0000

Welcome to the third episode of the Quirrel Newsletter.

This one is a quick one, I just want to let you know about a new feature: The incident dashboard.

"incident" is what Quirrel calls an error in your application code.
Until now, Quirrel didn't do anything when it encountered one - not exactly optimal, eh?
Fear no longer! Quirrel will now send you an e-mail whenever one of your job handlers fails:

You'll then be able to see the incident's full job payload in the incident dashboard:

Since Quirrel jobs are end-to-end-encrypted, the dashboard also allows you to decrypt your job's payloads on the client-side :)

Quirrel users with on-premise hosted instances can specify how they want to receive incidents using the INCIDENT_RECEIVER_ENDPOINT environment variable.

But the incident dashboard is not the only thing that's changed.
I'm currently preparing a big improvement to Cron jobs and published an RFC:

https://github.com/quirrel-dev/quirrel/issues/31

If you've got a spare minute (and you most certainly have, if you're reading this), I'd love to get some feedback on that.

Alright, that's it for this episode - a quick one, as promised!

Thanks to all the nice people who helped bring these changes into existence, especially Max! Really appreciate your support :D

Best
Simon

Quirrel Newsletter #2 - v0.10 🎉

Simon Knott — Mon, 16 Nov 2020 15:31:38 +0000

Hi there!

Two weeks ago, I published you the first issue of this Newsletter.
Today, let me tell you about the new features in v0.9 and v0.10, which was released a couple of minutes ago.

But before we do that: Let me invite you to our new Quirrel community over on Spectrum!
It's still a little quiet over there, so come by and chat with us:

https://spectrum.chat/quirrel

Alright, on to the new features.

Quirrel now has human-friendly CLI output, which makes it even more fun to use:

Also, the Quirrel UI now shows all running CRON jobs.

On the API side, there are two new options for jobs:

Marking a job as exclusive guarantees it to be executed on its own, with no other jobs from the same queue running at the same time.
This can be useful any time you want to prevent race conditions from happening, e.g. when implementing an order queue.

When creating a job, you can now mark it with override: true. Now when a job with the same ID already exists, the it is overriden by your job. This is a way to easily specify behaviour for use cases where idempotency is key.

Other stuff that happened

an awesome call with Rishi from London, who requested the "exclusive jobs" feature (thanks, Rishi! great call :D)
a very interesting conversation with Wes Bos about pricing models ...
... which has led me to start working on the new pay-per-use pricing model

Chatting with Quirrel users has proven itself as an incredibly effective way to get good feedback.
If you can make time for it, I'd love to talk to you as well! Just drop me a DM on Twitter :D

That's been it for the second edition of the Quirrel Newsletter. Have a great week!

Best
Simon

Quirrel Newsletter #1 - Nibble Ahead

Simon Knott — Mon, 02 Nov 2020 17:10:47 +0000

Hello everyone, Simon from Quirrel here!

Its my pleasure to present you the first issue of the Quirrel Newsletter.
The Newsletter won't be published on a strict schedule, but rather as I see fit.
It covers all things Quirrel: Improvements to the tool itself, interesting stories from Quirrel users, insights into the business of a bootstrapped MicroSaaS and lots more.

Let's get into it, shall we? :D

Quirrel, if you haven't heard about it, is a job queueing solution for serverless deployments, developed by me. It's MIT licensed and I work in public, so check it out :D

What's happened in the last weeks

In true changelog manner, let me start by giving you an overview of the developments of the last weeks:

Quirrel now supports cron jobs
the development environment is now published to NPM
the onboarding experience has been improved
the client now features JSDoc comments for improved DX
you can now see the status of Quirrel infrastructure on the status page
delays can now be specified in a human-friendly way using vercel/ms
I've added Telemetry
a new Tutorial has been published

As you can see, a lot of work has been done, making Quirrel even more useful.

One of the more future-facing efforts I started is called "Owl".
Owl is a queuing library tailor-made for Quirrel, aimed at replacing bullmq.
It will make Quirrel more performant and remove the need to run Redis during development (that means no more Docker for you Next.js developers 🎉).

The past weeks have also been accompanied by a lot of wonderful interactions with Quirrel users.
I don't have space to list all of them, but I'd especially like to thank Max Finke, Jan Wilhelm and Tim Suchanek for brainstorming with me.

From a business perspective, it's going well.
quirrel.dev (the hosted Quirrel instance) currently has 53 registered users across four continents, and a handful of them have even deployed an application using it.
There's one paying customer, putting Quirrel at a MRR of 10$.
Not much, I know - but considering Quirrel is a 2-months-old product that's in full development, it's quite a good start.

The State of Quirrel

"Where are we, in the Quirrel journey?"

The answer to that question determines what I should focus on:
If the product is done, I should focus on promoting it and growing its userbase.
If it's in development or emerging, doing the opposite would be wise: Collaborate with existing users to improve the product, but don't expand yet (to prevent churn).

At the moment, Quirrel is definitely in development.
While I'm kind of satisfied with its features, I still need to find out more about the way people use Quirrel and their individual needs.

So if you use Quirrel, I'd love to hear from you! Drop a comment or leave me a DM on Twitter, I won't bite 😇

Once I'm confident that I achieved "product market fit", I'll ramp up the Quirrel media presence and bring more users onboard.

Outlook

So now that we know about the Status Quo, let's take a look ahead.

There's a couple of upcoming changes I'm planning to make:

By the end of the week, Quirrel will transition to Owl. This will enable Quirrel users to run the local development version of Quirrel without having to run Redis.

I'm also experimenting with Fire-and-forget HTTP requests, which has the potential to substantially improve throughput on large Quirrel instances.
This change will be made once the performance improvements become necessary.

Quirrel.dev will also see a slight change to its pricing model. The current 20$/month plan costs the same, wether you make 500 or 10.000 calls.
This strongly favours high-usage projects for no apparent reason.
I'm currently thinking about introducing a pay-per-use model with a lower base fee and additional charges based on usage, which is a model that multiple users have already suggested.

Last, but not least, let's talk about client libraries.
While at the moment, I'm talking about Quirrel as the job queueing solution for Next.js, it will be equally useful for other frameworks in the future.
I'm currently planning on creating integrations for Redwood, Sapper, Nuxt and Express. Which framework would you like to see a client library for?

Thanks for reading!

If you've made it this far, that shows quite some dedication.

I'd like to emphasise on what I wrote earlier:

If you use Quirrel (and even if you don't), I'd love to hear from you! Drop a comment or leave me a DM on Twitter, I won't bite 😇

Anyway, I hope this Newsletter was interesting to you. What new stuff did you learn about Quirrel? Let me know :D

Have a great week,

Simon

Building a water-drinking reminder with Next.js and Quirrel

Simon Knott — Thu, 15 Oct 2020 08:34:03 +0000

Hi there! Welcome to the first ever post on the Quirrel blog!
Quirrel is an open-source task queueing service that's easy to use and sparks joy.
In this tutorial, we're going to use it to create a water drinking reminder - because if you're anything like me, it's easy to forget sometimes.

Check out the finished version here: water-reminder.quirrel.dev

To follow this tutorial, you should already know a little bit about Next.js.
If you’re totally new to it, check out the Next tutorial first.

Let's get started!

The first thing we're gonna do is to create a new Next.js project.



$ npx create-next-app water-reminder

Open up water-reminder in your favourite editor and run npm run dev to startup the development environment.

Take a look into pages/index.js and replace its content with the following:



// pages/index.js
export default function Home() {
  return (
    <main>
      <h1>
        Water Drinking Reminder
      </h1>

      <p>
        I want to be reminded under the following e-mail:
      </p>

      <form
        onSubmit={(evt) => {
          evt.preventDefault();
          const formData = new FormData(evt.target);
          const email = formData.get("email");
          alert(email);
        }}
      >
        <input
          name="email"
          type="email"
          placeholder="E-Mail"
        />
        <button type="submit">
          Submit
        </button>
      </form>
    </main>
  );
}

It contains some markup and a simple form that lets you submit the e-mail you want to be reminded at.
At the moment, it will just alert the typed email.
In your browser, open up localhost:3000.
It should look similar to this:

Submitting the form to the backend

Setup a new API Route by creating pages/api/setupReminder.js and adding the following code:



// pages/api/setupReminder.js
export default async (req, res) => {
  const email = req.body;
  console.log(`I'll setup the reminder for ${email}.`);
  res.status(200).end();
}

Now instead of alert-ing the form value, let's post it to the newly created API route.
Go back to pages/index.js and replace the following:



// pages/index.js
onSubmit={(evt) => {
  evt.preventDefault();
  const formData = new FormData(evt.target);
  const email = formData.get("email");
- alert(email);
+ fetch("/api/setupReminder", { method: "POST", body: email });
}}

Submitting the form will now cause the e-mail to be printed to your development console:

Now that we've hooked up the form with the API Route, let's get into Quirrel and E-Mail sending.

Setting up Quirrel

Quirrel is a task queueing service.
It takes requests à la "call me back at /api/queue in 10 minutes", stores them and makes sure to call back /api/queue as requested.
What's awesome about Quirrel is that it can run fully locally using the Quirrel CLI:



$ npm install -g quirrel
$ quirrel
🎉 Starting the Quirrel Dev Environment 🎉

Now create a new queue by creating pages/api/queues/reminder.js and typing the following:



// pages/api/queues/reminder.js
import { Queue } from "quirrel/next";

export default Queue(
  "api/queues/reminder", // the route it's reachable under
  async (recipient) => {
    console.log(`Sending an E-Mail to ${recipient}`);
  }
);

Queue takes two arguments: The first one is it's API route location, and the second one is the worker function.
Whenever a job is executed, the worker function is called.

To use our newly created Queue, simply import it from the API Route:



// pages/api/setupReminder.js
+ import reminderQueue from "./queues/reminder";
// 👆 don't forget this

export default async (req, res) => {
  const email = req.body;
- console.log(`I'll setup the reminder for ${email}.`);
+ await reminderQueue.enqueue(
+   email,
+   {
+     id: email,
+     delay: "30min",
+     repeat: {
+       every: "30min",
+       times: 16, // 16 * 30min = 8h
+     },
+   }
+ );
  res.status(200).end();
};

Calling .enqueue will schedule a new job.
The first argument is the job's payload while the second argument contains some options:

id prevents having multiple reminders for the same e-mail address
repeat makes the job execute on twice an hour, for a duration of 8 hours
delay adds an initial delay of 30 minutes, so the first job isn't executed immediately

To verify that this works, open up the Quirrel Development UI at ui.quirrel.dev.
It will connect to your local Quirrel instance and show all pending jobs in the "Pending" tab:

If it doesn't connect, that may be because you're using Safari. Try a different browser instead.

Submitting your email to the form at localhost:3000 will add a new job to the UI, and pressing "Invoke" will execute the job.
You'll now be able to see Sending an E-Mail to XYZ in your development logs.

Because it's a repeated job, it will be re-scheduled immediately, until it's been executed for the 16th time.

Before we proceed with the last part of the tutorial: Stand up, go to the kitchen and grab a glass of water 💧

Let's hook up E-Mail!

Now that the Queue is working, let's hook up the final thing: E-Mail!

Run npm install nodemailer and add your SMTP setup code to your reminder queue:



// pages/api/queues/reminder.js
import { Queue } from "quirrel/next";
+ import { createTransport } from "nodemailer";

+ const mail = createTransport({
+   host: "smtp.ethereal.email",
+   port: 587,
+   auth: {
+     user: "randall.renner66@ethereal.email",
+     pass: "Dp5pzSVa52BJwypJQm",
+   },
+ });

...

If you don't have any SMTP credentials at hand, you can get some demo ones at ethereal.email.

Then simply switch out the console.log call with a real email dispatch:



...

export default Queue(
  "api/queues/reminder",
  async (recipient) => {
-   console.log(`Sending an E-Mail to ${recipient}`);
+   await mail.sendMail({
+     to: recipient,
+     from: "waterreminder@quirrel.dev",
+     subject: "Remember to drink some water!",
+     text: "..."
+   })
  }
);

That's it! Now our app is fully working.
It may not be the best water reminder service ever, but it's your very own one.

Here are some ideas to improve it further:

Make duration and interval of the reminder configurable
Allow users to unsubscribe using a link in the email
Add some styles, maybe using Tailwind CSS

Deploying this to production is easy using the managed Quirrel service.
Simply follow this guide: Deploying Quirrel to Vercel

Conclusion

We've built a working water reminder in little under an hour.
You can see the finished project here: quirrel-dev/water-reminder-demo

If you've got experience with well-known task queueing libraries like beanstalkd or SideKiq, you may have noticed how easy-to-use Quirrel is.
The highly integrated client libraries for popular frameworks and the managed solution available, Quirrel is a great choice for JAMStack users.

And if you want to host Quirrel yourself, the MIT License allows you to do so.

Fire-and-forget HTTP Requests using Node.js

Simon Knott — Wed, 14 Oct 2020 09:00:00 +0000

In most cases, we care about the results of our HTTP Requests. But sometimes, it’s just about making the request, and not so much about the response. In these cases, it may be wise to save yourself some network bandwidth by ignoring the response entirely.

How does this work?

HTTP Requests are based on TCP connections. Running curl google.com will open up a TCP connection to google.com:80 and send roughly the following:

GET / HTTP/1.1
Host: google.com
User-Agent: curl/7.64.1
Accept: */*

Line	Meaning
1	Request Line. Contains Method, Location and Protocol Version
2	The only required header: “Host”
3, 4	Some Headers
5	blank line to separate header and body

Normally, you’ll then wait for the server to respond like so:

HTTP/1.1 200 
Content-Type: text/html

<html><head> ...

But the response isn’t of interest to us. We’ll just send off our request, then terminate the TCP connection and carry on with more interesting stuff.

A rough test implementation

The first thing we need is a test server we can make our calls against:

const http = require("http");

http
  .createServer((req, res) => {
    req.on("end", () => {
      setTimeout(
        () => {
          console.log("Done");
          res.statusCode = 200;
          res.end("Done");
        },
        1000
      );
    });
  })
  .listen(5000);

To simulate time-consuming work, this server delays responding by one second.

Now onto the interesting part: Making the manual HTTP request.

const net = require("net");

const client = new net.Socket();

client.connect(5000, "localhost", () => {
  // request line
  client.write("POST / HTTP/1.1\r\n");

  // headers
  client.write("Host: localhost:5000\r\n");
  client.write(`Content-Length: 11\r\n`);

  // blank line
  client.write("\r\n");

  // body
  client.write("Hello World");

  // end the connection prematurely
  client.destroy();
});

client.on("close", () => {
  console.log("Connection Closed");
});

It’s rather simple! We open up the connection to localhost:5000 and dispatch the request. The important line is the call to client.destroy(), which will end the socket immediately after the request has been transmitted.

Let’s see it in action:

Conclusion

Fire-and-forget HTTP calls aren’t that hard to make. They can be of use in bandwith-constrained environments - I will probably make use of them for Quirrel.

If this post was useful or interesting to you, make sure leave a comment or write me a message on Twitter! I always love to hear about my content helping others.

SuperJSON - JSON on steroids

Simon Knott — Sun, 04 Oct 2020 22:00:00 +0000

SuperJSON is a high-fidelity replacement to JSON.stringify. It supports data types like Date, RegExp, Map, Set etc., preserves referential equality and supports circular references. This blog post aims to compare it to alternatives and explain how it works.

What’s the Problem?

JSON is the defacto standard data interchange format for web applications. It’s human-readable, has broad tooling support and its choice on data types are compatible with most platforms.

As usage of JavaScript across backend and frontend became more popular, one of its value propositions have been let down by JSON: It doesn’t support data types like Date, RegExp, Map or Set, forcing developers to write cumbersome adapter logic to work around these limitations.

This problem became especially apparent while working on Blitz.js. Blitz.js is a fullstack React framework that (amongst other things) generates your applications’ API layer at compile time. It thus allows developers to import backend functions straight from the frontend, completely abstracting away network calls. Here’s the problem: If we used JSON.stringify, our developers would be surprised because it destroyed their nice data types! That’s where SuperJSON comes into play: It’s able to serialize any JavaScript value and thus saves our developers from quite some headache.

Other ways to solve this

SuperJSON is not the first contender, other approaches that solve this problem have existed.

devalue by Rich Harris is an algorithm that encodes value into JavaScript code that creates it - it turns { foo: "bar", self: this } into (function(a){a.foo="bar";a.self=a;return a}({})), which can then be evaluated to get back the original value. While it’s incredibly efficient¹, there isn’t any tooling support for it, and it also can’t be used to send data to the server - what if a malicious client sent while (true) { /* do nothing */ }?

A similar approach is taken by Yahoo’s Serialize JavaScript, but it suffers from the same problems.

Goals of SuperJSON

Contrary the approaches mentioned above, SuperJSON aims to …

… support any JavaScript value
… be safe from remote code execution
… be JSON-compatible, so existing tooling can be leveraged
… support circular references
… preserve referential equalities
… be human-readable, so it’s easy to debug
… be easy-to-understand, even if you don’t know it’s SuperJSON

How does SuperJSON solve it?

There are two parts to what we’re trying to achieve. We need to …

… transform any JavaScript value into a JSON-compatible one
… be able to reverse it!

Transforming into a JSON-compatible value

Transforming is quite easy, actually: For already compatible ones, it’s trivial: 42 becomes 42, "lol" becomes "lol", true becomes true and false becomes false.

It’s a bit harder for values where there is no JSON counterpart: Set { "foo", "bar" } becomes ["foo", "bar"], Map { 1 => 11, 2 => 22 } becomes [[1, 11], [2, 22] ] and NaN becomes "NaN".

So given this value …

{
    a: new Set([/a-Z/g]),
    b: new Map([
        [1, NaN],
        [2, NaN]
    ])
}

… we’ll create this json:

{
    "a": ["/a-Z/g"],
    "b": [
        [1, "NaN"],
        [2, "NaN"]
    ]
}

Easy, right? This can done recursively and fits into a couple of lines of code.

But we won’t be able to reverse this, right? Unless … we just take notes:

{
    "a": "set",
    "a.0": "RegExp",
    "b": "map",
    "b.0.1": "number",
    "b.1.1": "number"
}

This contains notes on any non-trivial transformations.²

We can then safely send both our JSON-ified value and the notes over the network. During deserialization, we then apply the notes to recreate the original value:

value["b"][1][1] = Number(value["b"][1][1])
value["b"][0][1] = Number(value["b"][0][1])
value["b"] = new Map(value["b"])
...

Not so hard, right? Now that we can preserve types, let’s see how to preserve referential equalitites.

Preserving Referential Equalities

Imagine the object that’s created by the following code:

const a = { id: "a" }
const b = { id: "b" }

{ options: [a, b], selected: a }

As stated in the Goals, we want to preserve the fact that options[0] === selected. This can be done by not only taking notes on types to recreate, but also on referential equality. For the value above, SuperJSON takes the following notes:

{
    "selected": ["options.0"]
}

We can then use these notes during deserialization like so:

value["options"]["0"] = value["selected"]

This feature can also be used to preserve circular references.

Go use it!

SuperJSON is currently in beta, it should be ready for you to use in your own projects:

npm install --save superjson
yarn add superjson


import SuperJSON from "superjson"

const string = SuperJSON.stringify(yourValue)
const profit = SuperJSON.parse(string)

We’re currently testing support for preserving class and Symbol instances, we’ll also work on making SuperJSON faster and smaller.

I’d love to hear what you think of it! Tweet at me: @skn0tt or write a comment below.

see this benchmark for comparison ↩

In reality, we’re encoding these in a tree structure to save space. But that’s a topic for another post 😄 ↩