DEV Community: Eduard Dyckman

Coding exercise: database migration tool in nodejs

Eduard Dyckman — Wed, 25 Sep 2024 11:29:12 +0000

Requirements

I want to have a database migration tool, that has the following properties:

Every migration is written in a single SQL file, meaning both "up" and "down" parts. This will allow Copilot to fill in the rollback migration. And the fact that it's a bare SQL also makes it the most flexible and supported solution.
The currently applied version should be managed by the tool. I want the tool to be self-sufficient.
I want the tool to support different databases, such as Postgres, MySQL, SQL Server, etc., so it should be extendable in that sense.
I don't want it to be oversized, so only drivers for the necessary database should be installed, ideally on demand.
I want it to be part of the javascript ecosystem since most of the projects I work on are a part of it.
Every migration should be performed inside of a transaction.

Introduction

A lot of these points were born from my experience with this awesome tool called tern. I was sad that javascript doesn't have the same! (Or maybe I suck at googling...). So I decided this could be a nice coding exercise for myself and a story that could be interesting to someone else :)

Development

Part 1. Designing the tool

Let's ~~steal~~ design the CLI tool!

All migrations would have the following naming scheme: <number>_<name>.sql, where the number would represent the migration version number, for example, 001_initial_setup.sql.
All migrations would reside in a single dir.
Database driver would be downloaded on demand, either some pre-bundled package or just issuing some sort of npm install <driver>.

So the syntax for the tool would be the following: martlet up --database-url <url> --driver <driver> --dir <dir> or martlet down <version> <same options>.

Where "up" should apply all migations that are not applied yet and down should rollback to the specified version.
Options have the following meaning and defaults:

database-url - connection string for the database, the default would be to lookup the env variable DATABASE_URL
driver - database driver to use. For the first version, I will only support Postgres with an option named "pg".
dir - directory where migrations reside, default is migrations

As you can see, I've started with figuring out how I would invoke the tool before writing any actual code. This is a good practice, it helps to realize requirements and reduce development cycles.

Part 2. Implementation

2.1 Parsing options

Ok, first things first! Let's create an index.js file and output the help message. It would look something like this:

function printHelp() {
  console.log(
    "Usage: martlet up --driver <driver> --dir <dir> --database-url <url>",
  );
  console.log(
    "       martlet down <version> --driver <driver> --dir <dir> --database-url <url>",
  );
  console.log(
    "       <version> is a number that specifies the version to migrate down to",
  );
  console.log("Options:");
  console.log('  --driver <driver>  Driver to use, default is "pg"');
  console.log('  --dir <dir>        Directory to use, default is "migrations"');
  console.log(
    "  --database-url <url> Database URL to use, default is DATABASE_URL environment variable",
  );
}

printHelp();

Now we will parse options:

export function parseOptions(args) {
  const options = {
    dir: "migrations",
    driver: "pg",
    databaseUrl: process.env.DATABASE_URL,
  };
  for (let idx = 0; idx < args.length; ) {
    switch (args[idx]) {
      case "--help":
      case "-h": {
        printHelp();
        process.exit(0);
      }
      case "--dir": {
        options.dir = args[idx + 1];
        idx += 2;
        break;
      }
      case "--driver": {
        options.driver = args[idx + 1];
        idx += 2;
        break;
      }
      case "--database-url": {
        options.databaseUrl = args[idx + 1];
        idx += 2;
        break;
      }

      default: {
        console.error(`Unknown option: ${args[idx]}`);
        printHelp();
        process.exit(1);
      }
    }
  }
  return options;
}

As you can see, I don't use any library for parsing; I just simply iterate over the arguments list and process every option. So, if I have a boolean option, I would shift the iteration index by 1, and if I have an option with a value, I would shift it by 2.

2.2 Implementing the driver adapter

To support multiple drivers, we need to have some universal interface to access a database; here is how it may look:

interface Adapter {
    connect(url: string): Promise<void>;
    transact(query: (fn: (text) => Promise<ResultSet>)): Promise<ResultSet>;
    close(): Promise<void>;
}

I think connect and close are pretty obvious functions, let me explain the transact method. It should accept a function that would be called with a function that accepts a query text and returns a promise with an intermediate result. This complexity is required to have a general interface that would provide ability to run multiple queries inside of a transaction. It's easier to grasp by looking at the usage example.

So this is how the adapter looks for the postgres driver:

class PGAdapter {
  constructor(driver) {
    this.driver = driver;
  }

  async connect(url) {
    this.sql = this.driver(url);
  }

  async transact(query) {
    return this.sql.begin((sql) => (
      query((text) => sql.unsafe(text))
    ));
  }

  async close() {
    await this.sql.end();
  }
}

And the usage example could be:

import postgres from "postgres";

const adapter = new PGAdapter(postgres);
await adapter.connect(url);
await adapter.transact(async (sql) => {
    const rows = await sql("SELECT * FROM table1");
    await sql(`INSERT INTO table2 (id) VALUES (${rows[0].id})`);
});

2.3 On-demand driver installation

const PACKAGES = {
  pg: "postgres@3.4.4",
};

const downloadDriver = async (driver) => {
  const pkg = PACKAGES[driver];
  if (!pkg) {
    throw new Error(`Unknown driver: ${driver}`);
  }
  try {
    await stat(join(process.cwd(), "yarn.lock"));
    const lockfile = await readFile(join(process.cwd(), "yarn.lock"));
    const packagejson = await readFile(join(process.cwd(), "package.json"));
    spawnSync("yarn", ["add", pkg], {
      stdio: "inherit",
    });
    await writeFile(join(process.cwd(), "yarn.lock"), lockfile);
    await writeFile(join(process.cwd(), "package.json"), packagejson);
    return;
  } catch {}
  spawnSync("npm", ["install", "--no-save", "--legacy-peer-deps", pkg], {
    stdio: "inherit",
  });
};

We try to install the driver with yarn at first, but we don't want to generate any diffs in the directory, so we preserve yarn.lock and package.json files. If yarn is not available, we will fall back to npm.

When we ensured that the driver is installed, we can create an adapter and use it:

export async function loadAdapter(driver) {
  await downloadDriver(driver);
  return import(PACKAGES[driver].split("@")[0]).then(
    (m) => new PGAdapter(m.default),
  );

2.4 Implementing the migration logic

We start by connecting to the database and getting the current version:

await adapter.connect(options.databaseUrl);
console.log("Connected to database");

const currentVersion = await adapter.transact(async (sql) => {
    await sql(`create table if not exists schema_migrations (
      version integer primary key
    )`);
    const result = await sql(`select version from schema_migrations limit 1`);
    return result[0]?.version || 0;
});

console.log(`Current version: ${currentVersion}`);

Then, we read the migrations directory and sort them by version. After that, we apply every migration that has a version greater than the current one. I will just present the actual migration in the following snippet:

await adapter.transact(async (sql) => {
    await sql(upMigration);
    await sql(
      `insert into schema_migrations (version) values (${version})`
    );
    await sql(`delete from schema_migrations where version != ${version}`);
});

The rollback migration is similar, but we sort the migrations in reverse order and apply them until we reach the desired version.

3. Testing

I decided not to use any specific testing framework but use the built-in nodejs testing capabilities. They include the test runner and the assertion package.

import { it, before, after, describe } from "node:test";
import assert from "node:assert";

And to execute tests I would run node --test --test-concurrency=1.

Actually, I was writing the code in a sort of TDD manner. I didn't validate that my migrations code worked by hand, but I was writing it along with tests. That's why I decided that end-to-end tests would be the best fit for this tool.
For such an approach, tests would need to bootstrap an empty database, apply some migrations, check that database contents are correct, and then roll back to the initial state and validate that the database is empty.
To run a database, I used the "testcontainers" library, which provides a nice wrapper around docker.

before(async () => {
    console.log("Starting container");
    container = await new GenericContainer("postgres:16-alpine")
    .withExposedPorts(5432)
    .withEnvironment({ POSTGRES_PASSWORD: "password" })
    .start();
});

after(async () => {
    await container.stop();
});

I wrote some simple migrations and tested that they worked as expected. Here is an example of a database state validation:

const sql = pg(`postgres://postgres:password@localhost:${port}/postgres`);
const result = await sql`select * from schema_migrations`;
assert.deepEqual(result, [{ version: 2 }]);
const tables =
    await sql`select table_name from information_schema.tables where table_schema = 'public'`;
assert.deepEqual(tables, [
    { table_name: "schema_migrations" },
    { table_name: "test" },
]);

4. Conclusion

This was an example of how I would approach the development of a simple CLI tool in the javascript ecosystem. I want to note that the modern javascript ecosystem is pretty charged and powerful, and I managed to implement the tool with a minimum of external dependencies. I used a postgres driver that would be downloaded on demand and testcontainers for tests. I think that approach gives developers the most flexibility and control over the application.

5. References

Generators in the wild

Eduard Dyckman — Sun, 06 Aug 2023 12:22:51 +0000

In the previous article, I talked about iterators and generators in JavaScript but did not provide any practical examples. With this article, I aim to fill this gap and demonstrate the specific pattern using generators through a simple web application.

This article may also be helpful for those looking to understand how redux-saga works.

Additionally, I have been eager to explore Deno (an alternative to Node) and will be using it as the runtime environment for the code, so the examples will be in TypeScript this time.

TLDR - Code

A bit about Deno

Since I've mentioned Deno, let me provide a brief overview of it. Deno is a runtime environment for executing JavaScript and TypeScript code, serving as an alternative to Node.js. Here are its main differences from Node:

Native TypeScript support: Deno supports TypeScript natively, meaning you don't have to manually transpile TypeScript into JavaScript; Deno's subsystem takes care of this for you.
Permission-based system: Deno's permissions are based on command-line flags, providing more control over what authority a script can have when running.
Full URLs instead of package names: In Deno, you specify full URLs instead of package names, allowing you to use different versions of packages within the same program.
Modern standard library: Deno comes with its own set of standard utilities that offer a more modern API compared to Node.js. For instance, all asynchronous operations return promises instead of relying on callbacks.

The application

So, what will our web application look like:

It will be a chatbot capable of doing just a few things:

Reporting the current time
Adding numbers
It will be able to interact with multiple users simultaneously

The web interface will be designed as follows:

For creating a web server with WebSocket support in Deno, specialized libraries are not required. Standard functions will be sufficient for our task. But first...

A bit more about generators

In the previous article, I introduced generators and iterators, but I covered only a fraction of what they are capable of. In this article, I won't cover all aspects either, but I'll focus on two features that will be useful for solving our initial task.

Asynchronous Iterators

On each invocation, an iterator returns some value. If the value returned for each step is a promise, such an iterator is referred to as an asynchronous iterator. For such iterators, we can traverse them using a special kind of loop: for await. For this to work, the object we want to support traversal with this loop must have a special method called Symbol.asyncIterator, which returns an asynchronous iterator (generators created with the async keyword return asynchronous iterators).

Example:

async function* timer() {
  let i = 0;
  while (true) {
    yield new Promise(
      (resolve) => setTimeout(() => resolve(++i), 1000)
    );
  }
}

for await (const tick of timer()) {
  console.log(tick);
}
// 1 2 3 ...

yield*

In addition to the yield operator used to return the current value of an iterator, there exists the yield* operator. It takes an iterator as a parameter and sequentially returns all its values. Into a generator, it returns the output of the iterator (the first value for which done is true, or in the case of a generator, the value passed to the return).

function* concat<T>(...iterables: Iterable<T>[]) {
  for (const iter of iterables) {
    yield* iter;
  }
}

for (const i of concat([1, 2], [3, 4])) {
  console.log(i);
}
// 1 2 3 4

The code

Creating an HTTP server listening on a specific port is straightforward in Deno:

import { serve } from "https://deno.land/std@0.74.0/http/server.ts";

const server = serve({ port: 8080 });
console.log("Listening on 8080");

for await (const req of server) {
  req.respond({ status: 200, body: 'Hello, world!' });
}

Github

As you can see, the server resulting from the serve function call is an asynchronous iterator. Each invocation of the next method returns a promise that resolves when an incoming connection is received.

Next, we need two things from our server: serving static files and handling WebSocket connections. Let's create a simple helper that chains request handlers together, similar to middlewares in Express.

type MiddlewarePayload = {
  url: URL;
  req: ServerRequest;
};

type MiddlewareFn = (options: MiddlewarePayload) => Promise<true | undefined>;

const combineProcessors = (...fns: MiddlewareFn[]) => async (options: MiddlewarePayload) => {
  for (const fn of fns) {
    const result = await fn(options);
    if (result) {
      return result;
    }
  }
}

Github

Each handler passed into such a combiner should return a promise that resolves to true if it processed the request; otherwise, it resolves to undefined.

Ultimately, the request handling code will look like this:

const processors = combineProcessors(index, staticFiles, wsMiddleware);

const server = serve({ port: 8080 });

console.log("Listening on 8080");

// The request's host is not important for our program,
// but it's required for the URL constructor
const BASE = "http://localhost";
for await (const req of server) {
  const url = new URL(req.url, BASE);
  const result = await processors({ url, req });
  if (!result) {
    req.respond({ status: 404 });
  }
}

Github

I won't explain the index and staticFiles functions here, as they handle serving static files and can be found on GitHub if you're interested. However, let's delve into the WebSocket connection handler in more detail.

Channels

The concept of channels existed long before the birth of the JavaScript language. This model describes inter-process communication and messaging in an asynchronous environment. Native implementations of channels can be found in many modern programming languages, such as Go, Rust, Kotlin, Clojure, and others.

If you are familiar with the concept of streams, transitioning to channels will be relatively straightforward. A stream, like a channel, provides asynchronous access to sequential data. The main difference lies in their access models: a stream uses a subscription model (when a message arrives, call the handler), while channels use a blocking model (give me the next message, and don't proceed until it arrives). Here's an example of their usage:

/** Streams **/
const stream = new Stream();
stream.subscribe(callback);
// Somewhere else in the code
stream.emit(data);

/** Channels **/
const ch = new Channel();
// Somewhere else in the code
ch.put(data);
// Somewhere else in the code
const data = await ch.take();

Here is how channels are implemented in our example:

class Channel {
  private takers: Array<(payload: string) => void> = [];
  private buffer: string[] = [];

  private callTakers() {
    while (this.takers.length > 0 && this.buffer.length > 0) {
      const taker = this.takers.shift()!;
      const payload = this.buffer.shift()!;
      taker(payload);
    }
  }

  take() {
    const p = new Promise<string>((resolve) => {
      this.takers.push(resolve);
    });
    this.callTakers();
    return p;
  }

  put(message: string) {
    this.buffer.push(message);
    this.callTakers();
  }

  async listen(sock: WebSocket) {
    for await (const event of sock) {
      if (typeof event === "string") {
        this.put(event);
      }
    }
  }
}

Github

Explanation of what is happening here:

There is an array buffer, where incoming messages are stored.
There is an array takers, where functions to resolve promises are stored.
On each put (put a message into the channel) and take (wait for and take a message from the channel) method call, a check is made to see if there is at least one message in the buffer and at least one taker. If so, the message is resolved, removed from the buffer, and the taker is removed from the takers array.
There is also a helper listen method, which subscribes to all messages from the given socket and puts them into the channel.

Why did we need a channel in the first place? What's wrong with the subscription model? This will become evident later; for now, I'll just say that this allows us to write asynchronous code as if it were synchronous (which is precisely what async/await were created for).

What about generators?

Indeed, if you were expecting to see the practical application of generators earlier in the article, you might have been surprised. However, the time has come to explore the core concept.

As a reminder of what we aim to achieve and what we already have: we receive messages from users via a WebSocket into a channel, and in response, we send messages from the bot back to the user via WebSocket.

Let's define the shape of an "effect":

type Effect = {
  type: string;
  [key: string]: any;
};

If you are familiar with the Flux or Redux architecture, you will recognize this structure — it's very similar to an action! In our case, the effect will serve a very similar purpose. In Redux, the following formula applies:

const newState = reducer(state, action);

For us, it will work like this:

while (true) {
  const { value: effect, done } = iter.next(current);
  // Code to handle the effect goes here
  if (done) break;
}

Here's the idea: let's represent our entire dialogue with users as a generator. This generator will yield effects and accept the results of handling these effects. We'll have a special code that runs the generator and also handles the effects. And here it is:

export async function handleWs(sock: WebSocket) {
  const incoming = new Channel();
  incoming.listen(sock);

  let current: string = "";
  const iter = dialog();
  while (true) {
    const { value: effect, done } = iter.next(current);
    if (!effect) {
      break;
    }
    switch (effect.type) {
      case "say": {
        sock.send(effect.text);
        break;
      }
      case "listen": {
        current = await incoming.take();
        break;
      }
    }
    if (done) {
      break;
    }
  }
}

Github

This function is called when a new WebSocket connection is established.

We use two types of effects (though there could be as many as needed):

say effect - indicates that we need to send a response to the user.
listen effect - indicates that we need to wait for a message from the user. Although the loop is infinite, it won't cause any blocking of the process, thanks to the presence of await inside the loop. This will interrupt the execution of the loop until a message is received in the channel. This is made possible by the use of channels.

Now let's take a look at how the actual dialogue looks:

const say = (text: string) => ({ type: "say", text } as const);
const listen = () => ({ type: "listen" } as const);

function* dialog() {
  yield say('Welcome to "Do what I say BOT"');
  while (true) {
    const message: string = yield listen();
    if (message.toLowerCase().includes("time")) {
      yield say(`It is ${format(new Date(), "HH:mm:ss")}`);
    } else if (message.toLowerCase().includes("sum")) {
      yield* sumSubDialog();
    } else {
      yield say(`I don't know what to say!`);
    }
  }
}

function* sumSubDialog() {
  yield say("Okay, what numbers should we sum?");
  let result = 0;
  let message = yield listen();
  while (true) {
    const num = Number(message);
    if (isNaN(num)) {
      break;
    } else {
      result += num;
    }
    yield say("Got it!");
    message = yield listen();
  }
  yield say(`The result is: ${result}`);
}

Github

The dialogue is represented as a generator that yields different effects at each step. The say effect is executed immediately, sending a message to the socket, and the generator's code continues without waiting for data from the external environment. The listen effect suspends the generator's execution until a message is received, which is then immediately passed back to the generator.

The advantages of this approach are as follows:

Conciseness: Generators provide a compact and readable representation of business logic, as long as the code does not become too deeply nested (which can be mitigated by decomposing generators).
Convenience of decomposition: You can split the main generator's logic into multiple sub-generators using the yield* operator, which simplifies code structuring.
Simplicity of effects: Effects are simple structures, well-typed, easily constructed, and can be serialized and transmitted over the network (though the use case for this is not immediately clear).
Isolation of dialogues: Each call to the generator returns a new iterator with its own closure, enabling multiple parallel dialogues without resource leaks (assuming no global variables are used inside the generator).

One major downside to this approach is that the types of returned values from the yield operator are not automatically inferred. This is logical since the generator's code serves as a signature, and the type of what can be passed to the next of the created iterator is already inferred from it. Therefore, types for the yielded values have to be manually provided in our approach.

I want to highlight an important aspect of this approach: The code inside the generator (the dialogue description) is very abstract. It doesn't deal with message passing through channels, error handling for transmission, or anything specific to the execution environment. It represents pure business logic — precise and noiseless description of business processes.

In an ideal scenario, the generator should be a pure function. For a generator, this means that for the same input parameters, the returned iterators should be identical (the same sequence of input data generates the same sequence of output values). The purity of generators ensures the encapsulation of business logic, meaning it describes specific use case and doesn't rely on the execution environment. It also makes it easy to write tests for your business logic: you just need to check that the sequence of effects returned by the generator matches the expected sequence.

Want to read more?

js-csp. CSP in js implementation.
redux-saga. The most popular effects library in js
Coroutines

Iterators and generators

Eduard Dyckman — Sun, 18 Jun 2023 12:40:56 +0000

JavaScript is currently the most popular programming language according to many platforms (such as GitHub). However, does popularity equate to being the most advanced or beloved language? It lacks certain constructs that are considered integral parts of other languages, such as an extensive standard library, immutability, and macros. But there is one detail that, in my opinion, doesn't receive enough attention - generators.

In this article I want to explain possible use cases for iterators and generators and how they improve the verbosity of one's code. I hope, that after reading this article the following piece of code will make all the sense:

while (true) {
    const data = yield getNextChunk();
    const processed = processData(data);
    try {
        yield sendProcessedData(processed);
        showOkResult();
    } catch (err) {
        showError();
    }
}

This is the first part of a series: Iterators and generators.

Iterators

So, an iterator is an interface that provides sequential access to data.

As you can see, the definition doesn't mention anything about data structures or memory. Indeed, a sequence of null values can be represented as an iterator without occupying memory space.

Let's have a few examples:

Array is probably a first thing that comes to mind when you think about iterators. It's a data structure that stores a sequence of values in memory. It's also an iterator, because it provides sequential access to its elements.

const arr = [1, 2, 3];
for (const item of arr) {
    console.log(item);
}

The same goes for strings. They are stored in memory as a sequence of characters and provide sequential access to them.

const str = 'abc';
for (const char of str) {
    console.log(char);
}

Let's look at the following function example:

const fn = () => Math.random();

This function can be considered an iterator, because it provides sequential access to random numbers.

Then why do we need iterators if arrays, one of the basic data structures in the language, allow us to work with data both sequentially and in arbitrary order?

Let's imagine that we need an iterator that implements a sequence of natural numbers or Fibonacci numbers or any other infinite sequence. It would be difficult to store an infinite sequence in an array. We would need a mechanism for gradually populating the array with data and removing old data to prevent filling up the entire memory of the process. This unnecessary complexity adds extra implementation and maintenance overhead, whereas a solution without an array can be achieved in just a few lines of code:

const getNaturalRow = () => {
    let current = 0;
    return () => ++current;
};

Iterators can also be used to represent data retrieved from an external channel, such as a WebSocket.

In JavaScript, any object that has a next() method, which returns a structure with value (the current iterator value) and done (a flag indicating the end of the sequence), is considered an iterator. This convention is described in the ECMAScript language standard. Such an object implements the Iterator interface. Let's rewrite the previous example in this format:

const getNaturalRow = () => {
    let current = 0;
    return {
        next: () => ({ value: ++current, done: false }),
    };
};

In JavaScript, there is also the Iterable interface. It represents an object that has a @@iterator method (accessible via Symbol.iterator constant) that returns an iterator. Objects that implement this interface can be iterated using the for..of loop. Let's rewrite our example once again, this time as an Iterable implementation:

const naturalRowIterator = {
    [Symbol.iterator]: () => ({
        _current: 0,
        next() { return {
            value: ++this._current,
            done: this._current > 3,
       }},
   }),
}

for (num of naturalRowIterator) {
    console.log(num);
}
// output: 1, 2, 3

As you can see, we had to make flag "done" to change at some point, otherwise the loop would be infinite.

Generators

The next stage in the evolution of iterators was the introduction of generators. They provide syntactic sugar that allows values of an iterator to be returned as if they were the result of a function. A generator is a function declared with an asterisk function* and returns an iterator. The iterator itself is not explicitly returned; instead, the function yields values of the iterator using the yield keyword. When the function completes its execution, the iterator is considered done (subsequent next method calls will return { done: true, value: undefined }.

function* naturalRowGenerator() {
    let current = 1;
    while (current <= 3) {
        yield current;
        current++;
    }
}

for (num of naturalRowGenerator()) {
    console.log(num);
}
// output: 1, 2, 3

Even in this simple example, the main nuance of generators is apparent: the code inside a generator function does not execute synchronously. The execution of a generator code occurs incrementally, as a result of next method calls on the corresponding iterator. Let's examine how the generator code executes using the previous example. We will use a special cursor to mark where the execution of the generator is paused.

At the moment of calling naturalRowGenerator, an iterator is created.

function* naturalRowGenerator() {
    ▷let current = 1;
    while (current <= 3) {
        yield current;
        current++;
    }
}

Next, when we call the next method three times, or in our case, iterate through the loop three times, the cursor is positioned after the yield statement.

function* naturalRowGenerator() {
    let current = 1;
    while (current <= 3) {
        yield current; ▷
        current++;
    }
}

And for all subsequent next calls, as well as after exiting the loop, the generator completes its execution. The result of subsequent next calls will be { value: undefined, done: true }.

Passing parameters to iterators

Let's imagine that we need to add the functionality to reset the current counter and start counting from the beginning in our iterator of natural numbers.

naturalRowIterator.next() // 1
naturalRowIterator.next() // 2
naturalRowIterator.next(true) // 1
naturalRowIterator.next() // 2

It's clear how to handle such a parameter in a custom iterator, but what about generators?
It turns out that generators support parameter passing!

function* naturalRowGenerator() {
    let current = 1;
    while (true) {
        const reset = yield current;
        if (reset) {
          current = 1;
        } else {
          current++;
        }
    }
}

The passed parameter becomes available as the result of the yield operator. Let's try to clarify this using the cursor approach. At the moment of creating the iterator, nothing has changed. Now let's stop after the first next method call:

function* naturalRowGenerator() {
    let current = 1;
    while (true) {
        const reset = ▷yield current;
        if (reset) {
          current = 1;
        } else {
          current++;
        }
    }
}

The cursor is positioned after returning from the yield operator. With the next next call, the value passed into the function will set the value of the reset variable. But what happens to the value passed in the very first next call? It goes nowhere! If you need to pass an initial value to the generator, you can do so through the generator's arguments. Here's an example:

function* naturalRowGenerator(start = 1) {
    let current = start;
    while (true) {
        const reset = yield current;
        if (reset) {
          current = start;
        } else {
          current++;
        }
    }
}

const iterator = naturalRowGenerator(10);
iterator.next() // 10
iterator.next() // 11
iterator.next(true) // 10

Conclusion

We have explored the concept of iterators and their implementation in JavaScript. Additionally, we have learned about generators, a syntactic construct for conveniently implementing iterators.

Although in this article, I provided examples with numeric sequences, iterators in JavaScript can solve a wide range of tasks. They can represent any data sequence and even many finite state machines. In the next article, I would like to discuss how generators can be used to build asynchronous processes (coroutines, goroutines, CSP, etc.).

Backend without ORM

Eduard Dyckman — Sat, 10 Jun 2023 11:23:40 +0000

Few words about ORM

ORM (Object-relational mapping) is a programming technique that allows you to map data from external storage (such as databases) to your local programming objects.
This approach allows developers to abstract from database specificities and to perform CRUD operations as common function calls. This is beneficial since we don't have to know actual database syntax, some ORMs can even work on SQL and NoSQL databases simultaneously.
But this article's title doesn't say "Why everyone should always ORMs", so I want to point out the downsides of ORMs and suggest an alternative approach.

Why you don't need ORM

Communication with database

The lowest level representation of the database data is pure bytes (well it's actually electrons stored and moved in wires and semiconductors if we go all the way down, but we will stay in the software field). So when any driver talks to the actual database over the network it sends and receives packets of bytes. Further, based on the database protocol, which describes how to read and process these bytes, the driver transforms the bytes into meaningful data structures. So, as we can see, the database driver works sort of like an ORM here: it maps the raw data into your programming language constructs. And ORM maps this data even further into different constructs. The data received from the driver is ready to be used in your application. That is the first point on why you don't need ORM.

Raw data is powerful

So you read some data from the database and you get arrays (or lists, or sequences, or an iterator, depending on your language and driver of choice) in return. Right of the box, you have powerful tools to work with that data, because your standard library contains many functions that work on arrays and maps. On contrary, these functions may or may not work on ORM-specific objects. Also, raw data is perfectly serializable and is ready to be transferred via wire.

Does your ORM support tests

When you write unit tests for entities that work with your stored data, ideally you want to be ignorant of the database layer. Raw data is decoupled from its origin by nature and can be mocked as you prefer.
ORM, on the other hand, is dependent on its implementation and may support some mock storage, otherwise, you have to always execute tests in front of some database.

ORM libraries authors don't know about your domain

But you do. Sometimes more specific solutions can be beneficial for your project. Maybe there is some addition to the generated SQL, that can vastly improve the speed of querying.

Databases can be complex

Beyond simple CRUD operations database can support many amazing features, like sharding, different index types, non-standard conditional operators, and non-standard built-in functions. Probably an ORM library lacks the support for many features of the database you are using.
Also, if you pick an ORM library and build upon it, you are tieing yourself with that library and at some point, if there is a database solution that may be beneficial to your application domain, you may end up in a situation where you need to massively rework your codebase.

Rich said that ORM is bad

Seriously, if you haven't seen "Simple made easy" talk by Rich Hickey, do yourself a favor, it's very enlightening.

How to build without ORM

Let's figure out how to build our projects with the "no ORM" approach. We want to build a solution that is testable separately from the database, operates on data, and has layers with well-defined boundaries (kudos to Uncle Bob).
I will build an example using node, typescript, and neon (postgres). It will contain a single entity - User with attributes name and age.

Model layer

So how do we represent a model within our concept? It's just an interface!
link

export type User = {
  id: number;
  name: string;
  age: number;
};

If later we want to add some methods, that operate on User somehow, then we just write a function that accepts User as a parameter or returns a User shaped object.
Now we want to connect the User entity to the database: store it, retrieve it, modify and delete it.

Repository layer

Now let's add the entity Repository. It will have CRUD methods that operate on raw data and will perform database operations.
link

export type Repository<T extends Record<string, any>> = {
  name(): string;
  create(data: Omit<T, "id">): Promise<T>;
  readMany<K extends keyof T>(filters: Filter<T, K>[]): Promise<T[]>;
  read(id: string): Promise<T>;
  update(id: string, data: T): Promise<T>;
  delete(id: string): Promise<T>;
};

The Repository is our missing database operations layer. It can perform all of the "get, store data" stuff. Also, if some entities require additional operations we can extend the basic Repository for them with additional operations like createOrReplace or updateMany.

Database adapter

Database adapter would be the function that returns a Repository instance. Here is an example of how it can look for the postgres database. link

If we want to test an adapter we probably would do it against a real database in a snapshot manner with something like these steps:

Create a test repository using the tested adapter
Run some methods
Check that the database state matches our expectations

link

Our Repository is pretty useful

Actually, our Repository instance is not only good for database management. For example, we can build REST API from it. link
And write a test for that implementation as well. link

Conclusions

I gotta be honest

Fairly, the current state of ORMs is not as bad as it was before. There are modern frameworks like Prisma, that acknowledge some of the problems I described above.
For example, Prisma generates a client, that returns just typed data when using built-in query methods, so it is decoupled by design.
I'd say it is a good sign when you can view the database framework as an advanced driver.
Still, using Prisma makes it very hard to migrate to an unsupported database, so you need to keep that in mind.

The TL;DR

Probably the main thought I want to convey is: design layers of your application in advance and with thinking ahead. And since it's almost impossible to predict the functionalities your application should provide in five years, two years, or one year you need to create a very flexible architecture, and layering your designs with well-defined boundaries is the best approach from what I know.