DEV Community: Martin Adámek

MikroORM 6: Polished

Martin Adámek — Mon, 08 Jan 2024 08:01:49 +0000

After more than a year in the development, I am thrilled to announce the next major version of MikroORM has just become stable. It brings many improvements throughout the whole system, and doubles down on type-safety and strictness.

In case you don’t know…

If you never heard of MikroORM, it’s a TypeScript data-mapper ORM with Unit of Work and Identity Map. It supports MongoDB, MySQL, PostgreSQL, and SQLite drivers currently. Key features of the ORM are:

You can read the full introductory article here (but note that many things have changed since that was written) or browse through the docs.

Quick summary of 5.x releases

Before we dive into all the things v6, let’s mention some of the important additions from 5.x feature releases:

em.upsert() and em.upsertMany()
custom pivot table entity
automatic relation discovery
fulltext search support
explicit serialization
rel() and ref() helpers
new Collection helpers (map/filter/reduce/exists/find/indexBy/...)

But enough of the history lesson, let’s talk about the future!

Type safety

One of the biggest improvements in v6 is by far the overhauled typing. While v5 brought a base for this with strict populate hint and the Loaded type, v6 doubles down on it. Many of the internal types have been refactored to improve both strictness and autocomplete capabilities. So what actually changed?

Strict partial loading

The most visible part is the partial loading, also known as the fields option. Let's take a look at this example:

// article is typed to `Loaded<Article, never, 'title' | 'author.email'>`
const article = await em.findOneOrFail(Article, 1, { 
  fields: ['title', 'author.email'],
});

const id = article.id; // ok, PK is selected automatically
const title = article.title; // ok, title is selected
const publisher = article.publisher; // fail, not selected
const author = article.author.id; // ok, PK is selected automatically
const email = article.author.email; // ok, selected
const name = article.author.name; // fail, not selected

The Loaded type now understands partial loading too, and this example will fail to compile because of accessing the author's name which is not loaded. Note that we also skipped the populate hint from this example, as it is inferred from our partial loading hint.

What if you wanted to exclude just a few columns instead of white-listing what you want to load? We got you covered, v6 adds a new exclude option which does exactly that - and it is strictly typed as well!

// article is typed to `Loaded<User, never, never, 'email'>`
const user = await em.findOneOrFail(User, 1, { 
  exclude: ['email'],
});

const id = user.id; // ok, PK is selected automatically
const name = user.name; // ok, selected
const email = user.email; // fail, excluded

Check out the live demo on StackBlitz.

`Opt` type

While v5 introduced the strict typing for em.create(), it was a bit cumbersome, as we now have to distinguish properties with a runtime default (so technically optional properties, but on type level they are seen as required). A new symbol called OptionalProps was introduced to mark such defaults, so they are not required in the em.create type. The symbol approach was mainly problematic when you wanted to define some properties like this in a custom base entity.

In v6, you can leverage the new Opt type, which is used on property level (as opposed to the entity level OptionalProps symbol). This effectively removes the problems with extensions and added generics. You can use the type in two ways:

with generics: middleName: Opt<string> = '';
with intersections: middleName: string & Opt = '';

Both will work the same, and can be combined with the OptionalProps symbol approach.

import { Opt, Entity, PrimaryKey, Property } from '@mikro-orm/core';

@Entity()
class User {

  @PrimaryKey()
  id!: number;

  @Property()
  firstName!: string;

  @Property()
  // highlight-next-line
  middleName: string & Opt = '';

  @Property()
  lastName!: string;

}

`Hidden` type

Similarly to the Opt type used for marking optional properties, we have the Hidden type (and HiddenProps symbol) for marking properties that should be hidden when serializing.

@Entity()
class Book {

  @Property({ hidden: true })
  hiddenField: Hidden<Date> = Date.now();

  @Property({ hidden: true, nullable: true })
  otherHiddenField?: string & Hidden;

}

Those properties won't be accessible on the DTO:

const book = await em.findOneOrFail(Book, 1);
const bookDTO = wrap(book).toObject();

bookDTO.hiddenField; // fails

Populating all relations

Previously, you were allowed to populate all relations via populate: true, but it wasn't type-safe - the resulting Loaded type was not respecting this option. In v6, you can use populate: ['*'] which will work with the Loaded type correctly.

const user = await em.findOneOrFail(User, 1, { populate: ['*'] });

The populate hint now also accepts false as a way to disable eager loading of relations (those marked with eager: true).

Populate based on filter

When you filter by a nested relation value, the target table is automatically joined, but nothing is selected, the join is only used for the where condition. In v6, you can use populate: ['$infer'] to automatically populate such relations:

// this will populate all the books and their authors, all via a single query
const tags = await em.find(BookTag, {
  books: { author: { name: '...' } },
}, { 
  populate: ['$infer'],
});

Primary key type inference

If you use composite keys or non-standard primary key names, you probably know about PrimaryKeyType and PrimaryKeyProp symbols. While they worked fine, there was no need to have two of them—and people were often confused how they work, as one required a union type of primary property names, while the other was a tuple type. This is now consolidated into a single PrimaryKeyProp symbol, which accepts a tuple with property names

@Entity()
class Foo {

  @ManyToOne(() => Bar, { primary: true })
  bar!: Bar;

  @ManyToOne(() => Baz, { primary: true })
  baz!: Baz;

-  [PrimaryKeyType]?: [number, number];
-  [PrimaryKeyProp]?: 'bar' | 'baz';
+  [PrimaryKeyProp]?: ['bar', 'baz'];

}

Some methods and interfaces like Ref allowed you to pass in the primary key property via second generic type argument, this is now also removed in favor of the automatic inference.

Simplified `BaseEntity`

The optional ORM BaseEntity used to have two generic parameters, one for the entity type and the other for the primary key type. They are both removed in v6. The former has been replaced with this type, the latter with the PrimaryKeyProp symbol.

-class User extends BaseEntity<User> { ... }
+class User extends BaseEntity { ... }

Implicit serialization

Next, let's talk about the changes in serialization. There are two ways to serialize your entities—implicit via wrap(entity).toObject(), which is called automatically when you do JSON.stringify(entity), and explicit via serialize() helper.

Implicit serialization now works entirely based on populate and fields hints. This means that, unless you explicitly marked some entity as populated via wrap(entity).populated(), it will be part of the serialized form only if it was part of the populate hint:

// let's say both Author and Book entity has a M:1 relation to Publisher entity
// we only populate the publisher relation of the Book entity
const user = await em.findOneOrFail(Author, 1, {
  populate: ['books.publisher'],
});

const dto = wrap(user).toObject();
console.log(dto.publisher); // only the FK, e.g. `123`
console.log(dto.books[0].publisher); // populated, e.g. `{ id: 123, name: '...' }`

Moreover, the implicit serialization now respects the partial loading hints too. Previously, all loaded properties were serialized, and partial loading worked only on the database query level. Since v6, we also prune the data on runtime. This means that unless the property is part of the partial loading hint (fields option), it won't be part of the DTO - only exception is the primary key, you can optionally hide it via hidden: true in the property options. The main difference here will be the foreign keys, those are often automatically selected as they are needed to build the entity graph, but will no longer be part of the DTO.

const user = await em.findOneOrFail(Author, 1, {
  fields: ['books.publisher.name'],
});

const dto = wrap(user).toObject();
// only the publisher's name will be available, previously there would be also `book.author`
// `{ id: 1, books: [{ id: 2, publisher: { id: 3, name: '...' } }] }`

This also works for embeddables, including nesting and object mode. And speaking of embeddables—they now also support the fieldName option, again, including the nesting and object mode, effectively allowing partial loading on the object embeddables (so JSON properties) too.

`forceObject`

When you serialize an entity with unpopulated relation, it will result in a foreign key value, e.g. book.author will be number if you don't populate the author relation. In v6, you can use the forceObject serialization option to get an object there instead, e.g. book.author will be { id: 1 } instead of just 1.

To have the DTO properly typed, you can use the Config symbol, preferably in your own base entity, as this flag will affect all your entities globally:

import { Config, DefineConfig, PrimaryKey } from '@mikro-orm/core';

class BaseEntity {

  // highlight-next-line
  [Config]?: DefineConfig<{ forceObject: true }>;

  @PrimaryKey()
  id!: number;

}

The DefineConfig type will offer intellisense to the type config options. Right now, it only accepts a single property, but there might be more options like this going forward.

Joined strategy

The joined loading strategy was around for a while, but it had several implementation problems resulting in different behavior when compared to the default select-in strategy. But that actually changes now, the joined strategy is back on track and should be completely aligned with the select-in behavior.

So what actually changed? The most important part is the support for populateWhere: 'all', which is the default behavior, and means "populate the full relations regardless of the where condition". This was previously not working with the joined strategy, as it was reusing the same join clauses as the where clause. In v6, the joined strategy will use a separate join branch for the populated relations.

Since the strategies now behave the same, this finally unlocked the switch of the defaults for all the SQL drivers—the joined strategy is the new default. The joined strategy should usually be faster unless you join a lot of to-many relations (which would result in huge cartesian products).

Filters on relations

Filters are now also applied to the relations, as part of JOIN ON condition. If a filter exists on a M:1 or 1:1 relation target, such an entity will be automatically joined, and when the foreign key is defined as NOT NULL, it will result in an INNER JOIN rather than LEFT JOIN. This is especially important for implementing soft deletes via filters, as the foreign key might point to a soft-deleted entity. When this happens, the automatic INNER JOIN will result in such a record not being returned at all.

Cursor-based pagination

As an alternative to the offset-based pagination with limit and offset, you can now paginate based on a cursor. A cursor is an opaque string that defines a specific place in ordered entity graph. You can use em.findByCursor() to access those options. Under the hood, it will call em.find() and em.count() just like the em.findAndCount() method, but will use the cursor options instead.

const currentCursor = await em.findByCursor(User, {}, {
  first: 10,
  after: previousCursor, // cursor instance
  orderBy: { id: 'desc' },
});

// to fetch next page
const nextCursor = await em.findByCursor(User, {}, {
  first: 10,
  after: currentCursor.endCursor, // opaque string
  orderBy: { id: 'desc' },
});

// to fetch next page
const nextCursor2 = await em.findByCursor(User, {}, {
  first: 10,
  after: { id: lastSeenId }, // entity-like POJO
  orderBy: { id: 'desc' },
});

The Cursor object provides the following interface:

Cursor<User> {
  items: [
    User { ... },
    User { ... },
    User { ... },
    ...
  ],
  totalCount: 50,
  length: 10,
  startCursor: 'WzRd',
  endCursor: 'WzZd',
  hasPrevPage: true,
  hasNextPage: true,
}

Raw SQL fragments

The raw SQL fragments used to be detected automatically, which wasn't very precise. In v6, a new raw static helper is introduced to deal with this:

const users = await em.find(User, {
  [raw('lower(email)')]: 'foo@bar.baz',
});

This helper now replaces the removed expr() function, which was only an escape hatch for strictly typed FilterQuery, but wasn't required on runtime. It offers similar API, e.g. you can pass in a callback and get the current alias (based on the scope of execution) for given column:

const users = await em.find(User, {
  books: {
    [raw(alias => `lower(${alias}.title)`)]: 'some title'
  },
});

Unlike in v5, this is now required way to mark your raw SQL fragments. Without it, you'd end up with the fragment being quoted as a regular string value.

The raw query can be also parametric, you can use ? for values and ?? for keys:

const users = await em.find(User, {
  // this will result in properly quoted sql, e.g. `lower("email")`
  [raw('lower(??)', ['email'])]: 'foo@bar.baz',
});

And while the raw helper is the most universal one you can use, there is also a new sql tagged template function, which resolves to it too, if you prefer that kind of interface:

const users = await em.find(User, { [sql`lower(email)`]: 'foo@bar.baz' });

The fragments can be also used in your entity definition, to set raw database defaults. This is basically a shortcut for prop.defaultRaw option:

@Property({ default: sql`now()` })
createdAt = new Date();

And there is more to this, the sql function also offers several helper functions you can use, namely:

sql.ref()
sql.now()
sql.lower()
sql.upper()

Read more about this in Using raw SQL query fragments section.

Subquery operators `$some`, `$none` and `$every`

In addition to the regular operators that translate to a real SQL operator expression (e.g. >=), you can also use the following collection operators:

operator	description
`$some`	Finds collections that have some record matching the condition.
`$none`	Finds collections that have no records matching the condition.
`$every`	Finds collections where every record is matching the condition.

This will be resolved as a subquery condition:

// finds all authors that have some book called `Foo`
const res1 = await em.find(Author, {
  books: { $some: { title: 'Foo' } },
});

// finds all authors that have no books called `Foo`
const res2 = await em.find(Author, {
  books: { $none: { title: 'Foo' } },
});

// finds all authors that have every book called `Foo`
const res3 = await em.find(Author, {
  books: { $every: { title: 'Foo' } },
});

The condition object can be also empty:

// finds all authors that have at least one book
const res1 = await em.find(Author, {
  books: { $some: {} },
});

// finds all authors that have no books
const res2 = await em.find(Author, {
  books: { $none: {} },
});

Subquery joining

Subqueries are now better supported all over the place. Namely, you can join a subquery, as well as use a subquery in qb.from() method. One use case where this is handy is when you want to limit a joined relation, e.g. you have a 1:M collection, and you are interested only in the first item. In the following example, we join on the Author.books collection, overriding the implicit join branch with a custom subquery that has a limit 1 on it.

// subquery can be a knex query builder as well
const subquery = await em.createQueryBuilder(Book, 'b')
  .where({ ... })
  .orderBy({ title: 'asc' }).limit(1);

const authors = await em.createQueryBuilder(Author, 'a')
  .select('*')
  // pass in both the property path and the subquery into the first argument as a tuple
  .leftJoinAndSelect(['a.books', subquery], 'b')
  // you can join more relations on top of the subquery join
  .leftJoinAndSelect('b.tags', 't')
  .getResultList();

Dataloader support for references and collections

MikroORM now provide out-of-box support for loading Reference and Collection properties via dataloader. This feature needs to be enabled either globally (via ORM config) or locally (via FindOptions):

await MikroORM.init({
  dataloader: true,
});

Then you can use Promise.all on such objects, and it will automatically resolve to a batched select query:

const authors = await orm.em.find(Author, [1, 2, 3]);
await Promise.all(authors.map(author => author.books.load()));

// or when the dataloader support is not enabled globally:
await Promise.all(authors.map(author => author.books.load({ dataloader: true })));

This is especially useful with GraphQL since it automatically solves its notorious N+1 problem, without you even noticing it: you won't even need Promise.all since all the requests will occur within a single tick of the event loop and will be coalesced by the dataloader library.

More about this in the new dataloader section. You can also check out this example repository which leverages the dataloader (as well as Accounts.js library).

Shout out to Niccolò Belli, who contributed this feature and is working on a more advanced version which supports dataloader also for em.find().

Logging improvements

Logging support has been greatly improved. You can now set up a custom logger context:

const res = await em.findAll(Author, { loggerContext: { meaningOfLife: 42 } });

// ...

class CustomLogger extends DefaultLogger {
  log(namespace: LoggerNamespace, message: string, context?: LogContext) {
    console.log(context?.meaningOfLife);
    // 42
  }
}

This context can be specific to the EntityManager fork, and will get the EntityManager ID automatically, so you can now track which request context/fork fired what queries.

const fork = em.fork({ loggerContext: { meaningOfLife: 42 } });
console.log(fork.id); // 3
// the logger context here will be { id: 3, meaningOfLife: 42 } 
const res = await fork.findAll(Author);

The logger also supports query labels (simple way to alter what gets printed), index hints and query comments, and more.

const author = await em.findOne(Author, { id: 1 }, { logging: { label: 'Author Retrieval - /authors/me' } });
// [query] (Author Retrieval - /authors/me) select "a0".* from "author" as "a0" where "a0"."id" = 1 limit 1 [took 2 ms]

The label can be also set via loggerContext.

Logging can be now selectively enabled/disabled via FindOptions. this works in both ways, if you globally disable logging, you can selectively enable it via FindOptions, as well as the other way around.

// MikroORM.init({ debug: true });
const author = await em.findOne(Author, { id: 1 }, { logging: { enabled: false } });
// Overrides config and displays no logger output

// ...

// MikroORM.init({ debug: false });
const author = await em.findOne(Author, { id: 1 }, { logging: { enabled: true } });
// Overrides config and displays logger output

// ...

// MikroORM.init({ debug: ['query-labels'] });
const author = await em.findOne(Author, { id: 1 }, { logging: { debugMode: ['query'] } });
// Overrides config and displays logger output for query

Read more about the logger improvements in the logging section.

Improved change-tracking of M:N relations

M:N relations were always a bit problematic, the way they were implemented was only checking the owning side for changes. Thanks to the propagation of changes, it allowed working with the inverse side too, as long as the items you added/removed from the collection were loaded.

const tag = await em.findOne(BookTag, 1);
// tag.books in an inverse side, this used to fail, but now it works!
tag.books.add(em.getReference(Book, 123));
await em.flush();

This restriction is no longer valid, and changes made to inverse sides of M:N collections are also tracked. Moreover, all queries that are altering pivot tables are now properly batched.

Extending `EntityManager`

It is now possible to extend the EntityManager with your own custom methods. The type is inferred automatically from the config if possible.

import { MikroORM, EntityManager } from '@mikro-orm/sqlite';

class MyEntityManager extends EntityManager {

  myCustomMethod(base: number): number {
    return base * Math.random();
  }

}

const orm = await MikroORM.init({
  entities: [...],
  dbName: ':memory:',
  // highlight-next-line
  entityManager: MyEntityManager,
});
console.log(orm.em instanceof MyEntityManager); // true
const res = orm.em.myCustomMethod(123);

`GeneratedCacheAdapter` for production usage

One of the ways you can define your entity metadata is leveraging the TypeScript compiler API via ts-morph, which allows extracting the type information that would be otherwise lost on compilation (and is not available via reflect-metadata). While this approach works nice locally, it had several hard problems around it, the most obvious one is the dependency on TypeScript, which you don't want to have in your production builds.

In v6, MikroORM lets you generate a production cache bundle into a single JSON file via CLI:

npx mikro-orm cache:generate --combined

This will create ./temp/metadata.json file which can be used together with GeneratedCacheAdapter in your production configuration:

import { GeneratedCacheAdapter, MikroORM } from '@mikro-orm/core';

await MikroORM.init({
  metadataCache: {
    enabled: true,
    adapter: GeneratedCacheAdapter,
    options: { data: require('./temp/metadata.json') },
  },
  // ...
});

This way you can keep the @mikro-orm/reflection package as a development dependency only, use the CLI to create the cache bundle, and depend only on that in your production build.

The cache bundle can be statically imported, which is handy in case you are using some bundler.

Entity Generator improvements

EntityGenerator now automatically detects more M:N relations—including those with an autoincrement primary key (so fixed order), or even unrelated additional columns. Over time, we might get closer to a proper schema-first approach.

Shout out to Vasil Rangelov, who contributed this feature and is working on more improvements in the EntityGenerator, e.g. ability to override the generated entities' metadata.

Inference of default values

When defining properties with a runtime default value, the reflect-metadata provider fails to infer the type property. This is no longer a problem in v6, as the discovery mechanism now automatically tries to infer the type from the runtime defaults.

@Property()
-created: Data = new Date();
+created = new Date();

Note that this works only if your entity can be constructed without any constructor parameters. It is fine to have them, but the constructor cannot fail if they are not provided for this auto-detection to work.

Other notable changes

Virtual entities now allow M:1 and 1:1 relations.
New MikroORM.initSync() method allows initializing the ORM synchronously.
Propagation and change-tracking works with useDefineForClassFields enabled.
Removed static require calls that were problematic when bundling.
Removed persist/remove/flush from repository interface.
The type option is removed in favor of driver exports and defineConfig.
ORM extensions—a way to tell the ORM about optional dependencies like Seeder or Migrator without the need to declare peer dependencies.
All drivers now re-export the @mikro-orm/core package, so you no longer have to think about which package to import from - just use the driver package.
Native BigInt support, allowing to set the mapping to either bigint, number or string.
Embedded properties respect NamingStrategy, including object embeddables.
Entities are added to the identity map on em.persist() if they have a primary key.
Native support for generated columns.
Support for lateral sub-query joins.
Support for native enums in postgres.
Support Ref wrapper on scalar properties.
Discovery hooks onMetadata and afterDiscovered allowing to modify the metadata in any way you want.

And many many more, see the full changelog here. Also be sure to check the upgrading guide.

One more thing…

Over time, while some people liked the current documentation, there were also people disliking it. It wasn't really beginner-friendly, as it only described the distinct features, but was lacking some tutorials describing how to set things up as a whole.

A lot of the documentation for v6 has been updated and polished, and a completely new Getting Started Guide was added, accompanied by an example repository. It describes how to build and test an API from scratch with MikroORM, Fastify, ESM, Vitest, JWT, and some other tools. Unlike the rest of the docs, you can read it from top to bottom as a tutorial. I will continue extending the guide over time, especially the final section about type safety.

Like MikroORM? ⭐️ Star it on GitHub and share this article with your friends. If you want to support the project financially, you can do so via GitHub Sponsors.

MikroORM 5: Stricter, Safer, Smarter

Martin Adámek — Sun, 06 Feb 2022 18:59:29 +0000

The next major version of MikroORM has been just released. The title says: Stricter, Safer, Smarter — why?

Greatly improved type safety (e.g. populate and partial loading hints)
Auto-flush mode (so we never lose in-memory changes)
Automatic refreshing of loaded entities (say goodby to refresh: true)
Reworked schema diffing with automatic down migrations support
and many many more…

This time it took almost a year to get here — initial work on v5 started back in March 2021.

In case you don’t know…

You can read the full introductory article here (but note that many things have changed since that was written) or browse through the docs.

Quick summary of 4.x releases

Before we dive into all the things v5, let’s recap what happened in 4.x releases:

Result cache
Automatic transaction context
Nested embeddables and many other improvements in this domain
Using env vars for configuration

But enough of the history lesson, let’s talk about the future!

Improved type safety

Let’s jump right into the most interesting feature — strict typing (almost) everywhere! em.create(), toJSON(), toObject(), populate, partial loading, and order by hints, all of that (and even more!) is now strictly typed.

Let’s check the following example:

First, we use em.create() to build the whole entity graph in a single step. It will validate the payload for both types and optionality. Some properties on the entity might have default values provided via hooks or database functions — while we might want to define them as required properties, they should act as optional in the context of em.create(). To deal with this problem, we can specify such properties that should be considered as optional via OptionalProps symbol:

Some property names are always considered as optional: id, _id, uuid.

Then we load all Author entities, populating their books and the book tags. All of the FindOptions here are strictly typed, moreover, we could even skip the populate hint as it can be inferred from fields option automatically.

We might still need some type casting for DTOs. The serialized form of an entity can be very unpredictable — there are many variables that define how an entity will be serialized, e.g. loaded relation vs reference, property serializers, lazy properties, custom entity serializer/toJSON method, eager loading, recursion checks, … Therefore, all relations on the EntityDTO type are considered as loaded, this is mainly done to allow better DX as if we had all relations typed as Primary | EntityDTO (e.g. number | EntityDTO), it would be impossible to benefit from intellisense/autosuggestions. Imagine this scenario:

Validation improvements

Adding on top of the compile-time validation, we also get a runtime validation right before insert queries are fired, to ensure required properties have their values. This is important mainly in mongo, where we don’t have optionality checks on the schema level.

When we try to use the CLI without installing it locally, we also get a warning. And what if we forget to update some of the ORM packages and ended up with version mismatch and multiple installed core packages? We now validate that too!

Reworked schema diffing

Schema diffing has been one of the weakest spots. Often, additional queries were produced or it was even impossible to get to a fully synchronized state.

Schema diffing has been completely reworked to address all currently known issues, and adding a bit more on top of that:

Diffing foreign key constraints
Proper index diffing (before we compared just names)
Custom index expressions
Comment diffing
Column length diffing (e.g. numeric(10,2) or varchar(100))
Changing primary key types
Schema/namespace diffing (Postgres only)
Automatic down migrations (no SQLite support yet)
Check constraints support (Postgres only)

Smarter migrations

In the production environment, we might want to use compiled migration files. Since v5, this should work almost out of the box, all we need to do is to configure the migrations path accordingly. Executed migrations now ignore the file extension, so we can use both node and ts-node on the same database. This is done in a backward-compatible manner.

Creating new migration will now automatically save the target schema snapshot into the migrations folder. This snapshot will be then used if we try creating a new migration, instead of using the current database schema. This means that if we try to create new migration before we run the pending ones, we still get the right schema diff (and no migration will be created if no additional changes were made).

Snapshots should be versioned just like the regular migration files.

Auto-flush mode

Up until now, flushing was always an explicit action. With v5, we can configure the flushing strategy, similarly to how JPA/hibernate work. We have 3 flush modes:

FlushMode.COMMIT - The EntityManager tries to delay the flush until the current transaction is committed, although it might flush prematurely too.
FlushMode.AUTO - This is the default mode, and it flushes the EntityManager only if necessary.
FlushMode.ALWAYS - Flushes the EntityManager before every query.

FlushMode.AUTO will try to detect changes on the entity we are querying, and flush if there is an overlap:

More about flush modes in the docs.

Automatic refreshing of loaded entities

Previously, when an entity was loaded and we needed to reload it, providing explicit refresh: true in the options was required. Refreshing of entity also had one problematic side effect — the entity data (used for computing changesets) were always updated based on the newly loaded entity, hence forgetting the previous state (resulting in possibly lost updates done on the entity before refreshing).

Now we always merge the newly loaded data with the current state, and when we see an updated property, we keep the changed value instead. Moreover, for em.findOne() with a primary key condition, we try to detect whether it makes sense to reload an entity, by comparing the options and already loaded property names. In this step the fields and populate options are taken into account to support both partial loading and lazy properties.

For complex conditions in em.findOne() and for any queries via em.find(), we always do the query anyway, but now instead of ignoring the data in case such entity was loaded, we merge them in the same manner.

Seeder package

MikroORM v5 now has a new package for seeding your database with initial or testing data. It allows creating entities via the same EntityManager API as usual, adding support for entity factories, and generating fake data via faker (the newly release community version).

See the seeder docs for more examples.

Polymorphic embeddables

Polymorphic embeddables allow us to define multiple classes for a single embedded property and the right one will be used based on the discriminator column, similar to how single table inheritance works. While this currently works only for embeddables, support for polymorphic entities will be probably added in one of the 5.x releases.

Check out the documentation for a complete example.

There are many other small improvements in embeddables, as well as many issues were addressed. Two examples:

Support for many-to-one relations (storing only primary key and being able to populate the relation same as with regular entities)
Support for onCreate and onUpdate property options

Populating lazy scalar properties

Previously, the only way to populate a lazy scalar property was during the initial load of containing entity. If such entity was already loaded in the identity map (without this property), we needed to refresh its state — and potentially lose some state. MikroORM v5 allows to populate such properties via em.populate() too. Doing so will never override any in-memory changes we might have done on the entity.

Creating references without EntityManager

When we wanted to create a reference, so an entity that is represented only by its primary key, we always had to have access to the current EntityManager instance, as such entity always needed to be managed.

Thanks to the new helper methods on the Reference class, we can now create entity references without access to EntityManager. This can be handy if you want to create a reference from an inside entity constructor:

The Reference wrapper is an optional class to allow more type safety over relationships. Alternatively, we can use Reference.createNakedFromPK().

This will create an unmanaged reference, that will be then merged to the EntityManager once owning entity gets flushed. Note that before we flush it, methods like Reference.init() or Reference.load() won’t be available as they require the EntityManager instance.

Smarter expr helper

The expr() helper can be used to get around strict typing. It was an identity function, doing nothing more than returning its parameter — all it did was to tell TypeScript the value is actually of a different type (a generic string to be precise).

We can now use the helper in two more ways:

With a callback signature to allow dynamic aliasing of the expression
With an array argument to allow comparing tuples

Awaitable QueryBuilder

QueryBuilder is now aware of its type, and the getResult() and execute() methods are typed based on it. We can also await the QueryBuilder instance directly, which will automatically execute the QB and return the appropriate response. The QB instance is now typed based on usage of select/insert/update/delete/truncate methods to one of:

SelectQueryBuilder — awaiting yields array of entities
CountQueryBuilder — awaiting yields number
InsertQueryBuilder — awaiting yields QueryResult
UpdateQueryBuilder — awaiting yields QueryResult
DeleteQueryBuilder — awaiting yields QueryResult
TruncateQueryBuilder — awaiting yields QueryResult

Wildcard schema entities

Up until now, we were able to define entities in a specific schema, or without a schema. Such entities then used the schema based on ORM config or FindOptions. This allowed us to read entities from a specific schema, but we were missing the power of Unit of Work here.

With v5, entity instances now hold schema name (as part of WrappedEntity). Managed entities will have the schema from FindOptions or metadata. Methods that create new entity instances like em.create() or em.getReference() now have an options parameter to allow setting the schema. We can also use wrap(entity).getSchema() and wrap(entity).setSchema().

Entities can now specify wildcard schema via @Entity({ schema: '*' }). That way they will be ignored in SchemaGenerator unless the schema option is specified.

If we specify schema, the entity only exists in that schema
If we define * schema, the entity can exist in any schema, always controlled by the parameter
If we skip schema option, the value will be taken from global ORM config

More about this topic can be found here.

Deep assigning of entities

Another weak spot was assigning new values to existing entities. While wrap().assign() was originally designed to update a single entity and its values, a lot of users wanted to assign an entity graph, updating relations in a single step too.

With v5, the way how EntityAssigner detects what entity should be updated has changed. Assigning a deep entity graph should be possible by default, without any additional options. It works based on matching entity primary keys, so if you want to issue an update for a relationship instead of creating new relation, make sure you first load it and pass down its primary key to the assign helper:

If we want to always update the entity, even without the entity PK being present in data, we can use updateByPrimaryKey: false:

More examples on this topic can be found in the docs.

Experimental support for ES modules

While MikroORM v5 is still compiled and published as CommonJS, we added several improvements that should allow using it with ESM projects too. Namely, we use the gen-esm-wrapper package to allow using named imports, and we use one nasty trick to keep dynamic imports instead of compiling them to require statements — for that we need to use MIKRO_ORM_DYNAMIC_IMPORTS env var. This should allow us to use folder-based discovery with ES modules, which was previously not possible.

Other notable changes

Partial loading support (fields) for joined loading strategy
AsyncLocalStorage used by default in the RequestContext helper
onLoad event (like onInit, but allows async and fires only for loaded entities, not references)
Exporting async functions from CLI config
Configurable aliasing strategy for SQL
Allow providing custom Logger instance
persist option in em.create() and persistOnCreate global configuration
M:N support in entity generator
Support for specifying transaction isolation level
Controlling where condition for populate hints
Revamped API docs
and many many more, see the full changelog here

Also be sure to check the upgrading guide.

What’s next?

Here is a list of things I would like to focus on going forward:

allow specifying pivot entity for M:N relations (so we can have additional columns there, but still map it as M:N for reading purposes)
support for database views (or maybe just entities representing SQL expressions)
more drivers — namely better-sqlite3 and cockroach sounds like low hanging fruit, given knex now supports those natively

Like MikroORM? ⭐️ Star it on GitHub and share this article with your friends. If you want to support the project financially, you can do so via GitHub Sponsors.

MikroORM 4.1: Let’s talk about performance

Martin Adámek — Tue, 13 Oct 2020 13:13:34 +0000

I just shipped version 4.1 of MikroORM, the TypeScript ORM for Node.js, and I feel like this particular release deserves a bit more attention than a regular feature release.

In case you don’t know…

If you never heard of MikroORM, it’s a TypeScript data-mapper ORM with Unit of Work and Identity Map. It supports MongoDB, MySQL, PostgreSQL and SQLite drivers currently. Key features of the ORM are:

You can read the full introductory article here or browse through the docs.

So what changed?

This release had only one clear goal in mind — the performance. It all started with an issue pointing out that flushing 10k entities in a single unit of work is very slow. While this kind of use case was never a target for me, I started to see all the possibilities the Unit of Work pattern offers.

Batch inserts, updates and deletes

The biggest performance killer was the amount of queries — even if the query is as simple and optimised as possible, firing 10k of those will be always quite slow. For inserts and deletes, it was quite trivial to group all the queries. A bit more challenging were the updates — to batch those, MikroORM now uses case statements.

As a result, when you now flush changes made to one entity type, only one query per given operation (create/update/delete) will be executed. This brings significant difference, as we are now executing fixed number of queries (in fact the changes are batched in chunks of 300 items).

JIT compilation

Second important change in 4.1 is JIT compilation. Under the hood, MikroORM now first generates simple functions for comparing and hydrating entities, that are tailored to their metadata definition. The main difference is that those generated functions are accessing the object properties directly (e.g. o.name), instead of dynamically (e.g. o[prop.name]), as all the information from metadata are inlined there. This allows V8 to better understand the code so it is able to run it faster.

Results

Here are the results for a simple 10k entities benchmark:

In average, inserting 10k entities takes around 70ms with sqlite, updates are a tiny bit slower. You can see results for other drivers here: https://github.com/mikro-orm/benchmark.

Acknowledgement

Kudos to Marc J. Schmidt, the author of the initial issue, as without his help this would probably never happen, or at least not in near future. Thanks a lot!

Like MikroORM? ⭐️ Star it on GitHub and share this article with your friends. If you want to support the project financially, you can do so via GitHub Sponsors.

MikroORM 4: Filling the Gaps

Martin Adámek — Tue, 08 Sep 2020 19:30:04 +0000

After 4 months of active development, I am thrilled to announce the release of MikroORM 4. When I started to work on v4, the goal was to make it relatively small release, mainly to drop support for TypeScript 3.6 and Node.js 8, and to split the project into multiple packages, so we can have more fine grained control over the dependencies (mainly because of ts-morph having TS as a runtime dependency).

But what a major release would that be, without having a bunch of new features as well, right?

Photo by Ryoji Iwata on Unsplash

In case you don’t know…

You can read the full introductory article here or browse through the docs.

Quick summary of 3.x releases

Before I dive into all the things v4, let’s recap the major features that landed in 3.x releases:

Monorepo

The first major change I want to talk about is the split into multiple packages. As mentioned above, the biggest motivation for this change was to get rid of TS as a runtime dependency, when it is not needed. Another nice example is knex, which is used as a base layer for SQL driver, but has no meaning for mongodb users. Lastly, it turned out Highlight.js, that was used for query highlighting, is also quite fat and slow, so I ended up writing custom highlighters that are built for CLI and are (almost) dependency free.

In v4, there are 12 packages and 2 highlighters, you install only what you use, and you have control over what is needed in production and what is just a dev dependency. This is especially useful for serverless users, where cold start speeds matter.

It felt natural to offer some shortcuts on the EntityManager and EntityRepository level, so we now have flavours of those classes in place, that offer things like em.execute(sql) or em.aggregate(). To access those driver specific methods, be sure to use the classes from driver packages:

Database connectors like pg or sqlite3 are now dependencies of the driver packages (e. g. @mikro-orm/sqlite).

Filters

Probably the most interesting feature of v4 are filters, also known as association scopes. They allow you to define data visibility rules, both global and bound to entity. One common application of filters are soft deletes, or automatic tenant conditions.

Filters are applied to those methods of EntityManager: find(), findOne(), findAndCount(), findOneOrFail(), count(), nativeUpdate() and nativeDelete(). Filters can be parametric, the parameter can be also in form of callback (possibly async). You can also make the filter enabled by default.

Filter can be defined at the entity level, dynamically via EM (global filters) or in the ORM configuration.

Global Filters

We can also register filters dynamically via EntityManager API. We call such filters global. They are enabled by default (unless disabled via last parameter in addFilter() method), and applied to all entities. You can limit the global filter to only specified entities.

Filters as well as filter params set on the EM will be copied to all its forks.

EventSubscribers and flush events

As opposed to regular lifecycle hooks, we can now use EventSubscriber to hook to multiple entities or if you do not want to pollute the entity prototype. All methods are optional, if you omit the getSubscribedEntities() method, it means you are subscribing to all entities.

Flush events

There is a special kind of events executed during the commit phase (flush operation). They are executed before, during and after the flush, and they are not bound to any entity in particular.

beforeFlush is executed before change sets are computed, this is the only event where it is safe to persist new entities.
onFlush is executed after the change sets are computed.
afterFlush is executed as the last step just before the flush call resolves. it will be executed even if there are no changes to be flushed.

Flush event args will not contain any entity instance, as they are entity agnostic. They do contain additional reference to the UnitOfWork instance.

Following example demonstrates the hidden power of flush events — they allow to hook into the change set tracking, adjusting what will be persisted and how. Here we try to find a CREATE change set for entity FooBar, and if there is any, we automatically create a new FooBaz entity, connecting it to the FooBar one. This kind of operations was previously impossible, as in regular lifecycle hooks we can only adjust the entity that triggers the event.

Joined loading strategy

Loading of complex relations now support so called JOINED strategy. Its name is quite self-explanatory — instead of the default (SELECT_IN) strategy, it uses single SQL query and maps the result to multiple entities.

Single Table Inheritance

STI is an inheritance mapping strategy where all classes of a hierarchy are mapped to a single database table. In order to distinguish which row represents which type in the hierarchy a so-called discriminator column is used.

If no discriminator map is provided, it will be generated automatically.

Following example defines 3 entities — they will all be stored in a single database table called person, with a special column named type, that will be used behind the scenes to know what class should be used to represent given row/entity.

Embeddables

Embeddables are classes which are not entities themselves, but are embedded in entities and can also be queried. You’ll mostly want to use them to reduce duplication or separating concerns. Value objects such as date range or address are the primary use case for this feature.

Embeddables can only contain properties with basic @Property() mapping.

Following example will result in a single database table, where the address fields will be inlined (with prefix) to the user table.

Lazy scalar properties

In MikroORM 4, we can mark any property as lazy: true to omit it from the select clause. This can be handy for properties that are too large and you want to have them available only some times, like a full text of an article.

When we need such value, we can use populate parameter to load it as if it was a reference.

If the entity is already loaded and you need to populate a lazy scalar property, you might need to pass refresh: true in the FindOptions.

Computed Properties

Another small enhancement in entity definition is the @Formula() decorator. It can be used to map some SQL snippet to your entity. The SQL fragment can be as complex as you want and even include subselects.

Formulas will be added to the select clause automatically. In case you are facing problems with NonUniqueFieldNameException, you can define the formula as a callback that will receive the entity alias in the parameter.

Type-safe references

Next feature I would like to mention is rather hidden, and is a bit experimental. In MikroORM 4, all EntityManager and EntityRepository methods for querying entities (e.g. find()) will now return special Loaded type, where we automatically infer what relations are populated. It dynamically adds special get() method to both Reference and Collection instances, that you can use to ensure the relation is loaded on the type level.

QueryBuilder improvements

There have been quite a lot of small adjustments in QueryBuilder, to name a few things:

support for subqueries and qb.ref()
using sql snippets with qb.raw()
pagination support via subselects (QueryFlag.PAGINATE)
update & delete queries with auto-joining

Here are few examples of those features in action:

And many many more…

em.begin/commit/rollback() methods are back
using file globs for discovery (**/*.entity.ts)
custom driver exceptions (UniqueConstraintViolationException, …)
adding items to not-initialized collections
bulk deletes and other performance improvements
inference of custom repository type (EntityRepositoryType)
property serializers

See the changelog for full list of new features and fixes.

More example integrations

Koa: https://github.com/mikro-orm/koa-ts-example-app
GraphQL: https://github.com/driescroons/mikro-orm-graphql-example
Serverless: https://github.com/thomaschaaf/serverless-mikro-orm-example-app

Upgrading

For smooth upgrading, read the full upgrading guide. Here are few notable breaking changes:

Default metadata provider is ReflectMetadataProvider, to use ts-morph, you need to install it from @mikro-orm/reflection and explicitly provide it in the ORM configuration. If you want to use ReflectMetadataProvider, be sure to see the list of its limitations.
TsMorphMetadataProvider now uses *.d.ts files in production mode, so be sure to enable them in your tsconfig.json.
@mikro-orm/core package is not dependent on knex, and therefore cannot provide methods like createQueryBuilder() — instead, those methods exist on SqlEntityManager. You can import it from the driver package, e.g. import { EntityManager } from '@mikro-orm/mysql;.
To use CLI, you need to install @mikro-orm/cli package.
When using folder based discovery, the options entitiesDirs and entitiesDirsTs are now removed in favour of entities and entitiesTs. You can now mix entity references with folders and file globs, negative globs are also supported.
For Nest.js users, there is a new @mikro-orm/nestjs package, which is a fork of the nestjs-mikro-orm module with changes needed for MikroORM 4.

What’s next?

Here are some features I’d like to work on in the near future:

Improved schema diffing
ts-morph reflection via custom TS compiler plugin
Query caching
MS SQL Server support

WDYT?

So thit is MikroORM 4, what do you think about it? What features or changes would you like to see next? Or what part of the documentation should be improved and how?

Like MikroORM? ⭐️ Star it on GitHub and share this article with your friends. If you want to support the project financially, you can do so via GitHub Sponsors.

MikroORM 3: Knex.js, CLI, Schema Updates, Entity Generator and more…

Martin Adámek — Thu, 16 Jan 2020 16:27:43 +0000

New major version of the TypeScript ORM has been released, read about its new features and breaking changes.

In case you don’t know…

You can read the full introductory article here or browse through the docs.

Integrated Knex.js

You probably know Knex.js already, but if you don’t, it is a “batteries included” SQL query builder for Postgres , MSSQL , MySQL , MariaDB , SQLite3 , Oracle , and Amazon Redshift designed to be flexible, portable, and fun to use.

Knex.js is now used as both a query builder and a query runner for all SQL drivers. This allows to simplify SQL driver implementations as well as brings some new possibilities.

Using Knex.js

You can access configured knex instance via qb.getKnexQuery() method. Then you can execute it via the Connection.execute() and map the results via EntityManager.map().

You can also get clear and configured knex instance from the connection via getKnex() method. As this method is not available on the base Connection class, you will need to either manually type cast the connection to AbstractSqlConnection (or the actual implementation you are using, e.g. MySqlConnection), or provide correct driver type hint to your EntityManager instance, which will be then automatically inferred in em.getConnection() method.

Driver and connection implementations are not directly exported from mikro-orm module. You can import them from mikro-orm/dist (e.g. import { PostgreSqlDriver } from 'mikro-orm/dist/drivers/PostgreSqlDriver').

Connection Pooling

With Knex.js used as a query runner, support for connection pooling is finally available. Tarn.js is used for this internally, using connection pool with min: 2, max: 10 for the MySQL and PG libraries, and a single connection for sqlite3 by default. Use pool option to change this when initializing the ORM.

More SQL Drivers?

One of the strongest reasons to integrate Knex.js was that it allows to simplify and unify SQL drivers and opens doors for implementing new SQL drivers. Knex.js currently supports (apart from those currently supported by MikroORM): MSSQL, Oracle and Amazon Redshift.

Thanks to AbstractSqlDriver and AbstractSqlConnection classes it should be fairly simple to implement them. I am open for PRs for those drivers, as I would like to focus on developing new ORM features mainly, instead of learning new SQL dialects I have never used. I will be happy to assist to anybody interested — feel free to reach me out either via Slack, email or GitHub issues.

Simplified Entity Definition

Now it is no longer needed to merge entities with IEntity interface, that was polluting entity's interface with internal methods. New interfaces IdentifiedEntity<T>, UuidEntity<T> and MongoEntity<T> are introduced, that should be implemented by entities. They are not adding any new properties or methods, keeping the entity's interface clean.

IEntity interface has been renamed to AnyEntity<T, PK> and it no longer has public methods like toJSON(), toObject() or init(). One can use wrap() method provided by ORM that will enhance property type when needed with those methods (e.g. await wrap(book.author).init()). To keep all methods available on the entity, you can still use interface merging with WrappedEntity<T, PK> that both extends AnyEntity<T, PK> and defines all those methods.

You will need to mark the entity by implementing one of *Entity interfaces:

IdEntity<T> for numeric/string PK on id property (id: number)
UuidEntity<T> for string PK on uuid property (uuid: string)
MongoEntity<T> for mongo, where id: string and _id: ObjectId are required
AnyEntity<T, PK> for other possible properties (fill the PK property name to PK parameter, e.g.: AnyEntity<Book, 'myPrimaryProperty'>')

To keep all public methods that were part of IEntity interface in v2, you can use WrappedEntity<T, PK> via interface merging.

Nested Queries

SQL driver now support nested where and orderBy conditions. This means that you can query by properties of a relationship and the relation will be automatically joined for you. They are available both in EntityManager and QueryBuilder APIs.

Strict Typing of Queries

Previously the where parameter of EntityManager’s find methods (find(), findOne(), count()) was weakly typed. It allowed users to pass pretty much anything there.

Now the query is strictly typed, only entity properties and operators can be used and the type of property value is also checked.

Improved Schema Generator

SchemaGenerator now supports creating, updating and dropping the schema. You can either get the SQL queries as array of strings or directly run them on the database.

Always check the generated SQL first before running it.

There is also new columnType property attribute you can use to specify the database specific column type explicitly.

Migrations

Better way to handle schema updates than using the SchemaGenerator directly is to use Migrations. MikroORM 3 has integrated support for migrations via umzug. It allows you to generate migrations with current schema differences.

By default, each migration will be all executed inside a transaction, and all of them will be wrapped in one master transaction, so if one of them fails, everything will be rolled back.

Generating Entities from Current Database

As a counterpart to the SchemaGenerator that propagates changes in your entities to the database schema, there is now EntityGenerator to help you with reverse engineering current database schema and creating entities based on it.

It supports basic entity definition including ManyToOne and OneToOne relationships. Currently ManyToMany will be generated as additional entity with two ManyToOne relations and you will need to refactor this yourself.

While it can help a lot, there is quite a lot of room for improvement. In future I would like to implement proper support for ManyToMany relations as well for enums and indexes. Another possible extension would be to allow editing existing entities (syncing them with current schema).

CLI

While you can use SchemaGenerator and EntityGenerator manually, much easier way is to use new CLI tool. Simply create configuration file in root directory or add its path to package.json. TypeScript files are also supported via ts-node:

Now you can use the CLI with help of npx:

To verify your setup, you can use the mikro-orm debug command. Once you have it configured properly, you can also re-use it when initializing the ORM:

// when no options parameter is provided, CLI config will be used
const orm = await MikroORM.init();

Custom Mapping Types

With Custom Types we can now enhance how the database value will be represented in the ORM. You can define custom types by extending Type abstract class, it has 4 optional methods:

convertToDatabaseValue(value: any, platform: Platform): any

Converts a value from its JS representation to its database representation of this type. By default returns unchanged value.

convertToJSValue(value: any, platform: Platform): any

Converts a value from its database representation to its JS representation of this type. By default returns unchanged value.

toJSON(value: any, platform: Platform): any

Converts a value from its JS representation to its serialized JSON form of this type. By default converts to the database value.

getColumnType(prop: EntityProperty, platform: Platform): string

Gets the SQL declaration snippet for a field of this type. By default returns columnType of given property.

Here is a simplified version of DateType that is already present in the ORM:

And Many More…

There are many more new features, see the changelog to read the full list. Here are few of them worth mentioning:

Improved support for References
Navite Enum support
em.findAndCount() and em.findOneOrFail() methods
ReflectMetadataProvider as a fast alternative to ts-morph reflection
Improved logging with query highlighting
Support for bundling via Webpack
Eager loading
Read Connections
More strict entity definition validation

Notable Breaking Changes

Here is a short list of breaking changes. You can see the full list in the docs: https://mikro-orm.io/docs/upgrading-v2-to-v3/.

Auto-flushing Disabled by Default

If you had autoFlush: false in your ORM configuration before, you can now remove this line, no changes are needed in your app.

Default value for autoFlush is now false. That means you need to call em.flush() yourself to persist changes into database. You can still change this via ORM's options to ease the transition but generally it is not recommended as it can cause unwanted small transactions being created around each persist.

Transactions API

Transactions now require using em.transactional() method, previous methods beginTransaction/commit/rollback are now removed.

Making it a bit more Professional…

Not a big deal, but probably worth mentioning — MikroORM’s repository has been transferred to new MikroORM GitHub Organization and the website is now moved to mikro-orm.io. Old links should be properly redirected, if you find some 404, please let me know thru GitHub issues!

Website has also been redesigned — now it is built with Docusaurus (v2) and provides fulltext search by Algolia. The docs are now also versioned.

Check it out!

What’s next?

Here are some features I am planning to work in the near future:

Composite primary keys
Transactions in MongoDB
Complex hydration of joined result sets
Slow query log
M:N support in entity generator

There are also some interesting suggestion in the Github issues, like Dataloader integration.

WDYT?

So that is MikroORM 3, what do you think about it? What features or changes would you like to see next? Or what part of the documentation should be improved and how?

Like MikroORM? ⭐️ Star it on GitHub and share this article with your friends.

Handling transactions and concurrency in MikroORM

Martin Adámek — Tue, 18 Jun 2019 17:44:12 +0000

Wait, what? MikroORM?

If you never heard of MikroORM, it’s a TypeScript data-mapper ORM with Unit of Work and Identity Map. It supports MongoDB, MySQL, PostgreSQL and SQLite drivers currently.

You can read the full introductory article here or browse through the docs. The project is under active development, so be sure to check out the change log as well.

MikroORM is heavily inspired by Doctrine ORM. This article is highly inspired by doctrine documentation as the behaviour in MikroORM is pretty much the same. All credits for the general explanation of this topic goes to them!

Note about persisting

There are 2 methods we should first describe to understand how persisting works in MikroORM: em.persist() and em.flush().

em.persist(entity, flush?: boolean) is used to mark new entities for future persisting. It will make the entity managed by given EntityManager and once flush will be called, it will be written to the database. Second boolean parameter can be used to invoke flush immediately. Its default value is configurable via autoFlush option.

Default value of autoFlush is currently set to true, which will change in upcoming major release. Users are encouraged to either set autoFlush to false or use em.persistLater() (equal to em.persist(entity, false)) and em.persistAndFlush() methods instead. Every time persisting is mentioned in this article, it is with autoFlush set to false in mind.

To understand flush, lets first define what managed entity is: An entity is managed if it’s fetched from the database (via em.find(), em.findOne() or via other managed entity) or registered as new through em.persist().

em.flush() will go through all managed entities, compute appropriate change sets and perform according database queries. As an entity loaded from database becomes managed automatically, you do not have to call persist on those, and flush is enough to update them.

Transaction demarcation

Transaction demarcation is the task of defining your transaction boundaries. For the most part, MikroORM already takes care of proper transaction demarcation for you: All the write operations (INSERT/UPDATE/DELETE) are queued until em.flush() is invoked which wraps all of these changes in a single transaction. However, MikroORM also allows (and encourages) you to take over and control transaction demarcation yourself.

Approach 1: Implicitly

The first approach is to use the implicit transaction handling provided by the MikroORM EntityManager. Given the following code snippet, without any explicit transaction demarcation:

Since we do not do any custom transaction demarcation in the above code, em.flush() will begin and commit/rollback a transaction. This is sufficient if all the data manipulation that is part of a unit of work happens through the domain model and thus the ORM — in other words, unless you run some write queries manually, via QueryBuilder, or use one of em.nativeInsert/Update/Delete helpers.

Here is a bit more complex example where multiple entities are involved:

We load one author by id, all his books and their tags as well as their publisher. For simplicity, let’s assume the author has one book associated, which has one book tag and one publisher.

Then we update multiple things on book of that author, editing name of the tag, adding new one, and changing publisher’s name. As we are working with already managed entities (retrieved from EntityManager), we can simply flush without needing to persist those entities.

The flush call here will compute all differences and run database queries accordingly. They will all be encapsulated in a transaction, as you can see from following list of fired queries:

Approach 2: Explicitly

The explicit alternative is to use the transactions API directly to control the boundaries. The code then looks like this:

Explicit transaction demarcation is required when you want to include custom DBAL operations in a unit of work (e.g. when firing native SQL UPDATE queries) or when you want to make use of some methods of the EntityManager API that require an active transaction (e.g. locking) — such methods will throw a ValidationError to inform you of that requirement.

A more convenient alternative for explicit transaction demarcation is to use em.transactional(cb). It will automatically start the transaction, execute your asynchronous callback and commit it. In case of an exception during those operations, the transaction will be automatically rolled back and the exception will be re-thrown. An example that is functionally equivalent to the previously shown code looks as follows:

In the callback parameter, you will get forked EntityManager that will contain a copy of the current Identity Map. You should use this copy instead of the parent one for all queries inside the transaction. It will be flushed prior to transaction commit.

Exception Handling

When using implicit transaction demarcation and an exception occurs during em.flush(), the transaction is automatically rolled back.

When using explicit transaction demarcation and an exception occurs, the transaction should be rolled back immediately as demonstrated in the example above. Users are encouraged to use em.transactional(cb) which will handle that automatically.

As a result of this procedure, all previously managed or removed instances of the EntityManager become detached. The state of the detached objects will be the state at the point at which the transaction was rolled back. The state of the objects is in no way rolled back and thus the objects are now out of sync with the database. The application can continue to use the detached objects, knowing that their state is potentially no longer accurate.

If you intend to start another unit of work after an exception has occurred you should do that with a new EntityManager. Simply use em.fork() to obtain fresh copy with cleared identity map.

Concurrency and locking

Why we need concurrency control?

If transactions are executed serially (one at a time), no transaction concurrency exists. However, if concurrent transactions with interleaving operations are allowed, you may easily run into one of those problems:

The lost update problem
The dirty read problem
The incorrect summary problem

Take a look at this article for in-depth explanation of those.

To mitigate those problems, MikroORM offers support for Pessimistic and Optimistic locking strategies natively. This allows you to take very fine-grained control over what kind of locking is required for your entities in your application.

Optimistic Locking

Database transactions are fine for concurrency control during a single request. However, a database transaction should not span across requests, the so-called “user think time”. Therefore a long-running “business transaction” that spans multiple requests needs to involve several database transactions. Thus, database transactions alone can no longer control concurrency during such a long-running business transaction. Concurrency control becomes the partial responsibility of the application itself.

MikroORM has integrated support for automatic optimistic locking via a version field. In this approach any entity that should be protected against concurrent modifications during long-running business transactions gets a version field that is either a simple number or a Date (timestamp). When changes to such an entity are persisted at the end of a long-running conversation the version of the entity is compared to the version in the database and if they don’t match, a ValidationError is thrown, indicating that the entity has been modified by someone else already.

To define a version field, simply use @Property decorator with version flag set to true. Only Date and number types are allowed.

Version numbers (not timestamps) should be preferred as they can not potentially conflict in a highly concurrent environment, unlike timestamps where this is a possibility, depending on the resolution of the timestamp on the particular database platform.

When a version conflict is encountered during em.flush(), a ValidationError is thrown and the active transaction rolled back (or marked for rollback). This exception can be caught and handled. Potential responses to a ValidationError are to present the conflict to the user or to refresh or reload objects in a new transaction and then retrying the transaction.

The time between showing an update form and actually modifying the entity can in the worst scenario be as long as your applications session timeout. If changes happen to the entity in that time frame you want to know directly when retrieving the entity that you will hit an optimistic locking exception.

You can always verify the version of an entity during a request either when calling em.findOne():

Or you can use em.lock() to find out:

Using optimistic locking correctly, you have to pass the version as an additional parameter when updating entity. See following example:

Your frontend app loads an entity from API, the response includes the version property. User makes some changes and fires PUT request back to the API, with version field included in the payload. The PUT handler of the API then reads the version and passes it to the em.findOne() call.

Pessimistic Locking

MikroORM supports Pessimistic Locking at the database level. Every Entity can be part of a pessimistic lock, there is no special metadata required to use this feature. Pessimistic Locking requires active transaction, so you will have to use explicit transaction demarcation.

MikroORM currently supports two pessimistic lock modes:

Pessimistic Write (LockMode.PESSIMISTIC_WRITE), locks the underlying database rows for concurrent Read and Write Operations.
Pessimistic Read (LockMode.PESSIMISTIC_READ), locks other concurrent requests that attempt to update or lock rows in write mode.

You can use pessimistic locks in three different scenarios:

Using em.findOne(className, id, { lockMode })
Using em.lock(entity, lockMode)
Using QueryBuilder.setLockMode(lockMode)

This is how it looks like in action:

Like MikroORM? ⭐️ Star it on GitHub and share this article with your friends.

Introducing MikroORM, TypeScript data-mapper ORM with Identity Map

Martin Adámek — Tue, 30 Apr 2019 16:40:42 +0000

Motivation

During my early days at university, I remember how quickly I fell in love with object oriented programming and the concepts of Object-relational mapping and Domain Driven Design. Back then, I was mainly a PHP programmer (while we did a lot of Java/Hibernate at school), so a natural choice for me was to start using Doctrine.

A few years ago, when I switched from PHP to Node.js (and later to TypeScript), I was really confused. How come there is nothing similar to Hibernate or Doctrine in the JavaScript world? About a year ago, I finally came across TypeORM, and when I read this line in the readme I thought I found what I was looking for:

TypeORM is highly influenced by other ORMs, such as Hibernate, Doctrine and Entity Framework.

I started playing with it immediately, but I got disappointed very quickly. No Identity Map that would keep track of all loaded entities. No Unit of Work that would handle transaction isolation. No unified API for references with very strange support for accessing just the identifier without populating the entity, MongoDB driver (which I was aiming to use) was experimental and I had a lot problems setting it up. After a few days of struggle, I went away from it.

By that time, I started to think about writing something myself. And that is how MikroORM started!

MikroORM is TypeScript ORM for Node.js based on Data Mapper, Unit of Work and Identity Map patterns.

Currently it supports MongoDB, MySQL, PostgreSQL and SQLite databases, but more can be supported via custom drivers right now. It has first class TypeScript support, while staying back compatible with Vanilla JavaScript.

Installation

First install the module via yarn or npm and do not forget to install the database driver as well. Next you will need to enable support for decorators

in tsconfig.json via experimentalDecorators flag. Then call MikroORM.init as part of bootstrapping your application.

Last step is to provide forked EntityManager for each request, so it will have its own unique Identity Map. To do so, you can use EntityManager.fork() method. Another way, that is more DI friendly, is to create new request context for each request, which will use some dark magic in the background to always pick the right EntityManager for you.

Defining entities

To define an entity, simply create a class and decorate it. Here is an example of Book entity defined for MongoDB driver:

As you can see, it’s pretty simple and straightforward. Entities are simple JavaScript objects (so called POJO), decorated with @Entity decorator (for TypeScript), or accompanied with schema definition object (for vanilla JavaScript). No real restrictions are made, you do not have to extend any base class, you are more than welcome to use entity constructors for specifying required parameters to always keep the entity in valid state. The only requirement is to define the primary key property.

You might be curious about the last line with Book as an interface. This is called interface merging and it is there to let TypeScript know the entity will have some extra API methods (like init() or isInitialized()) available as it will be monkey-patched during discovery process. More about this can be found in the docs.

Persisting entities with EntityManager

To save entity state to database, you need to persist it. Persist takes care or deciding whether to use insert or update and computes appropriate change-set. As a result, only changed fields will be updated in database.

MikroORM comes with support for cascading persist and remove operations. Cascade persist is enabled by default, which means that by persisting an entity, all referenced entities will be automatically persisted too.

Fetching entities

To fetch entities from database you can use find() and findOne() methods of EntityManager:

More convenient way of fetching entities from database is by using EntityRepository, that carries the entity name so you do not have to pass it to every find and findOne calls:

Working with references

Entity associations are mapped to entity references. Reference is an entity that has at least the identifier (primary key). This reference is stored in the Identity Map so you will get the same object reference when fetching the same document from database.

Thanks to this concept, MikroORM offers unified API for accessing entity references, regardless of whether the entity is initialized or not. Even if you do not populate an association, there will be its reference with primary key set. You can call await entity.init() to initialize the entity. This will trigger database call and populate itself, keeping the same reference to entity object in identity map.

Identity Map and Unit of Work

MikroORM uses the Identity Map in background to track objects. This means that whenever you fetch entity via EntityManager, MikroORM will keep a reference to it inside its UnitOfWork, and will always return the same instance of it, even if you query one entity via different properties. This also means you can compare entities via strict equality operators (=== and !==):

Another benefit of Identity Map is that this allows us to skip some database calls. When you try to load an already managed entity by its identifier, the one from Identity Map will be returned, without querying the database.

The power of Unit of Work is in running all queries inside a batch and wrapped inside a transaction (if supported by given driver). This approach is usually more performant as opposed to firing queries from various places.

Collections

OneToMany and ManyToMany collections are stored in a Collection wrapper. It implements iterator so you can use for of loop to iterate through it.

Another way to access collection items is to use bracket syntax like when you access array items. Keep in mind that this approach will not check if the collection is initialized, while using get method will throw error in this case.

More informations about collections can be found in the docs.

What’s next?

So you read through the whole article, got here and still not satisfied? There are more articles to come (beginning with integration manual for popular frameworks like Express or NestJS), but you can take a look at some advanced features covered in docs right now:

To start playing with MikroORM, go through quick start and read the docs. You can also take a look at example integrations with some popular frameworks.

Like MikroORM? ⭐️ Star it on GitHub and share this article with your friends.

This article was originally published on Medium: https://medium.com/dailyjs/introducing-mikro-orm-typescript-data-mapper-orm-with-identity-map-9ba58d049e02

DEV Community: Martin Adámek

MikroORM 6: Polished

In case you don’t know…

Quick summary of 5.x releases

Type safety

Strict partial loading

Opt type

Hidden type

Populating all relations

Populate based on filter

Primary key type inference

Simplified BaseEntity

Implicit serialization

forceObject

Joined strategy

Filters on relations

Cursor-based pagination

Raw SQL fragments

Subquery operators $some, $none and $every

Subquery joining

Dataloader support for references and collections

Logging improvements

Improved change-tracking of M:N relations

Extending EntityManager

GeneratedCacheAdapter for production usage

Entity Generator improvements

Inference of default values

Other notable changes

One more thing…

MikroORM 5: Stricter, Safer, Smarter

In case you don’t know…

Quick summary of 4.x releases

Improved type safety

Validation improvements

Reworked schema diffing

Smarter migrations

Auto-flush mode

Automatic refreshing of loaded entities

Seeder package

Polymorphic embeddables

Populating lazy scalar properties

Creating references without EntityManager

Smarter expr helper

Awaitable QueryBuilder

Wildcard schema entities

Deep assigning of entities

Experimental support for ES modules

Other notable changes

What’s next?

MikroORM 4.1: Let’s talk about performance

In case you don’t know…

So what changed?

Batch inserts, updates and deletes

JIT compilation

Results

Acknowledgement

MikroORM 4: Filling the Gaps

In case you don’t know…

Quick summary of 3.x releases

Monorepo

Filters

Global Filters

EventSubscribers and flush events

Flush events

Joined loading strategy

Single Table Inheritance

Embeddables

Lazy scalar properties

Computed Properties

Type-safe references

QueryBuilder improvements

And many many more…

More example integrations

Upgrading

What’s next?

WDYT?

MikroORM 3: Knex.js, CLI, Schema Updates, Entity Generator and more…

In case you don’t know…

Integrated Knex.js

Using Knex.js

`Opt` type

`Hidden` type

Simplified `BaseEntity`

`forceObject`

Subquery operators `$some`, `$none` and `$every`

Extending `EntityManager`

`GeneratedCacheAdapter` for production usage