Changes to STM in Effect 4

STM (Software Transactional Memory) is a way to manage concurrent modifications of shared state through transactions, roughly similar to database transactions.

Let's say we need to transfer money from one account to another: we need to deduct the amount from one account and add it to the other.

Playground — Effect v3

// Effect v3
const program = Effect.gen(function* () {
  const alice = yield* TRef.make(500)
  const bob = yield* TRef.make(500)

  const transfer = STM.gen(function* () {
    const aliceBalance = yield* TRef.get(alice)

    if (aliceBalance < 100) {
      yield* STM.retry
    }

    const bobBalance = yield* TRef.get(bob)

    yield* TRef.set(alice, aliceBalance - 100)
    yield* TRef.set(bob, bobBalance + 100)
  }) 

  yield* STM.commit(transfer)

  console.log(yield* TRef.get(alice)) // 400
  console.log(yield* TRef.get(bob))   // 600
})

Here we can see that STM<> is a separate computation type.

It is synchronous and pure — meaning it cannot be interrupted somewhere in the middle, unlike Effect<>.

Whenever we interact with a TRef, STM keeps a journal of those interactions. The journal stores a reference to the ref, its value at the beginning of the interaction, and its current value within the transaction.

Then, when we call STM.commit and commit the changes to the TRefs, STM performs a check under the hood:

Are the original values of the TRefs still the same as they were when the transaction started?

If yes, the changes are applied.

If not, the transaction is no longer valid and is restarted. This is possible precisely because STM is a pure computation.

And this continues until the transaction successfully commits.

Probably the biggest limitation of this model is that you cannot use Effect<> computations inside STM<>.

For example, this makes things like logging and debugging more difficult.

---

In v4, the model has changed. The Tx* modules are essentially STM built directly into Effect.

Playground — Effect v4

const program = Effect.gen(function* () {
  const alice = yield* TxRef.make(500)
  const bob = yield* TxRef.make(500)

  const transfer = Effect.tx(Effect.gen(function* () {
    const aliceBalance = yield* TxRef.get(alice)

    if (aliceBalance < 100) {
      yield* Effect.txRetry
    }

    const bobBalance = yield* TxRef.get(bob)

    yield* TxRef.set(alice, aliceBalance - 100)
    yield* TxRef.set(bob, bobBalance + 100)
  }))

  yield* transfer

  console.log(yield* TxRef.get(alice)) // 400
  console.log(yield* TxRef.get(bob))   // 600
})

The basic idea is the same, but under the hood v4 uses Effect.Transaction — a service that contains the STM transaction journal.

So if a computation uses Tx* primitives (TxRef, TxQueue, etc.), it makes sense to represent this at the type level by adding Transaction to the R channel.

It's similar to Scope for resource management.

And the equivalent of Effect.scoped, which provides a Scope through R, is Effect.tx, which defines the transaction boundary.

However, Effect.Transaction is a bit more subtle because it is implicit (Github).

The reason is that many operations on Tx* don't explicitly show their relationship to STM in the R channel. Under the hood, they are literally wrapped in Effect.tx.

For example, TxRef.set is essentially a tiny transaction with its own journal.

But we want the transaction boundary to encompass multiple Tx* operations — that's the whole point.

So Effect.tx is designed in such a way that if it detects an existing Effect.Transaction in the surrounding Context, it attaches to and reuses that existing transaction.

Consider this code:

const transfer = Effect.gen(function* () {
  const aliceBalance = yield* TxRef.get(alice)
  const bobBalance = yield* TxRef.get(bob)

  yield* TxRef.set(alice, aliceBalance - 100)
  yield* TxRef.set(bob, bobBalance + 100)
})

Without an explicit Effect.tx, this results in four isolated micro-transactions — two reads and two writes.

But if we wrap it in Effect.tx(...), instead of creating a new micro-journal for every operation and immediately committing it, each operation finds the current transaction provided by Effect.tx in the Context and writes to its journal instead of creating its own.

---

What are the downsides of this approach?

First of all, explicit is better than implicit.

Transaction barely appears in the R channel, which makes tracking transaction boundaries more complicated. I think it could have been made explicit, similar to how Scope works.

Second, you can now use asynchronous Effect computations inside a transaction.

But this means that the transaction can be stretched over time, increasing the probability that the variables involved in the transaction will change and cause conflicts.

Third, I'm not sure that Transaction is the best name. It would be nice to somehow make its relationship to STM more explicit.

On the other hand, this approach requires much less ceremony to use STM.

And generally speaking, this seems to be the direction Effect's DX is moving toward.

P.S. It's also worth mentioning that although the transaction is wrapped in an uninterruptible region, its execution can still be deferred at some nondeterministic point. This can also increase the probability of conflicts.

Motivation for the changes from Michael — Discord

discord.com