Building Robust Typescript APIs with the Effect Ecosystem

Introduction

In this post, we will explore how to create a complete API using the Effect ecosystem. Although the Effect documentation is excellent, there are not many examples of full applications.

If you have never heard of Effect, I suggest you take a look at the website: https://effect.website/

Effect is not just another framework but instead acts as the missing standard library for TypeScript. Because of this, I believe Effect is truly the future for TypeScript, especially when considering the challenges associated with writing production-ready applications in "plain" TypeScript. To understand more about these challenges and how Effect addresses them, watch this insightful video by Johannes Schickling, the founder of Prisma: https://www.youtube.com/watch?v=PxIBWjiv3og&ab_channel=Effect%7CTypeScriptatScale.

The purpose of this post is to demonstrate how to use the different components of the Effect ecosystem to build a modern API:

• Effect: The core module for handling and creating effects.
• Effect Schema: Use schemas to validate and transform data.
• Effect SQL: A toolkit for integration with SQL databases. We will use PostgreSQL in our example.
• Effect Platform/Node: Allows us to use the Node.js platform. We will use this to create our server, router, and handle external API calls.
• Effect opentelemetry: Lets us get tracing and metrics for our app. We integrate it to our console but its really easy to setup in honeycomb for example

By utilizing these modules from Effect, we gain massive benefits. Everything is type-safe, we have robust error handling, and there's no need to reach out to NPM for specific packages. Everything is included in Effect.

Make sure you clone the GitHub repo to see the complete code.
Link to repo: https://github.com/Mumma6/effect-node-server

Overview of the Project

This project involves building a straightforward application that allows you to create user profiles and add movies to these profiles. It incorporates basic CRUD (Create, Read, Update, Delete) operations, external API calls with movie details fetched from the OMDb API. All data, including user profiles and associated movie information, is stored in a PostgreSQL database.

The system's design is inspired by Domain-Driven Design (DDD), where "User" and "Movie" are treated as distinct domains. This approach facilitates managing complex functionalities and maintaining clear boundaries between different areas of concern within the application making it an excellent starting point and foundation to build upon.

Design

Domain Layer

The domain layer is divided into user and movie domains, each containing the following sub-layers:

Models: Define the data structures and schemas.
Repositories: Handle data access and database operations.
Infrastructure: Handle external API calls (movie only).
Services: Implement business logic and orchestrate calls to repositories and external APIs (for the movie domain).

Routes Layer

The routes layer defines the HTTP routes for the API. Each domain has its own set of routes:

User Routes: Handles user-related operations (create, read, update, delete).
Movie Routes: Handles movie-related operations (create, read, update, delete) and integrates with the OMDb API.

Lib Layer

The lib layer contains configuration and setup for the database:

Database: Contains configuration and initialization logic for connecting to PostgreSQL using the Effect library.

Entry Point

index.ts: The main entry point of the application. It initializes the application by setting up routes and starting the server.

Code

Rather than guiding you through a step-by-step coding tutorial, I'll highlight crucial code snippets from our project. This approach will focus on explaining the concepts and the architecture behind the implementation, offering insights into how the Effect ecosystem seamlessly integrates into our application.

Dependencies

The project uses these dependencies

"dependencies": {
    "@effect/opentelemetry": "^0.34.29",
    "@effect/platform": "^0.58.16",
    "@effect/platform-node": "^0.53.15",
    "@effect/schema": "^0.68.14",
    "@effect/sql": "^0.4.16",
    "@effect/sql-pg": "^0.4.16",
    "@opentelemetry/sdk-metrics": "^1.25.1",
    "@opentelemetry/sdk-trace-base": "^1.25.1",
    "@opentelemetry/sdk-trace-node": "^1.25.1",
    "dotenv": "^16.4.5",
    "effect": "^3.4.5",
    "ts-node-dev": "^2.0.0",
    "typescript": "^5.5.2"
  }

Entry point

In our index.ts, we initialize the server and the router. We provide all the necessary Layers for the project, similar to "dependency injection." This setup ensures that each component has access to the resources it needs to function correctly.

We also have the ability to add our custom middleware. Notably, the NodeRuntime.runMain function acts as the starting point of our entire application. This function boots up the server, ensuring that all configurations and services are loaded and ready to handle requests.

import { HttpMiddleware, HttpRouter, HttpServer, HttpServerResponse } from "@effect/platform"
import { NodeHttpServer, NodeRuntime } from "@effect/platform-node"
import { Effect, Layer } from "effect"
import { createServer } from "http"
import { DatabaseInitialisation, SqlLive } from "./lib/database"
import { AppRouter } from "./routes/routes"

import { UserService } from "./domain/user/service/user.service"
import { MovieService } from "./domain/movies/service/movie.service"
import { TracingConsole } from "./lib/tracing"

const myLogger = HttpMiddleware.make((app) =>
  Effect.gen(function* () {
    console.log("LOGGED")
    return yield* app
  })
)
const ServerLive = NodeHttpServer.layer(createServer, { port: 5000, host: "localhost" })

const HttpLive = HttpRouter.Default.unwrap(HttpServer.serve(myLogger)).pipe(
  Layer.provide(AppRouter),
  Layer.provide(ServerLive),
  Layer.provide(UserService.Live),
  Layer.provide(MovieService.Live),
  Layer.provide(DatabaseInitialisation.Live),
  Layer.provide(SqlLive),
  Layer.provide(TracingConsole)
)

NodeRuntime.runMain(Layer.launch(HttpLive))

Lib folder

In the lib directory, we define the structure of our database by creating tables and setting up the connection details for our PostgreSQL client. Typically, you would retrieve these connection details from environment variables specified in your .env file to enhance security and flexibility. However, for simplicity in this demonstration, I have hardcoded them directly into the configuration files.

import { PgClient } from "@effect/sql-pg"
import { Config, Context, Effect, Layer, Redacted } from "effect"

const make = Effect.gen(function* () {
  const sql = yield* PgClient.PgClient

  // Suppress NOTICE messages
  yield* sql`SET client_min_messages TO WARNING;`

  // Create users table if it does not exist
  yield* sql`CREATE TABLE IF NOT EXISTS users (
    id SERIAL PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    email VARCHAR(255) NOT NULL,
    created_at TIMESTAMP NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP NOT NULL DEFAULT NOW()
  );`

  // Create movies table if it does not exist
  yield* sql`CREATE TABLE IF NOT EXISTS movies (
    id SERIAL PRIMARY KEY,
    user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
    title VARCHAR(255) NOT NULL,
    year VARCHAR(4),
    genre VARCHAR(255),
    plot TEXT,
    imdb_id VARCHAR(255),
    created_at TIMESTAMP NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP NOT NULL DEFAULT NOW()
  );`
})

export class DatabaseInitialisation extends Context.Tag("DatabaseInitialisation")<DatabaseInitialisation, Effect.Effect.Success<typeof make>>() {
  static readonly Live = Layer.effect(this, make)
}

export const SqlLive = PgClient.layer({
  database: Config.succeed("effect"),
  host: Config.succeed("localhost"),
  username: Config.succeed("postgres"),
  port: Config.succeed(5432),
  password: Config.succeed(Redacted.make("123")),
})

In addition to database configurations, our lib directory also sets up tracing to monitor and diagnose application performance. For details on integrating with alternative tracing services like Honeycomb, refer to the GitHub repository.

import * as NodeSdk from "@effect/opentelemetry/NodeSdk"
import { BatchSpanProcessor, ConsoleSpanExporter } from "@opentelemetry/sdk-trace-base"

export const TracingConsole = NodeSdk.layer(() => ({
  resource: { serviceName: "example" },
  spanProcessor: new BatchSpanProcessor(new ConsoleSpanExporter()),
}))

Routing

The routes.ts file plays a crucial role in our application by organizing and managing all the routing configurations. Currently, it handles two primary routes: one for movies and another for users. However, the structure is designed to be extensible, allowing for the easy addition of more routes as the application grows.

Each route is grouped within its respective module, and specific route implementations are equipped with necessary dependencies using the Layer.provide method. This setup ensures that each route has access to the services and data layers it requires to function correctly.

import { HttpRouter } from "@effect/platform"
import { Effect, Layer } from "effect"
import { UserRoutes, UsersRouter } from "./users/users.routes"
import { MoviesRouter, MovieRoutes } from "./movies/movies.routes"

export const AppRouter = HttpRouter.Default.use((router) =>
  Effect.gen(function* () {
    yield* router.mount("/users", yield* UsersRouter.router)
    yield* router.mount("/movies", yield* MoviesRouter.router)
  })
).pipe(Layer.provide(UserRoutes), Layer.provide(MovieRoutes))

The MoviesRouter handles routing for movie-related requests. It's defined as a specialized router with a unique tag, which helps in organizing and isolating movie-related routes within the larger application. Here’s how we define and organize these routes:

export class MoviesRouter extends HttpRouter.Tag("MoviesRouter")<MoviesRouter>() {}

export const MovieRoutes = Layer.mergeAll(GetMovies, CreateMovie, UpdateMovie, GetMovieById, GetMoviesForUserId, DeleteMovieById).pipe(
  Layer.provideMerge(MoviesRouter.Live)
)

In this setup, we use Layer.mergeAll to combine various movie-related route handlers like GetMovies, CreateMovie, and others into a single layer. This layer is then provided to the MoviesRouter using Layer.provideMerge, ensuring that all route handlers have access to the necessary dependencies and configurations.

The GetMovies route demonstrates how we handle a typical GET request to fetch all movies. This route uses services from the domain layer to perform its operations and includes robust error handling to manage potential issues during execution:

import { HttpRouter, HttpServerRequest, HttpServerResponse } from "@effect/platform"
import { Effect, Layer, pipe } from "effect"
import { Schema } from "@effect/schema"
import { MovieService } from "../../domain/movies/service/movie.service"

const GetMovies = MoviesRouter.use((router) =>
  pipe(
    MovieService,
    Effect.flatMap((service) =>
      router.get(
        Routes.All,
        Effect.gen(function* () {
          const movies = yield* service.getAllMovies()
          return yield* HttpServerResponse.json(movies)
        }).pipe(
          Effect.catchTags({
            ParseError: (error) =>
              HttpServerResponse.json({ message: "Invalid request data for fetching all movies", details: error.message }, { status: 400 }),
            SqlError: (error) =>
              HttpServerResponse.json({ message: "Database error while fetching all movies", details: error.message }, { status: 500 }),

            HttpBodyError: (error) =>
              HttpServerResponse.json(
                { message: "Error processing request body while fetching all movies", details: error.reason },
                { status: 400 }
              ),
          }),
          Effect.withSpan("GetAllMoviesRoute")
        )
      )
    )
  )
)

Key Features:

• Service Integration: The route integrates with MovieService to fetch all movies, encapsulating business logic and data fetching in a service layer.
• Error Handling: Robust error handling is implemented using Effect.catchTags. This method catches and handles different types of errors, allowing the application to respond with appropriate HTTP status codes and error messages:

ParseError: Catches errors related to request data parsing.
SqlError: Handles errors that occur during database operations.
HttpBodyError: Manages errors related to processing the HTTP request body.

• Distributed Tracing: Effect.withSpan("GetAllMoviesRoute") is used for adding tracing to monitor the performance and troubleshoot issues in the route execution.

By structuring routes in this manner, we ensure that our application not only efficiently handles operations but also gracefully manages errors, providing clear feedback to the client and maintaining the integrity of the application.

Service

The MovieService is responsible for movie-related functionalities in our application. It acts as an intermediary between the MovieRepository, which handles database operations, and MovieInfrastructure, which deals with external API calls to fetch additional movie details.

Here’s how the service is structured:

const make = Effect.gen(function* () {
  const repository = yield* MovieRepository
  const infrastructure = yield* MovieInfrastructure

  const getAllMovies = () => repository.getAllMovies()

  const getMovieById = (id: string) =>
    pipe(
      repository.GetMovieById(id),
      Effect.map(
        Option.match({
          onNone: () => "Movie not found",
          onSome: (movie) => movie,
        })
      )
    )

  // rest of the service implementations

  return {
    getAllMovies,
    getMovieById,
    createMovie,
    updateMovie,
    deleteMovie,
    getMoviesByUserId,
  } as const
})
export class MovieService extends Context.Tag("MovieService")<MovieService, Effect.Effect.Success<typeof make>>() {
  static readonly Live = Layer.effect(this, make)
}

Infrastructure

The MovieInfrastructure layer in our application is specifically designed to handle all external API interactions. In this example, we focus on fetching movie information from the OMDb API, but the structure is scalable to accommodate multiple external API sources as needed. This modularity ensures that extending our application to integrate with additional services is straightforward.

import { Config, Context, Effect, Layer, pipe } from "effect"
import { HttpClientRequest, HttpClient } from "@effect/platform"
import { ApiMovieSchema } from "../models/movie.model"

const basePath = "http://www.omdbapi.com/"

const make = Effect.gen(function* () {
  const key = yield* Config.string("OMDB_KEY")
  const getMovieInformation = (name: string) =>
    pipe(
      HttpClientRequest.get(`${basePath}?t=${name}&apikey=${key}`),
      HttpClient.fetch,
      ApiMovieSchema.decodeResponse,
      Effect.retry({ times: 3 }),
      Effect.withSpan("GetMovieInformation")
    )

  return {
    getMovieInformation,
  } as const
})

export class MovieInfrastructure extends Context.Tag("MovieInfrastructure")<MovieInfrastructure, Effect.Effect.Success<typeof make>>() {
  static readonly Live = Layer.effect(this, make)
}

In the "models" directory we define the expected structure of the JSON response from the OMDb API. Each field like Title is mapped directly to the corresponding field in the API response.

Decoding Method: decodeResponse is a static method that wraps around the schema to provide a convenient way to parse and validate the JSON body of the HTTP response. It ensures that the incoming data matches our specified schema, which is critical for maintaining the integrity and consistency of the data used within our application.

export class ApiMovieSchema extends Schema.Class<ApiMovieSchema>("ApiMovieSchema")({
  Title: Schema.String,
  // Many more fields here...
}) {
  static decodeResponse = HttpClientResponse.schemaBodyJsonScoped(ApiMovieSchema)
}

Repository

The repository layer in our application is crucial for handling all database interactions. It uses SQL providers to communicate with the database, leveraging schemas to define the data structures expected from the database responses and to format the data sent in queries.

Here’s how we define and structure the MovieRepository:

import { Schema } from "@effect/schema"
import { PgClient } from "@effect/sql-pg"
import { Context, Effect, Layer } from "effect"
import { SqlResolver, SqlSchema } from "@effect/sql"
import { InsertMovieSchema, Movie, UpdateMovieSchema } from "../models/movie.model"

const make = Effect.gen(function* () {
  const sql = yield* PgClient.PgClient

  const AddMovie = yield* SqlResolver.ordered("InsertMovie", {
    Request: InsertMovieSchema,
    Result: Movie,
    execute: (requests) => sql`INSERT INTO movies ${sql.insert(requests)} RETURNING *`,
  }).pipe(Effect.withSpan("AddMovieResolver"))

  // rest of the implementations

  return {
    addMovie: AddMovie.execute,
    GetMovieById: GetMovieById.execute,
    getAllMovies: GetAllMovies,
    updateMovie: UpdateMovie.execute,
    deleteMovie: DeleteMovie.execute,
    getMovieByUserId: GetMovieByUserId,
  } as const
})
export class MovieRepository extends Context.Tag("MovieRepository")<MovieRepository, Effect.Effect.Success<typeof make>>() {
  static readonly Live = Layer.effect(this, make)
}

Conclusion

Throughout this post, we've explored how to build a complete and modern API using the Effect ecosystem. From setting up an organized and efficient routing system to integrating external API services and managing database interactions seamlessly, the Effect library has shown its strength in creating scalable and maintainable backend applications.