DEV Community: Andreas Jim

Scala: A Love Story

Andreas Jim — Wed, 21 Apr 2021 13:57:49 +0000

I wrote my first programs when I was 14, in Basic on a Sinclair ZX81. Now, looking back at almost 30 years of writing software from 3-line scripts to enterprise content management systems and microservices in the cloud, maybe a handful experiences stand out. Starting to program in Scala is one of them.

Until 2010, I had programmed mostly back-end code, and almost exclusively in Java. I had used quite a few other languages, it was just that none of them seemed to have any clear advantages. Java came with a huge ecosystem of frameworks and libraries, and I felt like the verbosity was an adequate price to pay for type safety.

At one point I had the opportunity to be the technical lead for a large in-browser application. We started out using plain JavaScript while trying to adhere to the functional paradigm: Avoid mutability, favour evaluation over execution. In the beginning this worked reasonably well. The codebase was small, the code was concise and easy to read, we were able to deliver features at a remarkable pace. But when the codebase grew and more developers joined the team, our productivity started to decline. We had to add (and maintain) more code documentation. It was nearly impossible to change something in a core module, there were just too many potential usages, particularly in a dynamically typed language that supports implicit type conversion and allows things like something[methodName]();. In the end we ended up migrating the critical modules to TypeScript, which improved the situation considerably. Lesson learned.

But the experience from the early days of our JavaScript project stuck with me. It felt like with Java, and with the frameworks we used, there was no linear relation between the complexity of the models or logic you wanted to implement, and the amount of code you had to write. Even the simplest tasks required a lot of boilerplate. The nagging question kept popping up in the back of my mind: Why can't we everything at once: Concise code, type safety, record types, closures? There had to be something better.

I purchased the very entertaining book Seven Languages in Seven Weeks. Although I found Haskell fascinating and tempting, I knew it was unrealistic to introduce it in our company. Scala on the other hand looked like it could be the holy grail: All the characteristics I was looking for, no need to abandon the JVM and its cornucopia of tools and libraries, and the possibility for coexistence with Java and therefore incremental adoption. After implementing some simple programs to identify any immediate risks of committing to the language and its ecosystem, I started to introduce Scala in customer projects. Luckily, I was fortunate enough to work with open-minded, curious, and ambitious team members who were also experienced enough to appreciate the benefits of the language. We immediately applied our experience with functional programming, and embraced immutability. Libraries like Slick and Akka HTTP (we actually started out with its predecessor, Spray) made building database-backed REST services a breeze. And the resulting code was robust and highly maintainable. Scala's expressive type system and type inference made it easy to build a restrictive, consistent domain model without bloating the code. There was virtually no overhead. Any boilerplate could be easily abstracted out. In the end, the application code felt natural, concise and elegant. Programming was fun again.

The name Scala is portmanteau of scalable and language. This is far from a marketing ploy. The language is designed from the ground up to excel in the full spectrum from 3-line scripts to vast, high-performance distributed systems. From my experience, I would attribute this to the following characteristics:

Scala's syntax is particularly suited for building APIs and DSLs.
Its expressive type system (higher-kinded types etc.) encourages abstraction and generic programming.
Scala supports functional programming, and functional programs are inherently composable.
There is are many excellent libraries that leverage these features.

Scala has sparked a huge ecosystem of very high quality libraries (Cats, Scalaz, shapeless, to name but a few). I think a major reason for this is that Scala attracts developers who value the advantages of the JVM, but are fed up with the limitations of the Java programming language and understand the benefits of an expressive type system and functional programming.

Admittedly, it is not trivial to transition from imperative to functional programming. At first glance, some of the functional concepts seem alien and overly complicated, and many beginner tutorials already confront the reader with terms from category theory. If somebody with an imperative background commits themselves to the functional paradigm, they usually have a thorough understanding of its benefits, and why (and when) the transition is worthwhile.

After having programmed almost exclusively in Java for eight months, I miss the convenience of the Scala language and its standard library. In Scala, everything fits together seamlessly, types can be composed and decomposed, there is no need to write boilerplate-ridden classes for simple data types or shoehorn data into unsafe, cumbersome collection types. Scala, I miss you, and I hope we will meet again one day.

Quiz: Is Scala for me?

Andreas Jim — Fri, 19 Mar 2021 13:42:25 +0000

You're currently programming in Java (or any other language for that matter), and wonder whether learning Scala might be worth the effort? Here's a little quiz to help you decide.

Disclaimer: This is just a rough guide based on my experience, and should be taken with two grains of salt 😉

You have a function that calculates the sum of a list of numbers. You have to provide the same functionality for lists of strings (concatenate) and lists of lists (append).

I duplicate the function for each data type. (0 points)
I think about how to make the function generic so that it operates on these three data types, then I refactor it accordingly. (1 point)
I implement a monoid type (or include a library which provides one) and change the function to operate on lists of monoids. (2 point)

When writing software, do you think in short or long terms?

I want to get things done now. Maybe I will never touch this code again, and my program will be irrelevant next year. (0 points)
My code will outlast me. New requirements after five years? I'm not worried, I deliberately design my code to be maintainable. (2 point)

What do you do when you something bothers you in a programming language?

I live with it, it's just one of those things. Nothing is ever perfect. (0 points)
I implement a workaround and publish it as an open-source library. (2 points)

How would you describe yourself?

I do what I'm good at, and I do it the way I'm used to. (0 points)
I'm curious, ambitious and always up for a challenge. (2 point)

What type of technology do you prefer?

I want to be productive immediately, without learning new concepts. I don't know when I have to switch to something else again. (0 points)
I'm willing to invest some time to learn unfamiliar concepts, if I see the long-term benefits. (2 points)

What is your opinion on `null` references?

They are useful and convenient. (0 points)
They are sometimes bothersome, but I didn't really think about it yet. NullPointerExceptions are just part of life. (0 points)
null is terrible. I'm aware of java.util.Optional, but this is such a controversial subject that I haven't yet committed to it. (1 point)
null belongs in hell, and option types are a useful replacement. I just wish java.util.Optional wouldn't be so limited, and that it would be used everywhere. (2 points)

How do you feel about static vs dynamic typing?

The type system is always in the way. I find it too restrictive. (0 points)
I like static typing, it helps with autocomplete and catches mistakes during compile time. (1 point)
An expressive static type system allows me to build safe and consistent domain models, and to write better code in general. (2 points)

What is your opinion on runtime errors?

When they occur, I fix them. Nobody can anticipate everything. (0 points)
That's what unit tests are for. (0 points)
I design my code to be robust, and handle any potential exceptions. (1 point)
Code safety is one of my major design concerns. By accurately modelling my data and treating errors as first-class citizens I can avoid runtime errors almost entirely. (2 points)

How do you prevent regressions?

The Q/A department is responsible for regression testing. (0 points)
I have unit tests with 120% code coverage. (0 points)
I have carefully designed the domain model using advanced type system features based on category theory, and I implemented a guarded finite state machine DSL that allows me to generate the workflow code directly from the specification. (2 points)

Results

0 points: Don't touch Scala with a long stick.

18 points: Head straight to the download page!

Anything in between: Maybe it's worth taking a closer look?

Safe Domain Objects with MapStruct and Immutables

Andreas Jim — Wed, 16 Dec 2020 09:45:24 +0000

Even in Java you are not doomed to deal with mutable objects and nullable fields. With the help of annotation processor libraries like MapStruct and Immutables you can add some safety and convenience to your codebase.

During the last couple of months I have been working with Java, using mutable POJOs as domain objects and MyBatis for database access. Queries are declared in XML files, with only some basic and rather cumbersome support for modularisation and code reuse. Why would I want to deal with another XML-based expression language, when I have a perfectly capable programming language with complete IDE support at my disposal?

Without the compiler protecting me from my own mistakes and the dreaded NullPointerException lurking behind every corner, I dearly missed the comfort and safety of the Scala ecosystem. So I started wondering how to get something akin to case classes and Quill in the cold and harsh winter of the Java world. Let's start with a brief wishlist.

All I want for Christmas is

A substitute for case classes — immutable objects, preferrably with support for cloning (with modified fields) and optics.
Support for mapping primitive database fields to enums, value classes and complex types.
Dedicated domain objects for creating and reading (more on this below).
Type-safe, modular queries.
Minimal boilerplate would be the icing on the gingerbread biscuit.

Libraries

After a little bit of research, I identified the following libraries as possible candidates for the persistence and domain layers:

Immutables — Annotation processor to generate simple, safe and consistent value objects
MapStruct — Another annotation processor; we will use it to map between domain objects and data transfer objects (DTOs). I picked this library out of several similar ones since it seems to be mature and performant and supports Immutables out of the box.
Jinq — Type-safe queries
Functional Java — Because we can't have Cats.

The example code for this article can be found on GitHub.

The Domain Model

Let's start with our domain model. We define the interfaces of our domain objects and let Immutables generate the implementations.

@Value.Immutable @Tuple
public interface Book {

    String getTitle();

}

@Value.Immutable @Tuple
public interface Character {

    String getName();

    Option<Species> getSpecies();

}

A few things worth mentioning:

The @Tuple annotation denotes that we would like to use a single static method to initialize our objects, as opposed to a builder. This way we can ensure at compile-time that all fields are initialized.
The type Species is an enum. JPA can map this to an ordinal value.
We are using the Option class from the Functional Java library, a safer and more convenient alternative to Java's Optional. We use a custom JPA attribute converter to map this field to a nullable field in the DTO.

Domain object identifiers

You may have noticed that our domain classes are lacking an identifier which will mapped to a primary key in the DTO. This is by design: Primary keys are often automatically generated by the database. When we create a new domain object, it doesn't have an identifier yet. We could model it as an Option, but this wouldn't accurately reflect reality — a domain object never has an ID before it is persisted, and it always has an ID when it has been retrieved from the database. Instead, we wrap our domain object in a separate object to augment it with the ID:

@Value.Immutable
public interface WithId<ID, A> {

    ID getId();

    A getData();

}

We will use value objects for our IDs as an additional safety measure. Luckily, Immutables supports them out of the box by means of wrapper types:

@Value.Immutable @Wrapped
abstract class _BookId extends Wrapper<Integer> {}

@Value.Immutable @Wrapped
abstract class _CharacterId extends Wrapper<Integer> {}

Based on these definitions, the Immutables annotation processor will generate the concrete classes BookId and CharacterId.

Last but not least, we define a many-to-many relation between books and characters:

@Value.Immutable @Tuple
public interface BookToCharacter {

    BookId getBookId();

    CharacterId getCharacterId();

}

Data transfer objects (DTOs)

Due to the limitations of JPA, in addition to our domain classes, we have to use dedicated mutable classes for the data transfer objects. This results in quite a bit of knowledge duplication and boilerplate. If you know a solution for this problem, I would be very happy to hear about it in the comments section. It would more or less render this article obsolete, which I wouldn't be particularly unhappy about.

Since the DTOs are significantly more verbose than the domain objects, I will only show the most relevant sections.

@Entity
public class BookDto implements WithIdDto<BookId, Book> {

    @Override
    public WithId<BookId, Book> toEntity() { … }

    @Id
    @GeneratedValue(strategy = GenerationType.AUTO)
    private int id;

    private String title;

    // Getters and setters
}

@Entity
public class CharacterDto implements WithIdDto<CharacterId, Character> {

    @Override
    public WithId<CharacterId, Character> toEntity() { … }

    @Id
    @GeneratedValue(strategy = GenerationType.AUTO)
    private int id;

    private String name;

    @Enumerated(EnumType.ORDINAL)
    @Convert(converter = OptionConverter.class)
    private Option<Species> species;

    // Getters and setters

}

Both of these classes implement the WithIdDto interface, which defines the toEntity() method. This method allows us to convert a DTO to the domain object, including the ID which is derived from the primary key.

And, last but not least, the DTO for our relation, which uses a composite primary key. We are using an @IdClass instead of an @EmbeddedId to simplify the mapping from/to the domain class.

@Entity
@IdClass(BookToCharacterDto.BookToCharacterId.class)
public class BookToCharacterDto {

    public static class BookToCharacterId implements Serializable {

        private final int bookId;
        private final int characterId;

        // Constructor and getters
    }

    @Id
    private int bookId;

    @Id
    private int characterId;

    // Getters and setters
}

Mapping between domain objects and DTOs

The heavy lifting is done by MapStruct, we just have to declare our mappers. First, we define a common interface for all domain-object-to-DTO mappers:

public interface DtoMapper<E, D> {

    D toDto(final E entity);

    E fromDto(final D dto);

}

Now we declare the individual mappers:

import static org.mapstruct.factory.Mappers.getMapper;

public final class Mappers {

    @Mapper(uses = BookIdMapper.class)
    public interface BookMapper
            extends DtoMapper<Book, BookDto> {
        BookMapper instance = getMapper(BookMapper.class);
    }

    @Mapper(uses = CharacterIdMapper.class)
    public interface CharacterMapper
            extends DtoMapper<Character, CharacterDto> {
        CharacterMapper instance = getMapper(CharacterMapper.class);
    }

    @Mapper(uses = { BookIdMapper.class, CharacterIdMapper.class })
    public interface BookToCharacterMapper {
        BookToCharacterMapper instance =
                getMapper(BookToCharacterMapper.class);
        BookToCharacterDto toDto(final BookToCharacter entity);
    }

}

Note that BookToCharacterMapper doesn't extend DtoMapper; for some reason MapStruct refused to generate the mapper class.

I implemented a custom mapper for each wrapper type. There is probably a way to get rid of this boilerplate, but it didn't bother me enough yet to spend more time on this issue:

public class BookIdMapper {

    public BookId asBookId(final int id) {
        return BookId.of(id);
    }

    public int asInt(final BookId bookId) {
        return bookId.value();
    }

}

Finally, let's take a quick look at the implementation of the toEntity method in our DTOs:

@Entity
public class BookDto implements WithIdDto<BookId, Book> {

    @Override
    public WithId<BookId, Book> toEntity() {
        return ImmutableWithId.of(
                BookId.of(id),
                Mappers.BookMapper.instance.fromDto(this)
        );
    }

    // …
}

Querying data

Now that we have declared our domain objects and defined how they are persisted, we can start using them. I won't go into the details of building queries with Jinq — this is covered extensively in the documentation — and rather just show some example code.

public class BookService extends AbstractPersistenceService {

    …

    public Stream<P2<
                WithId<BookId, Book>,
                WithId<CharacterId, Character>
            >> getBooksWithCharacters() {
        return Stream
                .iterableStream(books(entityManager)
                .leftOuterJoin(
                        (b, source) -> source.stream(BookToCharacterDto.class),
                        (b, r) -> b.getId() == r.getBookId()
                )
                .leftOuterJoin(
                        (p, source) -> source.stream(CharacterDto.class),
                        (p, c) -> p.getTwo().getCharacterId() == c.getId()
                )
                .toList())
                .map(TupleOps::p2)
                .map(p -> p
                        .map1(TupleOps::p2)
                        .map1(P2::_1)
                        .map1(WithIdDto::toEntity)
                        .map2(WithIdDto::toEntity));
    }

}

At this level, we still have to deal with the DTO classes, but only to build queries, so there is no danger of introducing mutable state.

The joins result in nested Jinq tuples. At the end we transform these tuples into FunctionalJava tuples, and map the contained DTOs to their respective entities. In a real-world project, I would make this code generic and reuse it across all services.

Using the service

Now we can take a look at our test method, which persists and reads domain objects using the service:

    @Test
    public void testPersistence() {
        transactional(() -> {

            final WithId<BookId, Book> book =
                bookService.createBook(ImmutableBook.of("Night Watch"));

            final Effect1<Character> addCharacter = (ch) -> {
                final WithId<CharacterId, Character> chRead =
                    bookService.createCharacter(ch);
                bookService.addCharacterToBook(
                    book.getId(), chRead.getId());
            };

            final Character vimes = ImmutableCharacter
                .of("Samuel Vimes", some(HUMAN));
            final Character nobby = ImmutableCharacter
                .copyOf(vimes)
                .withName("Nobby Nobbs")
                .withSpecies(none());

            addCharacter.f(vimes);
            addCharacter.f(nobby);

            final Stream<P2<
                WithId<BookId, Book>,
                WithId<CharacterId, Character>
            >> booksWithCharacters =
                    bookService.getBooksWithCharacters();

            assertTrue(booksWithCharacters.isNotEmpty());

            booksWithCharacters.foreach(tuple((bk, ch) -> {
                System.out.printf("%s (%s)%n",
                    ch.getData().getName(),
                    ch.getData().getSpecies());
                return Unit.unit();
            }));

        });
    }

Summary

We have managed to address some critical pain points, in particular we can map between immutable domain objects and DTOs with a reasonable amout of boilerplate and uild type-safe queries using Java code.

Evidently, there is quite a bit of room for improvement:

Our implementation supports both the Builder style (for MapStruct) and the Tuple style (for our code). Using the Builder style can lead to runtime errors since it doesn't ensure at compile time that all fields are initialised. Please leave a comment if you know how to get MapStruct to use the Tuple style for mapping.
The error messages emitted by the MapStruct and Immutables annotation processors leave a lot to be desired. I'm not particularly fond of meta programming in general; this experience confirms my preconception. Maybe it is possible to get more information by setting compiler arguments, but my attempts were unsuccessful.
Jinq queries are not completely type-safe, there can still be runtime errors ("Could not analyze lambda code").
We are not treating effects as first-class citizens. The Functional Java library provides an IO type for this purpose, but this is a topic for a future article.

Interpreting Tagless Final DSLs with Eff

Andreas Jim — Wed, 13 May 2020 12:03:21 +0000

Liquid syntax error: 'raw' tag was never closed

Cooking with Monads

Andreas Jim — Wed, 13 May 2020 11:42:34 +0000

Liquid syntax error: 'raw' tag was never closed

Effects and the Illusion of Correctness

Andreas Jim — Fri, 01 May 2020 09:56:12 +0000

Imperative programming makes it easy to write complex programs, but hard to write correct ones. One reason for this is that imperative languages refuse to properly deal with effects. The article shows why effects should treated as first-class citizens and how things can be improved by switching from imperative to functional programming.

Every useful computer program interacts with the world outside the actual source code, typically by reading and writing data from storage media and over network connections. Programmers call such an interaction an effect. Every time a program performs an effect, it leaves the safe, predictable environment of the program flow — a file can be missing, a server can be down. The inherent dangers of effects are undeniable, therefore how a program deals with effects makes all the difference.

Effects in Imperative Programming

Many software developers begin their journey with imperative programming. Usually it is immediately obvious what an imperative program is intended to do. Therefore such programs look simple, straightforward, and not intimidating.

The basis of each imperative program is a mental model of the execution flow of the program, typically derived from the “main success scenario” of a use case in the requirements specification. Since our cognitive capacity is limited, this model is — and has to be — a simplified one; nobody is capable of considering all possible execution flows of a reasonably complex real-world program at the same time.

Now, this mental model has to be expressed in the form of statements. As an inexperienced programmer, you approach this like an impatient and ignorant manager: You tell your employees (or in this case, the machine) what to do and you can’t be bothered with all the little risks and pitfalls that executing the tasks might actually entail. The fatal problem is that the programming language gives you the illusion that you have created a correct program, and you let yourself succumb to this illusion due to your lack of experience or dedication.

Consider the following example:

import scala.io.Source

def readFileImp(fileName: String): List[String] = {
  val source = Source.fromFile(fileName)
  val lines = source.getLines.toList
  source.close
  lines
}

The readFileImp method is clearly intended to read a list of lines file. But the fact that I'm using the verb intended indicates that this is not what the program actually does, at least not in all situations. What will happen if the file doesn't exist? In this case the program doesn't do what you intend it to do.

The problem here is that imperative languages usually don’t explicitly address the subject of effects. They don’t distinguish between safe and unsafe operations. The execution of each line could potentially result in an error. The programming language couldn’t care less — it doesn’t support you in the difficult task of writing correct software. Instead, it shifts the cognitive burden to you, the programmer. When writing or reviewing code, you have to continuously think about whether it could contain any hidden problems. This makes it incredibly hard to produce correct code — for a seasoned developer it is very stressful, for a beginner it is virtually impossible.

Most imperative programming languages provide the developer with a crutch to handle those hidden problems: Exceptions. The mere term spotlights the fact that imperative languages promote the practice of reflecting only a certain intended scenario in the source code. The developer is not supposed to clutter the code with the handling of undesirable cases. If a so-called exception occurs, it just stops the execution flow, potentially leaving the program in a hazardous state, and implicitly bubbles up the call stack, until a benevolent piece of code shows the kindness of handling it or presenting it to the user. If no such piece of code exists, the application gives up and quits. It is up to you, the programmer, to check the documentation of each and every called method for whether it might throw an exception. A programmer may think of exception handling as a safety net, but in reality it just moves the inherent risks of effects from plain view, where it belongs, to a hidden, convoluted side track outside the actual program flow.

The illusion of correctness that characterises imperative programs comes at a steep price. You can choose which price to pay: Either you live with the illusion at the expense of correctness (which typically involves continuous and extensive bugfixing down the road), or you try to maximise the correctness of your program by carefully examining each line of it and writing a ton of unit tests.

Effects in Functional Programming

The functional programming world has a term for code which contains hidden traps: impure. In contrast, a pure piece of code is innocent — it reveals everything it does in its signature, without hiding any dirty details. Every time you call it with certain parameters you can expect the same result and don’t have to worry about any unforeseeable effects. This property of functional programs is called referential transparency. Any combination of pure functions is again pure, but a single impure piece of code will compromise the purity of the whole program. In languages like Scala which support both the imperative and the functional paradigm, it is crucial to take care not to incorporate any impure code into a program. At times, this can be challenging; especially after years of programming imperatively. But in the end it is well worth it.

For comparison, here’s a typical functional program offering the same functionality as our example above:

import cats.effect.IO
import scala.io.Source

def readFileFunc(fileName: String): IO[Either[Throwable, List[String]]] =
  (for {
    source <- IO(Source.fromFile(fileName))
    lines <- IO(source.getLines.toList)
    _ <- IO(source.close)
  } yield lines).attempt

Technically, this program is also not free of errors, e.g. the source will not be closed if an error occurs while the file content is read, but it should be sufficient to illustrate some basic concepts.

Let’s see how this readFileFunc function differs from the readFileImp "function" in the first example. readFileImp is actually not a function: It claims to return a list of strings, but in reality it is only able to do so if the file actually exists. In contrast to that, the readFileFunc is honest about its return type. It returns an IO, which means that the function doesn't actually do anything, it only produces a "process" object which can be executed (e.g. using the unsafeRunSync method, whose name implies that you're now leaving the safe territory of referential transparency). The readFileFunc always produces the same process object, in any circumstance. Furthermore, the result type of the IO is either a Throwable (i.e. an error object) or a list of strings, which conveys that the execution might actually fail.

As a functional programmer, you safely compose your program by combining pure functions. Some of these functions might return unsafe processes (like the IO objects in our example program), but this is explicitly stated in their return type, which means that the caller is always aware of it. Only at the very last moment, after the composed program has been invoked and an effectful process is produced – commonly phrased "the end of the world" – this process and its effects are executed under strict observation.

The following picture illustrates how imperative and functional programming languages handle effects differently. The explosion symbolizes the execution of an effect; the bomb symbolizes an effect object which is waiting to be executed.

Icons by Freepik and Vectors Market from www.flaticon.com

The implicit handling of effects turns the whole imperative program into a minefield, whereas in a functional program effects are explicitly created, composed and executed “at the end of the world”, in a constricted and controlled environment.

Scala Libraries

To help you get started with managing effects, here’s a short list of Scala libraries dedicated to this purpose:

eff — Extensible effects based on the “free-er” monad
cats-effect — I/O effects for the cats ecosystem
Scala Effect — Structuring effectful programs in a functional way

Conclusion

While imperative programming hides complexity at the expense of correctness, functional programming aims at ensuring correctness while embracing complexity. If complexity cannot be hidden, it has to be managed. This might be one of the reasons why functional programming is considered to be so hard to learn: A beginner in imperative programming will implement a complex process using a seemingly simple program on the first day, because most of the complexity is implicit and hidden. A beginner in functional programming will only be able to express complex processes after several weeks or months of learning, because he has to gradually build up his knowledge about how to represent all the different aspects of the complex processes in his program. But from the very beginning of this journey, correctness and robustness will be essential and non-negotiable characteristics of his programs.

I would be very happy to hear about your opinion in the comments section. Thanks a lot for reading!

Cover image: Danger: Hard Hat Area by Kevin Jarrett on Unsplash

Tagless Final in Scala: Best Practices

Andreas Jim — Tue, 21 Apr 2020 08:44:11 +0000

Notwithstanding some criticism, the tagless final pattern continues to gain popularity among the Scala community. I want to share some things I learned while working with tagless final for about two years.

The tagless final pattern enables the programmer to declare modules with a very specific set of functions, and compose programs based on these modules. The foremost concerns of this approach are correctness and predictability. Today’s article is a collection of best practices and lessons learned and will hopefully help you utilize the pattern to its full potential and avoid any common pitfalls. The concepts and principles presented here are nothing new and will be familiar to every seasoned programmer; the article merely provides some guidance how to apply them in the context of tagless final programs.

Coding Style Recommendations

Naming

Let’s start with the most controversial of them all — naming. Although not directly related to tagless final, I find it worth sharing that the following naming scheme has proved to be practical:

ch.becompany.authn.AuthnDomain
ch.becompany.authn.AuthnDsl
ch.becompany.authn.AuthnService
ch.becompany.db.DbDomain
ch.becompany.db.DbDsl

The package and class names are redundant (as opposed to, e.g., ch.becompany.authn.Dsl), but this scheme has some advantages:

Autocomplete is a bit quicker since you don’t have to cycle between alternatives.
Generally, since each DSL has a unique name, no imports have to be renamed to avoid name collisions. This ensures that the same class name is used everywhere in the codebase, which aids readability.
Even when you need only one DSL in the scope, you can immediately see which DSL it is.

Context Bound Style and the apply Function

Scala offers two syntactic styles for declaring implicit dependencies: implicit parameters and context bounds. I generally prefer the context bound style because it makes the constraints for a type parameter immediately obvious, and it doesn’t require naming parameters. Here’s an example of the context bound style with multiple type parameters:

def registerUser[
F[_] : Monad : AuthnDsl,
G[_] : LogDsl
]: F[Unit] = ???

The only disadvantage of the context bound style that I’m aware of is that sometimes the compiler has trouble inferring the type parameter when the function is used, and requires a type ascription.

The companion object of the DSL trait should provide an apply function which can be used to resolve the DSL instance in the current implicit scope:

object AuthnDsl {
  def apply[F[_]](implicit ev: AuthnDsl[F]): AuthnDsl[F] = ev
}

This style has the additional benefit that the type of the DSL, in this case F, is immediately apparent:

AuthnDsl[F].register(…)

Design Considerations

Constraints and Overconstraining

A core concept of tagless final is declaring dependencies. The following function declares a dependency to an AuthnDsl instance for the type F:

def registerUser[F[_] : AuthnDsl]: F[Unit] =
  AuthnDsl[F].register(…)

Each dependency can also be viewed as a constraint on the type F, since it reduces the set of all possible concrete type instances for F. Each function should only declare those constraints that its implementation actually needs, for the following reasons:

The fewer constraints a function has, the more types it supports. In other words, functions with fewer type constraints are more generic and can be applied in more situations.
Each additional constraint increases the capabilities of a function. The more capabilities a function has, the less predictable it becomes.

The relationship between constraints and reasoning is one of the key benefits of programming with effects and the tagless final approach: Given that its implementation is disciplined, i.e. it doesn’t directly invoke any side effects, it is possible to predict what operations the resulting effect instance can contain.

Consider the following example:

def registerUser[F[_] : AuthnDsl : EmailDsl : Monad]: F[Unit] = ???

Just by looking at the function signature, without knowing anything about its implementation, you can be sure that the effect instance returned by this function can only

Execute effects related to authentication,
Execute effects related to e-mail, and
Combine the results using monadic operations.

Now let’s see what happens when we replace the Monad constraint with the Sync constraint from the cats-effect library:

def registerUser[F[_] : AuthnDsl : EmailDsl : Sync]: F[Unit] = ???

This change opens Pandora’s box. The function can now suspend arbitrary side effects in the returned effect instance — it could, to quote a coworker, “launch a rocket”. In addition, it can potentially instantiate other DSL interpreters which declare the Sync constraint. It is impossible to know what executing the effect instance returned by this registerUser function will actually do.

The practice of declaring unnecessary constraints, also called overconstraining, violates the principle of least power and should generally be considered a code smell.

Type classes and Coherence

Sometimes an application requires multiple implementations for a service. Consider for instance our authentication DSL: Our application may provide authentication against multiple providers, e.g. an LDAP back-end and a user table in a database.

Consider our authentication DSL. Let’s assume that we want to provide a function to authenticate against multiple back-ends. The obvious solution is to implement a dedicated DSL interpreter for each authentication mechanism:

import cats.Monad
import cats.implicits._
import ch.becompany.Domain.User

object AuthnService {

  def authenticate[F[_]:Monad](email: String,
                               password: String,
                               authenticators: AuthnDsl[F]*):
      F[Either[String, User]] =
    authenticators.toList
      .collectFirstSomeM(
        _.authenticate(email, password).map(_.toOption)
      )
      .map(_.toRight("Authentication failed"))

}

While this is entirely possible using the tagless final pattern, it is only recommended if the DSL is not considered a type class. The coherence principle of type classes states that, for each type class Dsl[F[_]], there may only be one interpreter for a specific type F in scope. The main goal of coherence is safety: Using different type class interpreters (e.g. different Order interpreters for the Int type) can lead to incorrect programs. Even though Scala doesn't enforce coherence, it is still advisable to keep this principle in mind before providing multiple interpreters for a DSL which is encoded as a type class.

Modularisation using DSLs

Most programs of a certain complexity can be expressed as a series of layers. The lowest layers deal with technical intricacies and are nowadays usually hidden in third-party libraries. When we move up the stack of layers, technical aspects give way to higher-level functionality that more directly represents the business requirements of our program.

Create Instances at the End of the World

In functional programming, the top-level or main code block of the application is often referred to as the end of the world. This is the place where everything happens that cannot be postponed or delegated anywhere else, such as executing side effects or instantiating DSLs. Program code should usually never create a DSL instance, but always consume it as a dependency. This way, the client code (ultimately the main block or test case) has the freedom to provide the desired implementations. Furthermore, if a DSL instance requires an expressive dependency à la cats.effect.Sync, passing the instance instead of the dependency limits the power of the enclosing module. Compare for instance the following functions:

def login[F[_] : Sync](username: String,
                       password: String): F[Unit] = {
  // requires Sync
  implicit val authnDsl: AuthnDsl[F] = AuthnDsl.interpreter[F] 
  authnDsl.authenticate(username, password)
}

def login[F[_] : AuthnDsl](username: String,
                           password: String): F[Unit] =
  authnDsl.authenticate(username, password)

The first example could potentially suspend arbitrary side effects in F and is therefore, judged solely based on its signature, much less predictable than the second example. Furthermore, since it hard-codes the DSL implementation, it cannot be used in conjunction with other AuthnDsl interpreters, such as mocks.

Another benefit of creating a DSL instance at the end of the world, i.e. in a single place, is the guarantee that the same implementation is being used throughout the complete application.

Minimize DSL Scope

When modularizing a program using DSLs, each DSL should ideally only address a single concern. This principle is congruent with the Unix philosophy — make each program (or in our case, DSL) do one thing well. Minimizing the scope of DSLs means that code which uses it can wield less power. Furthermore, implementations generally require fewer dependencies, which improves code safety and predictability.

Contain Dangerous Dependencies

As we have mentioned before, some dependencies are particularly expressive and therefore powerful. One prominent category are those which can be used to suspend arbitrary side effects, such as Sync, Async or IO from the cats-effect library. A code unit which has access to these dependencies wields virtually limitless power and is therefore unpredictable. As a consequence, it is desirable to limit the usage of these dependencies, if possible to the main block of the application.

Ideally, only the lowest-level technical DSLs which directly rely on side effects like I/O should be given the power of suspending arbitrary effects. If at one point you find yourself in a situation where you require access to Sync or a similar class, think about whether it would be appropriate to factor out the operation into a dedicated low-level DSL, and instantiate it at the end of the world.

DSL vs. Service

When should you implement a DSL, and when is it better to use a plain object (which we will call a service) to group a set of related functions? As we have already discussed, one reason to introduce a DSL is to hide precarious implementation dependencies such as cats.effect.Sync. Another purpose is to provide polymorphic modules, i.e. modules that have different implementations for different types – or, in some cases, can even have multiple implementations for a single type.

A DSL has a key drawback compared to a service: Each interpreter of the DSL has to implement every function of the DSL. All functions of the interpreter share a single set of constraints — the constraints declared by the function producing the interpreter. If some function implementations don’t use all of these constraints, these functions are technically overconstrained.

Because of the drawback mentioned above, only polymorphic functions — i.e. those which require different implementations in different interpreters — should be declared in the DSL. Everything else can be provided by a service object.

Error Handling

When it comes to error handling and propagation, there are basically two choices: Declaring the error channel in the signatures of the DSL functions, or using a constraint.

The first approach is straightforward:

trait AuthnDsl[F[_]] {
  def authenticate(username: String,
                   password: String): F[Either[Error, User]]
}

This option is particularly useful if the client code of the function is expected to be interested in any potential errors and to be able to handle them appropriately. This certainly applies to an authentication failure.

With the second approach, the declaration of the DSL doesn’t concern itself with error propagation and leaves this subject to the interpreter. Here’s an example:

trait AuthnDsl[F[_]] {
  def authorize(user: User): F[List[Role]]
}

object AuthnDsl {
  def interpreter[F[_] : MonadError[Throwable, ?]] = ???
}

Here the implementation declares a MonadError constraint, which means that the instance of the type F is expected to provide an error handling mechanism for a specific error type, in this case Throwable. This approach makes sense for errors which probably can't be handled by the client code and have to be propagated, sometimes even to the end of the world. In our example this would for instance be a network communication error occurring when accessing an LDAP server. As mentioned in the previous section, this approach results in overconstraining in case some of the DSL members require error handling and others don't, an implication to keep in mind when designing your DSLs.

Of course both approaches can be combined, e.g. using an error channel in the return type for business errors and an implementation constraint for low-level technical errors.

DSLs for Common Side Effects

As we have learned earlier, having direct access to low-level type classes for expressing side effects — like cats.effect.Sync – is potentially dangerous. For this reason it is advisable to provide (or use existing) DSLs for frequently used effectful functionality like logging or generating timestamps and UUIDs. It is also worth considering concealing functions from third-party libraries requiring expressive dependencies, such as http4s, behind façade DSLs to avoid leaking Sync & friends into the codebase.

Limitations

I came across a couple of limitations which I haven’t found any solutions or workarounds for. Please don’t hesitate to provide your insights in the comment section.

It is not possible to abstract over a set of constraints — each function has to declare the complete set independently.
It is not possible to identify any unused constraints in IntelliJ IDEA, a circumstance particularly problematic since unused constraints implicate overconstraining and are therefore a code smell. Scala 2.13 provides the compiler option -Wunused:implicits, but I haven't tried it yet and I don't know whether it works with context bounds.

Open Source Libraries

Here is a list of some freely available libraries for the cats-effect ecosystem. Some of them provide DSLs directly, others have to be wrapped in a DSL to be used with the tagless final pattern.

cats-tagless — Transform and compose tagless final encoded algebras
log4cats — Logging (Logger DSL)
fuuid — Generating UUIDs
pureconfig — Loading configuration files
doobie — Database access
http4s — HTTP (Client DSL)
fetch — Data access

Final Words

Despite its shortcomings, I believe that tagless final is a valuable addition to the toolbox of the functional Scala programmer. I hope this article contained some helpful advice. I would be very happy to hear about your experience with tagless final — please feel free to use the comment section to let us know about your thoughts, insights and additional best practices. Thank you very much for reading!

Cover image: Photo by Josue Isai Ramos Figueroa on Unsplash

Structuring Functional Programs with Tagless Final

Andreas Jim — Thu, 28 Mar 2019 12:08:40 +0000

Monads are valuable tools for handling various concerns in functional programs. In this article we show how domain-specific languages and the Tagless Final pattern can be utilised to build modular monadic programs.

Domain-Specific Languages and Interpreters

Domain-specific languages (DSLs) are a popular approach for modularising functional programs. A DSL is a set of functions which address a particular concern — this can be anything from an interface to a subsystem to cross-cutting concerns like logging. DSLs are usually layered, i.e. high-level DSLs (for expressing business processes) are built on top of lower-level DSLs (for accessing databases or connecting to remote APIs). In functional programming, DSLs are often called algebras, hinting at the concept’s origin in category theory.

If you are familiar with object-oriented programming, you can consider a DSL an analogy to an interface: The DSL defines the capabilities of a software module, without providing a concrete implementation. In functional programming, the implementation of the DSL is called an interpreter. An interpreter implements each function of the DSL.

Monads and Separation of Concerns

As outlined in the blog post Cooking with Monads, monads provide a way of structuring functional programs. In functional programming, we often use monads to explicitly handle certain aspects (“concerns”) of our program without having to express this aspect in the program code itself. Monads allow us to isolate specific concerns from our business logic, which leads to a better separation of concerns in our programs.

Some examples:

The Reader monad allows to pass a context, for instance a configuration, which all computation steps can access.
The State monad passes state information from one call to the next without the need for mutable data structures in our program code.
The Task monad provides a way to deal with concurrency, side effects and potential errors.

Each of these monads support us in relieving the business logic from some of the responsibility of dealing with the respective concerns.

Monadic DSLs and Tagless Final

In Scala, the individual functions of a DSL typically have monadic return values, which has the benefit that programs can be written as for-comprehensions. The Tagless Final pattern provides a way to declare a DSL in a generic way, without specifying a particular monad. Multiple interpreters can exist for a DSL, every one potentially targeting a different monad.

This approach has various benefits:

When writing a program based on Tagless Final DSLs, the target monad of the program can be changed in the future. This way, new features like parallel computation can be introduced without modifying the program itself.
The DSL can be used with different interpreters. A typical use case is providing an alternative interpreter for testing purposes, using a local data store instead of accessing an external system.
Multiple DSLs can be combined in a single for-comprehension by chosing interpreters for the same target monad.

Our Example

In our example code we will model a DSL called authn which provides functions for registering and authenticating users:

def register(email: String @@ EmailAddress, password: String):
    Either[RegistrationError, User]

def authn(email: String @@ EmailAddress, password: String):
    Either[AuthnError, User]

In case you’re wondering, String @@ EmailAddress is a tagged type, denoting that email is a string with the sole purpose of modelling an e-mail address.

You find the source code for the example application on GitHub.

The package structure looks as follows. We are coupling our code based on functional design, meaning that code with common functionality goes in the same package.

ch.becompany
  authn           Authentication functionality
    domain        Authentication domain code
    Dsl           Authentication DSL
  shapelessext    Extensions to the shapeless library
  shared          Shared code
    domain        Shared domain code
  Main.scala      Our main application

To run the example, execute the following command in the console:

sbt run

Modeling DSLs with Tagless Final

In the Tagless Final pattern, a DSL is modelled as a trait with a single type parameter, which has to be a type constructor with arity 1. We will call this type constructor F[_]. At this moment, it's actually not required that F is a monad. Later on, when implementing an interpreter for our DSL, the target monad of the interpreter will take the place of F.

Let’s model our authentication DSL in this style:

package ch.becompany.authn

trait Dsl[F[_]] {

  def register(email: String @@ EmailAddress, password: String):
      F[Either[RegistrationError, User]]

  def authn(email: String @@ EmailAddress, password: String):
      F[Either[AuthnError, User]]

}

We see that the return values of all DSL functions are wrapped in the container F. In the following program, the compiler infers the type parameter F[_] from the return type of the registerAndLogin function.

package ch.becompany

object Main extends App {

  def registerAndLogin[F[_] : Monad](implicit authnDsl: AuthnDsl[F]):
      F[Either[AuthnError, User]] = {

    val email = tag[EmailAddress]("john@doe.com")
    val password = "swordfish"

    for {
      _ <- authnDsl.register(email, password)
      authenticated <- authnDsl.authn(email, password)
    } yield authenticated
  }

}

The function signature ensures that F is a monad (by requiring the existence of an implicit value of the type Monad[F]). Therefore the functions of our DSL can be used in for-comprehensions. Even at this stage, we don't specify a concrete type for F; the registerAndLogin function could actually be part of a higher-level DSL.

Besides the Monad instance, the function requires an additional parameter: An instance for the authentication DSL (also called an interpreter), typed with the common type F. The parameter is declared as implicit to allow automatic resolution by the compiler; we will look into this in detail when we talk about interpreters.

Interpreters

Now that we have defined the syntax of our DSL in the respective trait, we have to implement the semantics. With the Tagless Final technique, this is done in an interpreter. For each DSL, multiple interpreters can exist; each of them targeting a specific type. Interpreters are typically modelled as type classes, so they can be automatically resolved by the compiler when a DSL is used with the respective target type.

In the beginning we will choose a target type which make it easy to test the concepts in a simple, self-contained program. In a real-world scenario, you would probably follow the same approach: Start with providing easy-to-use interpreters for your DSLs which can be utilised in test cases. This approach is comparable to implementing mocks, with the difference that our interpreter is a full-featured implementation of the DSL.

Later on we can proceed to implementing interpreters for more sophisticated target types covering additional concerns like concurrency and side-effects.

Interpreter for the Authentication DSL

We implement the interpreter in the companion object of the Dsl trait, thereby supporting the implicit resolution mechanism of the compiler.

package ch.becompany.authn

trait Dsl[F[_]] {
  …
}

object Dsl {

  type UserRepository = List[User]

  type UserRepositoryState[A] = State[UserRepository, A]

  implicit object StateInterpreter extends Dsl[UserRepositoryState] {

     override def register(email: String @@ EmailAddress, password: String):
         UserRepositoryState[Either[RegistrationError, User]] =
       State { users =>
         if (users.exists(_.email === email))
           (users, RegistrationError("User already exists").asLeft)
         else {
           val user = User(email, password)
           (users :+ user, user.asRight)
         }
       }

     override def authn(email: String @@ EmailAddress, password: String):
         UserRepositoryState[Either[AuthnError, User]] =
       State.inspect(_
         .find(user => user.email === email && user.password === password)
         .toRight(AuthnError("Authentication failed")))
   }

}

We want to store the registered users in a list, so our UserRepository type is a simple list of users:

type UserRepository = List[User]

We will utilise the State monad for passing the user repository from one DSL function call to the next:

type UserRepositoryState[A] = State[UserRepository, A]

Now we define the StateInterpreter, an interpreter for the authentication DSL targeting the UserRepositoryState monad. Note that the object is declared with the implicit modifier, which makes it visible to the compiler when a DSL interpreter for this target type is requested.

implicit object StateInterpreter extends Dsl[UserRepositoryState] {

   override def register(email: String @@ EmailAddress, password: String):
       UserRepositoryState[Either[RegistrationError, User]] =
     State { users =>
       if (users.exists(_.email === email))
         (users, RegistrationError("User already exists").asLeft)
       else {
         val user = User(email, password)
         (users :+ user, user.asRight)
       }
     }

   override def authn(email: String @@ EmailAddress, password: String):
       UserRepositoryState[Either[AuthnError, User]] =
     State.inspect(_
       .find(user => user.email === email && user.password === password)
       .toRight(AuthnError("Authentication failed")))
   }
}

Running the Program

Now that we have provided an interpreter for our DSL, we can execute the registerAndLogin program which we have implemented in our Main application.

package ch.becompany

object Main extends App {

  def registerAndLogin[F[_] : Monad](implicit authnDsl: AuthnDsl[F]):
      F[Either[AuthnError, User]] = {

    val email = tag[EmailAddress]("john@doe.com")
    val password = "swordfish"

    for {
      _ <- authnDsl.register(email, password)
      authenticated <- authnDsl.authn(email, password)
    } yield authenticated
  }

  val userRepositoryState = registerAndLogin[UserRepositoryState]

  val result = userRepositoryState.runEmpty

  val (users, authenticated) = result.value

  println("Authenticated: " + authenticated)
  println("Registered users: " + users)

}

By calling registerAndLogin with the UserRepositoryState type parameter value, we instruct the compiler to resolve the interpreter – declared as the implicit parameter authnDsl – for the UserRepositoryState target monad:

val userRepositoryState = registerAndLogin[UserRepositoryState]

We use the runEmpty method to pass an empty list of users as the initial state:

val result = userRepositoryState.runEmpty

The runEmpty method returns an instance of the Eval monad, whose computation produces a tuple consisting of the final state (in our case the user repository containing all registered users) and the return value of the program (in our case the authentication result). Now we can finally extract these values using the value method of the Eval monad, and print the result:

val (users, authenticated) = result.value
  println("Authenticated: " + authenticated)
  println("Registered users: " + users)
}

The output of our program looks as follows:

Authentiated: Right(User(john@doe.com,swordfish))
Registered users: List(User(john@doe.com,swordfish))

Next Steps: Combining Multiple DSLs

To support combining calls from different DSLs in a for-comprehension, all of these functions must return their values in the same monad.

In many cases it is possible to choose a monad which addresses all required concerns, typically side-effects and error handling. Examples are the Task type from the ScalaZ library or the IO type from the cats-effect library.

But to find a generic approach to deal with this restriction actually proves to be quite challenging. One possible solution is using Free monads, for example the Eff monad; this approach which will be presented in an upcoming article.

CSV parsing with Scala and shapeless

Andreas Jim — Mon, 17 Dec 2018 08:14:48 +0000

The shapeless library serves as an excellent foundation for building generic, reusable components. We demonstrate using the types `HList` and `Generic` to parse strings into case classes.

Introduction

This post complements the upcoming article Real-time log processing with Akka streams, which involves processing log entries from a web server log. For convenience and clarity, we want to parse the log file line strings into case class instances, allowing us to process them in a semantic fashion.

There are countless libraries available for parsing strings; we will use scala-csv and shapeless to demonstrate a generic and extensible approach. Some of the code we use is based on the CSV example in the shapeless codebase.

Get the source

The source code of the example project is available on github.

Log format

Our example uses a web server log format with the following components:

Remote IP address
Time in milliseconds (Unix time stamp)
Request path
User agent

This roughly corresponds to the following nginx log configuration:

log\_format my\_log\_format
  '"$remote\_addr","$msec","$request","$http\_user\_agent"';

A log entry looks like this:

"1.2.3.4","1466585706027","/foo","Chrome"

We will use the following case class to represent log events:

Parsing string values into objects

A parser type for individual CSV record elements is defined by the Parser trait. The parse result is either an object of type T or an error message:

For convenience, we provide a parser which handles exceptions and returns an error message. This covers the typical scenario of using parser methods from Java libraries, e.g. Integer.parseInt, which throws NumberFormatException, or DateFormat.parse, which throws ParseException:

The Parsers object provides a trivial parser for strings. The client application can provide additional custom parsers. Our application provides a parser for parsing millisecond expressions into Joda Time Instant objects and a parser for IP addresses:

Parsing CSV records

The LineParser[T] class parses a CSV record, which is represented as a List[String], into an instance of case class T. We use the HList (heterogenous list) type from the shapeless library to express the types of the list elements. The string list is first converted to a HList instance, which in turn is converted to a case class instance.

We will need some types from the shapeless library:

First we define a LineParser[Out] trait which defines the capability of parsing a list of strings into an object of type Out. In the case of an error, we expect that the parsing errors of individual list elements will be aggregated into a list represented by the Left incarnation of the Either result type.

Now we define a companion object for our LineParser trait which provides methods for parsing List[String] instances into HList instances. The methods are declared as implicit, making them available in the LineParsertrait. The object provides a method for each of the List incarnations Nil and Cons (::, although we are using +: due to a name collision with the shapeless :: type).

The hnilParser method is expected to emit an empty list and returns an error if it encounters a list containing one or multiple elements:

The hconsParser method parses a concatenation of a head element (type H) and a tail list (type T) while combining the errors which may occur during the parsing steps accordingly. Note the Parser type tag for the H type which ensures that the list element type H is a member of the Parser type class, i.e. that a parser for this type is provided. The parser is later on obtained using the expression implicitly[Parser[H]].

The following implicit method converts the HList R into the case class A, using the shapeless Generic type. We define an implicit parameter gen, whose type is an instance of the Generic trait with a representation of type R. The method call gen.from converts the HList representation into the desired instance of case class R.

The apply[A] method is parameterised with the type of the expected case class. It uses the implicitly provided parser to parse the list elements. The implicitly available method caseClassParser allows us to use a case class as the type parameter for the apply method.

Putting it all together

The CsvReader[T] class converts comma-separated strings into objects of type T.

The read method parses the lines of an Akka stream source, which emits elements of type String, into objects of type T. Note that the type parameter T has a context bound of type LineParser, which ensures that a parser is available for this type, as explained in the previous section.

The class uses the CSV parser class from the scala-csv library to split lines into lists of strings. In case the line was successfully split into a list of strings, a LineParser[T] is created and its apply method is invoked on the string list.

The CsvReader[T].read method can now be used to transform a source of strings into a source of elements of type T: