DEV Community: Giovanni Di Blasi

Avoid hidden traps in switching between languages: a tale of queries and sequences

Giovanni Di Blasi — Tue, 05 Nov 2019 07:26:42 +0000

It will have probably happened to you to switch between two or more languages or frameworks in a short time.
In these situations, it's important to keep in mind the differences between the languages and the tools that they offer. I'm not speaking about the syntax, but it's important to know the implementation details.

When I work to my side-project I often switch between C# and Javascript. With this post, I would like to share with you some reasoning about how to query sequences in these two languages.

Context

Let's define a task to solve so that we have a practical scenario to talk about.
Imagine to have a list of users (what original example!).
Each user object is first serialized and then encoded in base64.


const encodedUsers = [
        "eyAibmFtZSI6IlBldGVyIiwgImFnZSI6MjAgfQ==",   //{ "name":"Peter", "age":20 }
        "eyAibmFtZSI6IlNpbW9uIiwgImFnZSI6MzEgfQ==",   //{ "name":"Simon", "age":31 }
        "eyAibmFtZSI6IkVyaWMiLCAiYWdlIjo0MCB9"        //{ "name":"Eric", "age":40 }
];

So what we want is to find the first user over 30 years old.
Spoiler: the user is Simon 😄

Let's implement it

It's easy to imagine that we will need some methods as bricks to solve our task:

decode: Given a base64 input it returns the decoded version.
parse: Given a string that represent a user it returns a User object.
isOlderThan: Given an User object it returns true if the user age is greater than a given age.

Define them in C# and in Javascript:

private String Decode(string encodedUser)
{
    Console.WriteLine("decoding..." + encodedUser);
    return Encoding.UTF8.GetString(Convert.FromBase64String(encodedUser));
}

private User Parse(string serializedUser)
{
    var user =  JsonConvert.DeserializeObject<User>(serializedUser);
    Console.WriteLine("parsed..." + user.name);
    return user;
}

private bool IsOlderThan(int age, User user)
{
    Console.WriteLine("check age of: " + user.name);
    return user.age > age;
}

Javascript


const decode = encodedUser => {
    console.log("DECONDING: ",encodedUser);
    return Buffer.from(encodedUser,'base64').toString('utf8');
}

const parse = serializedUser => {
    const user = JSON.parse(serializedUser) 
    console.log("PARSED: ", user.name);
    return user;
}

const isOlderThan = (age, user)=>{
    console.log("CHECK AGE OF: ", user.name);
    return user.age > age;
}

Now that we have defined our methods, we need to transform our base64 strings in users so that we can test their ages.
For the C# implementation we could use LINQ to query our sequence and in Javascript we could use the map() method to apply our methods and first() to find our user:

C# (LINQ)

var userName =  encodedUsers
        .Select(Decode)
        .Select(Parse)
        .First(user => IsOlderThan(30, user))
        .name;

 Console.WriteLine("The first user over 30 years old is " + userName);
}

Javascript


var userName = encodedUsers
    .map(decode)
    .map(parse)
    .find(user=>isOlderThan(30, user))
    .name;

console.log("The first user over 30 years old is ", userName)

This two implementations look very similar, both allow us to apply our methods to the users sequence in a very expressive way.
How you can imagine they give the same result: the first user over 30 years old is Simon 🎉

Deep in dive

We had put some logs on our parse and decode methods, and we are damn curious to take a look at the output, right? 🤓
Ok, let's go!

C# (LINQ)

DECODING: eyAibmFtZSI6IlBldGVyIiwgImFnZSI6MjAgfQ==
PARSED: Peter
CHECK AGE OF: Peter

DECODING: eyAibmFtZSI6IlNpbW9uIiwgImFnZSI6MzEgfQ==
PARSED: Simon
CHECK AGE OF: Simon

The first user over 30 years old is Simon

Javascript


DECONDING:  eyAibmFtZSI6IlBldGVyIiwgImFnZSI6MjAgfQ==
DECONDING:  eyAibmFtZSI6IlNpbW9uIiwgImFnZSI6MzEgfQ==
DECONDING:  eyAibmFtZSI6IkVyaWMiLCAiYWdlIjo0MCB9

PARSED:  Peter
PARSED:  Simon
PARSED:  Eric

CHECK AGE OF:  Peter
CHECK AGE OF:  Simon

The first user over 30 years old is Simon

What??? The outputs are very different 🤔

Although both C# and Javascript provide tools that give us a very similar way to query a sequence, they are implemented with a totally different approach.

The first difference we can observe is the order of execution of our queries: in our C# implementation decode() is executed on the first item of the sequence than parse() is applied on the resulting value, the same happens for the second value and so on.

Instead In the javascript implementation decode() is executed for each item in the sequence, then parse() is applied to each item of the resulting sequence.

But the biggest difference is that in the C# implementation the parse() and decode() methods are never called for the user "Eric" 🤯

In fact, reading from Select documentation:

This method is implemented by using deferred execution. The immediate return value is an object that stores all the information that is required to perform the action. The query represented by this method is not executed until the object is enumerated.

So when we invoke Select passing our parse() and decode() methods we are collecting the queries without execute them!
This explains why they are never executed on the last item of our sequence:
First() methods stop to iterate the sequence to the second user (Simon), so on the third item any query will be executed.

From the Javascript map() documentation page instead we discover that this method uses the immediate execution approach:

map calls a provided callback function once for each element in an array, in order, and constructs a new array from the results.

This means that for every map() execution the query is immediately executed and a new sequence is created.
This is why the queries are executed on every item in our sequence.

If on the one hand the immediate execution could bring to execute avoidable queries, on the other the deferred execution could execute the queries on the same items more than once.
Take a look at this example:

C# (LINQ) - Deferred Execution

var users = encodedUsers
            .Select(Decode)
            .Select(Parse);

Console.WriteLine("The first user over 30 years old is " + users.First(user=>IsOlderThan(30, user).name);
//....
Console.WriteLine("The first user over 35 years old is " + users.First(user=>IsOlderThan(35, user)).name);

}

Output


DECODING: eyAibmFtZSI6IlBldGVyIiwgImFnZSI6MjAgfQ==
PARSED: Peter

DECODING: eyAibmFtZSI6IlNpbW9uIiwgImFnZSI6MzEgfQ==
PARSED: Simon

The first user over 30 years old is Simon

DECODING: eyAibmFtZSI6IlBldGVyIiwgImFnZSI6MjAgfQ==
PARSED: Peter

DECODING: eyAibmFtZSI6IlNpbW9uIiwgImFnZSI6MzEgfQ==
PARSED: Simon

DECODING: eyAibmFtZSI6IkVyaWMiLCAiYWdlIjo0MCB9
PARSED: Eric

The first user over 35 years old is Eric

We notice that decode() and parse() are execute twice on Peter and Simon. This happens because the deferred execution implies that the users variable contains the "Promise" to execute our queries over items, so the queries are executed each time we call first() method.
Implementing the same example in Javascript we notice that users variable contains the execution results of the two map methods, this is the output:

Javascript - Immediate Execution

const users = encodedUsers
    .map(decode)
    .map(parse);


console.log("The first user older than 30 is ", users.find(user=>isOlderThan(30, user)).name);
//...
console.log("The first user older than 33 is ", users.find(user=>isOlderThan(33, user)).name);

}

Output


DECONDING:  eyAibmFtZSI6IlBldGVyIiwgImFnZSI6MjAgfQ==
DECONDING:  eyAibmFtZSI6IlNpbW9uIiwgImFnZSI6MzEgfQ==
DECONDING:  eyAibmFtZSI6IkVyaWMiLCAiYWdlIjo0MCB9

PARSED:  Peter
PARSED:  Simon
PARSED:  Eric

CHECK AGE OF:  Peter
CHECK AGE OF:  Simon

The first user older than 30 is  Simon

CHECK AGE OF:  Peter
CHECK AGE OF:  Simon
CHECK AGE OF:  Eric

The first user older than 33 is  Eric

Look under the hood

I don't mean one implementation is better than the other, but that these are simply two different approaches to solve the same problem.
To not being aware of these differences could make hard for us writing good code: we could introduce unexpected behaviours like the one shown in this example.
So I think that when we approach a new language or framework we should take some time to look what's under the hood.

A Repository Pattern Implementation for Firestore Subcollections

Giovanni Di Blasi — Tue, 17 Sep 2019 05:53:27 +0000

If you already use Google Cloud Firestore, you probably know its data model and you noticed that Subcollections are one of the most interesting features.
I met them for the first time while working on my personal pet-project (a project where I can experiment with new technologies and methodologies).

So, I wondered if there was a way to adapt the Repository Pattern to deal with Firestore Subcollections.
The main aspect to address was that in the Repository Pattern each repository is usually related to a specific model, in other words, it is related to a type. But reading from Google documentation page:

A Subcollection is a collection associated with a specific document.

This means that, for a subcollection, a repository is not related to a type but to a specific Document.

Let's see this experiment results... 🔬

Scenario

To get the point imagine this practical example:
We have to deal with "Houses" and each of these have some "Rooms" (a bathroom, a kitchen, ...). In each room there is the "Furniture" (a bed, two chairs and so on).
To model this scenario we can imagine to implement these entities:

Furniture Item
Room
House

and they will be related in this way:

Houses   <--- a collection
 |
 |--- My-House <---a Document of Houses
        |--Address <---- a field of "My-House" document
        |--Rooms   <----- a Subcollection of "My-House"
            |
            |--Kitchen
            |--Bathroom
            |--Bedroom <--- a document of Rooms
                    |
                    |--FurnitureItems  <--- a Subcollection of Bedroom document
                            |
                            |-- Bed
                            |-- Wardrobe

The Repository

For the sake of simplicity let's focus on these two methods

"Add" adds a document to the collection
"GetById" returns a document by its Id

Speaking in terms of interface we have:

public interface IRepository<T>{
    Either<String,String> Add(T data);
    Either<String, T> GetById(String id);
}

Ignoring for a moment Subcollections, an implementing class could be:

public  class  FirestoreRepository<T> implements IRepository<T>{

    private CollectionReference collection;
    private Class<T> classType;

    public FirestoreRepository(Firestore firestore, String collectionPath, Class<T> classType){
        this.collection = firestore.collection(collectionPath);
        this.classType = classType;
    }

    public Either<String,String> Save(T data){ 
        try{
            ApiFuture<DocumentSnapshot> futureResult = collection.add(data).get().get();
            DocumentSnapshot result = futureResult.get();
            return Either.right(result.getId());
        }catch(Exception ex){
            return Either.left(ex.getMessage());
        }
    }

    public Either<String, T> GetById(String id) {
        try{
            ApiFuture<DocumentSnapshot> futureEntry = collection.document(id).get();
            DocumentSnapshot entry = futureEntry.get();
            return entry.exists()
                ? Either.right(entry.toObject(classType))
                : Either.left("Data not found for id: "+id);
        }catch(Exception ex){
            return Either.left(ex.getMessage());
        }
    }
}

Note:
The GoogleApi methods to retrieve a DocumentSnasphot are async, they indeed returns an ApiFuture (a GoogleAPI implementation of java Future).

For the sake of simplicity, I chose to avoid the async approach blocking the execution until the future is complete.
Yes, I know, It's not a good idea to do in real projects 🤓

The above implementation assumes that all the collection are at root level, in the constructor the collection reference is indeed created from the firestore object:

this.collection = firestore.collection(collectionPath);

This assumption is not correct when we use the firestore Subcollections. A Subcollection is related to a specific document, so we need
a DocumentReference to get one of its collections.
For example:

CollectionReference furnitureCollection = firestore
        .collection("Houses")
        .document("My-House")
        .collection("Rooms")
        .document("bathroom")
        .collection("FurnitureItems");

We notice that we need to chain CollectionReferences and DocumentReference to access to the right Subcollection.

So, as first, we need to modify the constructor of our FirestoreRepository to accept a CollectionReference.

public FirestoreRepository(CollectionReference collection, Class<T> classType){
        this.collection = collection;
        this.classType = classType;
    }

In this way we have removed the assumption that each collection is at root level.

The Repository Factory

At this point let me introduce the RepositoryFactory whose responsibility is:

chain CollectionReferences and DocuementReferences
create an IRepository instance for a specific Collection (or Subcollection).

For example, if we want to get the Repository for the "FurnitureItems"
of the document "bathroom" relative to the "My-House", we expect to have something like:

  IRepository<FurnitureItem> furnitureRepo = repoFactory
        .FromDocument(House.class, "My-House")
        .FromDocument(Room.class, "bathroom")
        .GetRepository(FurnitureItem.class);

    furnitureRepo.Add(new FurnitureItem("chair"));

Otherwise, if we want to query through the House collection we should have:

  IRepository<House> houseRepo = repoFactory.GetRepository(FurnitureItem.class);
  houseRepo.Add(new House());

We notice that the method FromDocument should return a new instance of RepositoryFactory. This allows us to keep a RepositoryFactory reference to access all its Subcollections.
For example consider that a "House", in addition to the "Rooms" collection, has a "Tenants" collection. In this scenario we can write:

RepositoryFactory myHouseRepoFactory = repoFactory.FromDocument(House.class, "MyHouse");

myHouseRepoFactory
    .FromDocument(Room.class, "kitchen") //this do not modify the collection root of myHouseRepoFactory
    .GetRepository(FurnitureItem.class)
    .Add(new FurnitureItem());

myHouseRepoFactory
    .GetRepository(Tenant.class)
    .Add(new Tenant());

Design the RepositoryFactory 📐 ✏️

Bearing in mind everything, our RepositoryFactory needs:

A map to link the Model classType with a collection name
A pointer to create a repository: if the repo is at root level then it should be a firestore instance, a DocumentReference otherwise.
A method to move the pointer to the next DocumentReference returning a new RepositoryFactory instance.
A method to get the IRepository instance for a specified collection (or Subcollection)

The Model-Collection Map

To link each model class type with the relative collection name we use a map like this:

private final static Map<Class<?>, String> collectionMap = Map.of(
        House.class, "Houses",
        Room.class, "Rooms",
        FurnitureItem.class, "Furniture"
    );

The Pointer

Keep in mind that we have two kinds of pointer:

The Firestore instance if the collection is at the root level
A DocumentReference if the collection is relative to a specific Document (a.k.a Subcollection)

We could keep a reference to Firestore and one to a DocumentReference, then we should have some logic to choose what of them to use.

public class RepositoryFactory{
    ....
    private Firestore firestore;
    private DocumentRefernce documentReference;
    private Boolean isRootLevel;

    public RepositoryFactory(Firestore firestore){
        this.firestore = firestore;
        isRootLevel = true;
    }

    private RepositoryFactory(DocumentReference documentReference){
        this.documentReference = documentReference;
        isRootLevel = false;
    }

    ....

    private CollectionReference getCollectionReference(String collectionName){
        return isRootLevel
            ? firestore.collection(collectionName)
            : documentReference.collection(collectionName);
    }
    ....
}

But when move the pointer from the root level to a document level, the firestore reference is no more useful.
Is there a way to keep only the pointer we need?

💡 Functional approach to the rescue!

We could indeed keep only a reference to a function that, given a collection name, retrieves a CollectionReference:

public class RepositoryFactory{
    ....
    private Function<String, CollectionReference> getCollectionReference;
    ....

    public RepositoryFactory(Firestore firestore){
        getCollectionReference = collectionPath -> firestore.collection(collectionPath);
    }

    private RepositoryFactory(DocumentReference document){
        getCollectionReference = collectionPath -> document.collection(collectionPath);
    }
    ....
}

In this way, each instance of RepositoryFactory keeps only the reference to the right pointer.

Move the pointer to a specific Document

To move through DocumentsReference(s) we don't want to modify the existing RepositoryFactory instance but we have to create a new one with the new DocumentReference as pointer:


 public  RepositoryFactory FromDocument(Class<?> classType, String documentId){
        String collectionName = collectionMap.get(classType);
        DocumentReference documentReference = getCollectionReference.apply(collectionName).document(documentId);
        return new RepositoryFactory(documentReference);
    }

Create a Repository for a given Model

At this point we need a method that, given a Model classType, returns a IRepository<> instance:

 public <T> IRepository<T> GetRepository(Class<T> classType){
        String collectionName = collectionMap.get(classType);
        return new FirestoreRepository<T>(getCollectionReference.apply(collectionName), classType);
    }

Pick up the pieces

Finally, our Repository Pattern implementation should be something like this:


//IRepository.java
public interface IRepository<T>{
    Either<String,String> Add(T data);
    Either<String, T> GetById(String id);
}

//FirestoreRepository.java
public  class  FirestoreRepository<T> implements IRepository<T>{

    private CollectionReference collection;
    private Class<T> classType;

    public FirestoreRepository(CollectionReference collection, Class<T> classType){
        this.collection = collection;
        this.classType = classType;
    }

    public Either<String,String> Add(T data){ 
        try{
            DocumentSnapshot result = collection.add(data).get().get().get();
            return Either.right(result.getId());
        }catch(Exception ex){
            return Either.left(ex.getMessage());
        }
    }

    public Either<String, T> GetById(String id) {
        try{
            DocumentSnapshot entry = collection.document(id).get().get();
        return entry.exists()
                ? Either.right(entry.toObject(classType))
                : Either.left("Data not found for id: "+id);
        }catch(Exception ex){
            return Either.left(ex.getMessage());
        }
    }
}

//RepositoryFactory.java
public class RepositoryFactory{

    private Function<String, CollectionReference> getCollectionReference;

    private final static Map<Class<?>, String> collectionMap = Map.of(
        House.class, "Houses",
        Room.class, "Rooms",
        FurnitureItem.class, "Furniture"
    );   

    public RepositoryFactory(Firestore firestore){
        getCollectionReference = collectionPath -> firestore.collection(collectionPath);
    }

    private RepositoryFactory(DocumentReference document){
        getCollectionReference = collectionPath -> document.collection(collectionPath);
    }

    public  RepositoryFactory FromDocument(Class<?> classType, String documentId){
        String collectionName = collectionMap.get(classType);
        DocumentReference documentReference = getCollectionReference.apply(collectionName).document(documentId);
        return new RepositoryFactory(documentReference);
    }

    public <T> IRepository<T> GetRepository(Class<T> classType){
        String collectionName = collectionMap.get(classType);
        return new FirestoreRepository<T>(getCollectionReference.apply(collectionName), classType);
    }
}

To sum up

This is how I implemented the repository pattern to deal with Firestore subcollections.
For this example I chose to use Java, but it should be easy to move to other languages.

Open points

In this post I have overlooked some aspects that could be addressed in separate topics:

How to deal with Google ApiFuture and Java CompletableFuture.
How to deal with queries specific for a model.

Technologies

Java
Firestore
Subcollections
Repository Pattern

Let me know your way to work with subcollections!