DEV Community: neil445

I got called out for using IEnumerable on methods

neil445 — Fri, 02 Apr 2021 04:01:39 +0000

I've always used IEnumerable<T> as the return value type and input value type for methods; currently, I'm working with a team with a standard of using array (T[]) or List<T>. I did try to sneak up a few methods at first with IEnumerable<T>, until I was told to not use it and use the "lightest" implementation (what?) instead; and Resharper complains about possible multiple enumeration on LINQ methods called on an IEnumerable<T>. Since then I've been thinking to dig from memory what I know, and write it down.

Why do I prefer to use `IEnumerable<T>`?

I want the flexibility using interfaces bring.

Say I have this method:

public void DoTasks(IEnumerable<string> tasks)

I can pass in an array, a List, or whatever that inherits IEnumerable, without having to bother converting it as opposed to:

public void DoTasks(List<string> tasks)

public void DoTasks(string[] tasks)

Immutability of the collection.

Most of the time you don't really add/remove anything from the collection, you'd just play with the contents of a list. I know, it sounds trivial but the key here is intent. You don't want to leave code that's tempting the next developer to do something hacky.

Deferred execution

IEnumerable<T> gives you the delicate art of deferred execution. There's already a lot out there explaining what all it is about, I'll just put it simply. Let's just say that IEnumerable<T> is not really there until you needed it, a familiar term that could give you the idea would be "lazy loading".
Let's say we have

var rooms = _roomsService.GetAllGuestRooms();

var vacantRooms = rooms.Select(r => r.Vacant);

foreach(var room in vacantRooms) 
{
   Console.WriteLine($"Room number: {room.Id}")
}

The Select expression is not yet executed here:

var vacantRooms = rooms.Select(r => r.Vacant);

The point where Select expression is executed, or when the IEnumerable<Room> result is "materialized", is here:

foreach(var room in vacantRooms) 
{
   Console.WriteLine($"Room number: {room.Id}")
}

What's the harm?

It's all about avoiding the danger deferred execution can bring. Here's Resharper's explanation why they warn about it, and they are correct about it.

Going back to the previous sample code, let's say that _roomsService.GetAllGuestRooms() looks like this:

public IEnumerable<Room> GetAllGuestRooms() 
{
   IEnumerable<Room> rooms = from r in db.Rooms
   where r.Type == "GUEST"
   select r;

   return rooms;
}

it returns an IEnumerable<Room> and it makes a database query when it's called.
We then add code to print all rooms

var rooms = _roomsService.GetAllGuestRooms();

foreach(var room in rooms) 
{
   Console.WriteLine($"Room: {room.Id}")
}

var vacantRooms = rooms.Select(r => r.Vacant);
foreach(var vacantRoom in vacantRooms) 
{
   Console.WriteLine($"Vacant: {vacantRoom.Id}")
}

Guess how many times it would make a database call? ONCE? No. TWICE. Despite reusing rooms, the code above would make two database calls because rooms was never materialized and stored in memory.

Getting around it

The simple way of getting around is materialize the IEnumerable and store it memory. Use .ToList().

var rooms = _roomsService.GetAllGuestRooms().ToList();

or, you may do this if the db query isn't expensive and doesn't take too long; your call.

public IEnumerable<Room> GetAllGuestRooms() 
{
   IEnumerable<Room> rooms = from r in db.Rooms
   where r.Type == "GUEST"
   select r;

   return rooms.ToList();
}

The above will have the method's return value have an underlying type of List<Room>.

So it's simply using .ToList() to materialize the IEnumerable<T>? Yes and no. I've seen this getting abused, other people put .ToList() left and right just to make sure IEnumerable<T> is materialized. However, it would allocate memory unnecessarily, sure GC would take care of it, but; don't make things work harder, make things work smarter. With that in mind, you have to be smart on what point you want IEnumerable<T> materialized, and use it when only necessary.

Small things that Count

Let's check IEnumerable.Count(). Does it materialize the collection? Should I materialize it first? It depends on the underlying type.

(Reference source code)[https://source.dot.net/#System.Linq/System/Linq/Count.cs] for .Count():

public static int Count<TSource>(this IEnumerable<TSource> source)
{
   if (source == null)
   {

 ThrowHelper.ThrowArgumentNullException(ExceptionArgument.source);
   }

   if (source is ICollection<TSource> collectionoft)
   {
      return collectionoft.Count;
   }

   if (source is IIListProvider<TSource> listProv)
   {
      return listProv.GetCount(onlyIfCheap: false);
   }

   if (source is ICollection collection)
   {
      return collection.Count;
   }

   int count = 0;
   using (IEnumerator<TSource> e = source.GetEnumerator())
   {
      checked
      {
         while (e.MoveNext())
         {
            count++;
         }
      }
   }

   return count;
}

Let's not forget:

IEnumerable<T> is inherited by ICollection<T>
ICollection<T> is inherited by IList<T>
IList<T> is inherited by List<T>
IList<T> is also inherited by T[]

.Count() checks if the underlying type is an ICollection<TSource>, ICollection, or an IListProvider<TSource>; if not, it would enumerate through the IEnumerable<T> to count the items.

How about IQueryable?

IQueryable<T> inherits from IEnumerable<T>. I've seen this go bad personally. There was another team's project using Azure CosmosDB. They returned IQueryable from their repository layer and they did not materialize them properly. The result is a nightmare utilization of Request Units (RU) and a huge bill that what was it supposedly.

Got it refreshed on my memory again. Let me know if I'm wrong, point me to right path.

If you can't find why your code is taking too long to execute or your database requests are off the charts, you may want to revisit how IEnumerable<T> is used on your code.

That's it. Happy Coding!

We used CosmosDB instead of a relational database, here's what we went through and learned.

neil445 — Wed, 13 May 2020 06:37:48 +0000

Almost two years ago, we took on a pretty sizable project. We're talking about a potential of users from everywhere in the world, if all goes well, and; a brilliantly complex domain that makes a lot of sense to the aim of the product.

I was involved on designing the solution architecture, everything was agreed upon except with the database. Taking to consideration the complexity of the relationships of the entities, I proposed to use Azure SQL and find a way with Elastic Database tools to scale it horizontally. On the other side of the table, it was decided to use CosmosDB, why? mainly for these reasons:

FAST. We're guaranteed that some queries would be faster than 10ms.
CosmosDB offers an almost painless scaling. Set a partition key, write a few lines of code to handle partitioning, and there you go.
Since we're dealing with multi-tenancy, we found that CosmosDB's partitioning could help us to properly segregate data for each tenant.

I've worked on CosmosDB back when it was still called DocumentDB. I was never sold that you could totally replace a relational database with a NoSQL database, though, you can still pull it off if you do it properly or the complexity permits. The key is modelling your entities correctly, you should identify when you need to reference or embed, it's all about how volatile the data is. I was given a sandwich for my idea and told me that we would go with CosmosDB. We did the dirty work, and here's what we went through:

Stored procedures run on their own transaction. You can't write a transaction that would run multiple stored procedures and sweetly roll back all the changes if something went wrong.
The SQL engine isn't really all your familiar SQL. SQL statements are very limited, and JOIN doesn't work like the JOIN you know from relational databases. You can't do cross collection queries.
Back then, the SDK doesn't support that much LINQ statements. We had a hard time implementing pagination. Currently, more LINQ statements are supported and OFFSET LIMIT statements was introduced to the SQL engine.
Partitioning can get in the way of your queries. We had a lot of filters in our application. There were times we need to restructure the partitions to accommodate filters or totally say we can't do that filters since the partitioning forbids us.
You have to deal with error 429's. I wouldn't go deep about it, it needs another wall of text. It basically means that your collection is getting overloaded with requests and you have to handle this error and retry the request. The SDK can help you handle it automatically or just pay more for more request units.
Hot partitions. We tried the auto-scale feature for the request units (RU's). The idea is when we hit the threshold, more RU's will be automatically allocated and will go back down if there's no need. Still we're getting a lot of error 429's, we found out that we're experiencing hot partitions. The partition keys developers set are the logical partitions, which are then mapped to physical partitions, these physical partitions take up a definite amount of RU's; if the physical partition's RU's are used up, all requests for all the logical partitions mapped to it will result to error 429.
I am personally convinced that you cannot live without error 429's, unless you are filthy rich and willing to pay the price. We optimized our code, did caching, and implemented automatic scaling. We still get error 429's and the cost didn't really go down.

I was never convinced that CosmosDB is the right way to go. We're spending way too much and we can't do so much about it. We hopelessly combed through the diagnostic logs to find some light, and even bothered support to give us answers for our questions that doesn't even make sense. Up until the day I left, we're still trying to make do with CosmosDB and bled our pockets for it. Through the experience, here's what we realized:

CosmosDB really does offer painless scaling through partitioning.
It would really shine for you if you are not doing fancy things with your data. If you have data that you would store as you display it or display it as store it, you could really benefit from the speed CosmosDB promised.
If you have huge amount of data that you just need to store outright, you'll could take advantage on how easy you can scale storage through partitioning. We had other projects where CosmosDB really paid-off. We have an IoT project that used CosmosDB to save tons of telemetry. The team was able to scale for terabytes of data without breaking a sweat and worrying for storage.
You could create collection triggers and have a change feed processor if you need to something event-driven.
You couldn't totally replace a relational database with a NoSQL Database for a domain with business logic relying heavily on entity relationship.

That's all. I do believe that CosmosDB is a great product. On the right design and with proper documentation, developers could really benefit with the speed and almost painless storage scaling. Let me know if I missed something, you agree or disagree with my thoughts, discussion is well appreciated.

DISCLAIMER: This post is not meant to criticize CosmosDB negatively or convince developers to stay away from it. CosmosDB is a really good NoSQL database and gives a lot of benefits on proper implementation.

I got caught in between during this recession brought by the pandemic; I lost my job. A very humbling experience.

neil445 — Wed, 29 Apr 2020 07:32:23 +0000

I looked for greener pastures and made careful preparations before leaving my previous company, I signed the contract to a new employer and properly waived everyone goodbye. I was already out of my previous employer, taking sweet time learning React and .NET Core before my start date for my new employer. Ten days before my start date, while I was having breakfast, the recruiter sent me an email that the contract I signed was rescinded and my employment is therefore terminated before I could even start. I got caught in between.

I didn't take time to let it sink in, I immediately fired up my laptop, updated my resume and sent it to 26 job vacancies. I posted on LinkedIn and got sympathy and help. I got much appreciated messages from my previous CEO, manager, and client hoping me well and are open to help me. Three days after, I started receiving invites to interviews and I entertained where I believed I am qualified. Out of 26, I got processed up to the final interview for 3 of them, 1 immediately said that they would consider other candidates, the other two I never heard back from them.

Through the interviews, I learned a lot. The things I thought I was doing right and some practices got exposed. I was wrong on several things and I could say that I'm going outdated with the recent developments in the software development.

Here's what I've learned so far going through the experience.

I've worked on my recent project for quite a while and I unconsciously limited myself from learning other things. I've been interested on React and ReactNative for quite a while and I didn't manage to find time to actually learn it, instead I confined myself on my subpar skills in Angular and that didn't help me through the interview. My point is, make time to learn things that stray from your comfort zone and what you're working on.
Read the job description on what is offered to you. You might end up applying to a position you're not qualified. Also, you'll get an idea how other organizations do the same job you're doing. For example, I've seen a lot of DevOps engineer positions looking for people with experience on Test Automation. You might want to touch base with your previous colleagues and suggest things you've seen so they can keep themselves up to date.
Some recruiters don't know exactly what they are looking for, and it is not their fault. I've been invited for DevOps engineer, Java Developer, Node.JS Developer, and many others; despite it was clear on my resume that my experience is on C# and Angular. Be patient and understanding, and it helps to give the recruiters an idea what to look for for those positions. Have the conversation end well for both of you, at the end of the day we're throwing ourselves at the mercy of the recruiters during initial contact.
Keep your resume clean and concise. I had feedback that my one page resume is clean and inviting to take a look on. Most recruiters on initial contact would only ask about your recent experience or a gist of your experience related to the position you're applying for. The rest of your selling point will be discussed when you face technical interview, that's when you should do the talking about your experience.
When talking during the interview, don't dig your own grave. Do not rush answering questions. I've done interview myself before and there are times we follow up on things that came out of the applicant's mouth. Be honest and admit what you don't know, it's better to come clean rather than have a hard time catching up on the job if you get hired.
Sell yourself on things you know you excel, but not too much so you don't toot your horn. When talking about things you aren't good enough, follow up on what you think you can do to improve it.
Lastly, there is no harm keeping professional relationships from your previous employment. They do come handy in times of need. So don't burn bridges!

That's it. I'm still going through this unfortunate circumstance and I know there's a lot of people on the same boat. I hope we get to learn more and get through this as soon as possible. I wish everyone good health and safety during these trying times.