DEV Community: Kven Ho

Dev Video Review: Firestore Data Structure, Limitations, and IMHO

Kven Ho — Tue, 31 Dec 2024 08:09:29 +0000

It’s been a long time since I last used Firestore (Before Firebase acquired by Google), as my work has mostly focused on server-side development. Traditional databases often make more sense in terms of cost and flexibility. However, while working on a recent React Native project, I had the chance to revisit Firestore and refresh my knowledge. Inspired by an official Firestore video, I decided to share some of the rules outlined in the video along with my opinions on Firestore's data structure.

Rule #1: Document have limit

Understanding database limitations is crucial when modeling your data structure. Firestore, like all database systems, has certain limits to prevent exhausting the compute and memory resources. Below are some key limitations you should consider:

Max 1MB limit per document
Technically, Firestore allows any data types, but just because you can store an image inside a document doesn’t mean you should. 1MB document should be enough for 99.99% of your use cases.
Max 20k fields limit in document
When you modeling the data structure, you have to prevent keep adding new fields in the document.
Max depth of sub-collections is 100
Avoid excessively nesting sub-collections in your data structure.

For more details, please check this link: https://firebase.google.com/docs/firestore/quotas#collections_documents_and_fields

Rule #2: You can't retrieve a partial document

Firestore doesn’t allow partial document retrieval. If you need to check a specific field, Firestore will return the entire document, and you’ll need to extract the field yourself.

Fortunately, Firestore charges based on document reads, not size or bandwidth (Talk more in Rule #4 later). However, took this consideration when modeling your data structure, especially when query come from mobile apps, as it may increase battery and data usage.

Rule #3: Queries are shallow

When you query a document, Firestore retrieves the document but not its sub-collections. Keep this in mind while modeling your data structure to avoid unexpected behavior.

Rule #4: You're billed by the number of reads and write your platform

Firestore is charges based on the number of reads and writes rather than the storage size. Optimize your data structure to minimize reads and writes, but don’t over-complicate your code for just saving a few dollars per month, totally not worth for the engineering cost.

Rule #5: Queries find documents in a single collection

Firestore queries can only retrieve documents from a single collection. This limitation means no joins (or groups) are possible, so you’ll need to plan your data modeling accordingly.

Rule #6: Arrays are weird... maybe not now

Firestore is supports arrays, and while there were limitations in the past. However, improvements have make them more reliable for now. You can now use special array operators for manipulation the array. Learn more here: https://firebase.blog/posts/2018/08/better-arrays-in-cloud-firestore

Cloudflare D1 and Prisma: Not a Good Combination (For Now)

Kven Ho — Sat, 07 Dec 2024 03:45:54 +0000

Recently, I started migrating my monolith project to a Cloudflare Worker. At the same time, I decided to give Cloudflare D1 a try. Since my project already uses Prisma to handle a PostgreSQL database, I thought the migration would be almost painless to D1. (How naive I was in the beginning!)

First Issue: No Support for ENUM Data Types

The first roadblock I encountered was that Cloudflare D1, as a SQLite-like database, doesn’t support the ENUM data type. I uses ENUM almost in every table in my project, and Prisma makes it incredibly straightforward to implement and manage ENUM datatype.

To work around this limitation, I had to create separate tables to store the ENUM data manually. Technically, PostgreSQL also creates a sort of "table" behind the scenes when you define an ENUM datatype, but here, I had to do it myself. It wasn’t ideal, but I managed to solve this issue.

Then, Prisma told me, Cloudflare D1 does not support interactive transactions. Prisma have two type of methods to handle database transaction, interactive transactions and sequential operations. I use interactive transactions in every part of my code because it give me control on the flow. I getting this error when I migrated my code to worker and want to create a new user, then get the user ID to create user profile. This is very typical data design and storing. At this point, I still thinking maybe I can generate the UUID as the user ID, before create user and user profile, problem solved.

Second Issue: No Interactive Transactions

I found out that Cloudflare D1 doesn’t support interactive transactions. Prisma offers two ways to handle transactions: interactive transactions and sequential operations. In my code, I use interactive transactions extensively because they give me control over the flow of operations.

I found this issue, when I migrated my code to the worker and tried to create a new user, I needed the user ID immediately to create a corresponding user profile. This is a common and straightforward use case for database design. However, without interactive transactions, we don't know the user ID after the user creation, and can't create the user profile without user ID.

Although I started to feel annoyed at that moment, I still managed to work around this issue by creating the user ID beforehand. That seemed like a reasonable fix, and it worked to some in extent.

The Final Boss: No ACID Compliance

No ACID compliance is a no-go for my use case. While testing further, I encountered this warning from Prisma:

prisma:warn Cloudflare D1 does not support transactions yet. When using Prisma's D1 adapter, implicit & explicit transactions will be ignored and run as individual queries, which breaks the guarantees of the ACID properties of transactions. For more details see https://pris.ly/d/d1-transactions

Transactions are crucial for ensuring data integrity, especially in scenarios where multiple operations need to be treated as a single transaction.

Conclusion

For now, I've decided to stick with my existing PostgreSQL setup and save myself the headache. Today I learned that if something is working, it's better not to change it.

4 Essential Tips for RESTful API Design

Kven Ho — Thu, 10 Oct 2024 05:15:39 +0000

Designing a clean and efficient RESTful API is key to ensuring that developers can interact with your service smoothly. A well-structured API not only saves time but also makes it easier to scale and maintain. Here are four tips to help you enhance your RESTful API design.

1. Avoid Actions in the URI

Instead of using action words in the URI, rely on HTTP methods like GET, POST, PUT, and DELETE to represent the action.

Why? This promotes consistency across different endpoints and allows developers to make educated assumptions about the API's behavior. Example: GET /users/{user_id} for fetching data, where GET is used for reading data only, and no modification occurs.

By following this rule, your API becomes more predictable and easier to work with, which speeds up the development process.

2. Limit to Two Resources in the URI

While it's perfectly possible to access a resource nested within another, try to avoid over-complicating your URIs by including too many resources in a single URI. Overly long URIs with multiple nested resources can make the API harder to maintain and evolve.

# Good Practice: Fetch tasks assigned to a user.
GET /users/{user_id}/tasks

# Avoid:
GET /users/{user_id}/tasks/{task_id}/comments

This makes the URI harder to maintain and change. Instead of adding more complexity, split the API:

# Better Solution:
GET /users/{user_id}/tasks
GET /tasks/{task_id}/comments

This keeps your API simple and easier to manage in the long run.

3. Use Query Parameters for Additional Options

For metadata like pagination, sorting, or limiting results, use query parameters to maintain clean URIs.

# Example: 
GET /users/{user_id}/tasks?limit=5&sort=created:DESC&page=2

This approach keeps the URI clean while still allowing flexible data retrieval options.

4. Always Wrap the Data in a Second-Level Object

When returning data, wrap it in an outer object, especially if you plan to include additional metadata in the response.

Avoid:
[ { "id": 1, "task": "Task 1" }, { "id": 2, "task": "Task 2" } ]

# Good Practice:
{
  "data": [ { "id": 1, "task": "Task 1" }, { "id": 2, "task": "Task 2" } ],
  "pagination": { "page": 1, "limit": 5 },
  "sort": "created:DESC"
}

This structure not only makes the response more organized but also allows you to include useful information such as pagination details, sort options, and more.

By following these simple yet effective tips, you can design a RESTful API that is easier to work with, scalable, and maintainable. Happy building!

How to Set Up CrewAI on macOS: A Step-by-Step Guide

Kven Ho — Mon, 07 Oct 2024 02:45:42 +0000

Setting up CrewAI on macOS requires careful configuration of Python, as the tool depends on specific versions. This guide will walk you through the process using pyenv, a powerful tool that allows you to manage multiple versions of Python on your system. Let’s dive in.

1. Install and Configure pyenv

To begin, you’ll need to install pyenv, which simplifies managing different Python versions. However, it’s crucial to use a specific command to avoid missing modules later on:

env PYTHON_CONFIGURE_OPTS="--enable-framework" pyenv install 3.12.7

This command ensures that the necessary framework support is built into Python, which is essential when running Python-based applications like CrewAI on macOS. CrewAI requires Python versions between 3.10 and 3.13 (during my writing time), so Python 3.12.7 is a safe choice.

For detailed steps on how to install Python with framework support on macOS, you can refer to the pyenv wiki.

2. Set up pyenv for Your Shell

Once you’ve installed the correct version of Python, you need to initialize pyenv so that your shell can use it. Run the following command to get the initialization script:

pyenv init

Next, copy the output and paste it into your .zprofile or .zshrc file, depending on which shell you’re using. This ensures that pyenv is automatically loaded every time you start a new terminal session. After editing the file, restart your terminal to apply the changes.

3. Set Python Version for the Shell

Now that pyenv is set up, you need to specify which Python version to use for your current shell session. To set Python 3.12.7, use the following command:

pyenv shell 3.12.7

Then, restart your terminal once more to ensure the change takes effect.

4. Install CrewAI and Necessary Tools

With the correct version of Python active, you can now install CrewAI and its required tools using pip. To do this, run:

pip install 'crewai[tools]'

Once the installation is complete, restart the terminal again to ensure everything is correctly set up.

5. Create Your First CrewAI Project

You’re now ready to create your first CrewAI project. Use the following command to initialize a new project:

crewai create crew <project_name>

This will set up a new AI-powered project that you can start building on. By following these steps, you’ll have CrewAI running smoothly on your macOS environment, ready to assist with AI-driven workflows and applications.

Conclusion

Setting up CrewAI on macOS requires a few specific steps, particularly in configuring the right Python environment. By using pyenv, you can easily manage Python versions and ensure compatibility with CrewAI. Remember to follow each step carefully, especially regarding terminal restarts, to avoid any issues along the way. With the right setup, CrewAI will be a powerful addition to your development toolkit!