We started to use Airtable at the early stages as I described in my previous article. Today, you will be reading how we tackled Airtable's API limits while the company grows to 10.000 car subscribers since 2019.
(Andreas Wixler (CTO) initiated the Slack channel for Airtable Action Squad)
After we hit the limit, we were no longer able to provide a reliable service to our customers. Therefore, we initiated an immediate action squad functioning cross-departments with the lead of Christian (VP Engineering - B2B), Tome (Tech Lead - Customer Product), and Alexander (VP Engineering - Data & Finance). I was representing the user acquisition team. Initially, we planned a sprint for two weeks, created our Kanban board in Jira, identified issues, and scheduled regular meetings for status updates on the tasks.
Before tackling the API limits, we created an Airtable migration plan and focused on certain topics for investigation earlier before we hit the issues. So, it's quite important to be ready when you hit the limit. But, we were not 100% ready when the issue occurred, it took a few weeks to solve the problems. Still, the performance we gained by the speed of moving fast while we grow was pushed by Airtable.
These were the core points we focused on earlier when we already decided on NocoDB as an alternative for Airtable:
- One database-instance per department (PostgreSQL) - AWS Aurora PostgreSQL
- Hourly backups
- Unlimited databases per department
- Connected with Google Workspace SSO
- Role-based control is department responsibility (VP Engineering of the Department, Department Head)
- Integromat - NocoDB Module - Customer Product write Integromat Module Retool - Connecting via API (recommend)
So we were getting ready but not as fast as we were growing. Customers just loved us. :)
Analysis of failure points
First, we sat down and looked for every scenario in Integromat and sometimes analyzed the API calls from the logs to have some metrics in front of us. We have a Slack bot that alerts us whenever there is an issue in Airtable modules (API limits). Every person checked their scenarios which they own. Some people came up with alternative ideas for better performance. We took the notes down.
These were the main issues we realized in our flows within all departments:
- Airtable's every base has a request limit of 5 per second.
- Airtable gives a warning at the top of a base because we reached and crossed total row limits. Sad. :/
- We almost collected everything in one base. This is because the company started with a simple base and then grew fast while the number of employees and orders increased quickly. We realized this issue earlier and created the new projects on different bases already.
- We were using tons of modules in Integromat running in parallel (we have more than 15 workers in our Integromat). Also, due to the pagination requests in some modules because of data being too big, we were increasing the risk of hitting timeout errors.
- Our checkout process was using services that were talking to Airtable directly and synchronously. If there was a limit/timeout issue, the customer couldn't proceed further.
- Airtable is calculating every formula, within a base, when there is an edit, creation, update, or deletion of data. This causes a delay and timeout issue from time to time. Sometimes the response is timed out, but the action is successful. This can also deceive our Integromat modules.
Work we have done to fix the issue(s)
Most of our Integromat scenarios are using Airtable for read-only operations. When Airtable introduced their Airtable Sync, we immediately increased the number of bases by taking away some responsibilities (tables, records) from the main base into different bases. From these bases, we created synced tables from the main base and used these bases as the reference point so we could gain their API request limit for read-only operations. Each department created its own small bases for read-only operations and updated its Integromat scenarios.
While doing this, we also started to use Sync Inc to have a copy of our Airtable base synced in PostgreSQL. Initially, we set the limit of writing at Sync to 1, and gradually we increased this limit (max 5) while monitoring our Airtable performance and data quality. This way we could connect this solution to our own NocoDB instance. After our beloved engineer Ishtiaque (Senior Engineer - Operations) created an Integromat module for NocoDB, we started reading from our own PostgreSQL instance UNLIMITED and we initiated the migration to the NocoDB this way. We first started with Sync solution, then switched to NocoDB.
We had to update our NocoDB modules. We used Integromat API to search for certain Airtable modules and then replaced them with NocoDB modules. Dina (Software Engineer - Operations) and Vijay (our NocoDB guru) contributed heavily to this. <3
I moved our calls to the Airtable from our checkout process. We created an Integromat module that would book the car for the customer asynchronously while the customer could proceed with their process - no waiting.
We analyzed company-wide each column in each table to detect if they are being used or not. It was a huge company effort because dependencies were too many, all spread over different tools. We removed all unneeded columns successfully. After that, we also optimized the formulas that we use and tried to decrease dependencies in formulas and between columns.
Thanks, everyone involved!
Highlights and achievements
- More than halved the numbers of records in VSM Halved the numbers of tables in VSM
- Reduced massively the # numbers of Airtable timeouts per day
- Switch to read from NocoDB directly where ever possible and write to Airtable
- Proof of concept migration of car sales to NocoDB
- Last but not least, worked for the first time all together cross-department, with a great time spirit and achieving the goals set at the beginning
Thanks, Emanuele (Senior Product Manager - Operations) for managing our sprints!
While we move away from Airtable with our core data, we will be still using it for some time. NocoDB is still in the early stages. We built a dedicated team to work on this open-source project to fit our needs by the time. At FINN, we will be always using No Code - whether Airtable or not.
I will write about how we are using NocoDB at FINN in one of my next articles.
Stay tuned and don't forget to like/follow! :)
Top comments (4)
Great post. Couple of quick questions:
What is the rational to always use a no code database? Is it your internal users or is there any other factor?
I see you mentioned API limits as one of the reason for the migration. Did you guys plan ahead of time that when you hit the volume of traffic, you will migrate or is it more of a reaction to the problem?
Thanks
1) FINN is a cross functional company within departments. We share the common data: fleet, customers, invoices, orders, etc. We need an interface where we can filter out the data we need and change it quickly when needed without needing any extra developer time.
2) There is a limit. For example after 100K rows with lots of formulas, you are basically done. Well, we could keep using airtable for a longer time than expected if they didn’t have an api limit. But there will be a point where we will always reach the limit. Therefore we are getting ready for now by preparing the structure.
One final question: Did you have any other candidates before settling down with NoCoDB?
Not really.