It's been a few months since my last post, and am excited to begin posting again!
Following my completion of EffectiveDonate, a nonprofit discovery website I developed for my capstone project, it was time to publicly deploy it. I've had experience deploying other projects I built at the Flatiron School to Heroku (a cloud platform service), so I decided to go with Heroku again for EffectiveDonate. However there were a few additional features of Heroku I needed to use for the first time since I was working with API data that I needed to refresh regularly, requiring me to schedule a job to run daily. Let me tell you how I did it!
According to Heroku, add-ons are components that support your application, such as data storage, monitoring, analytics, data processing, and more. Add-ons are managed either by third-parties or by Heroku itself. The purpose of add-ons is to allow developers to focus on the maintenance of their own code and to outsource the services supporting production to the add-ons.
You can install add-ons either through the CLI or the Heroku Dashboard.
My EffectiveDonate site uses project data from the GlobalGiving API, and I have a method on my Rails backend to fetch new projects. However, these projects are dynamic - they are active only until their funding goal hasn't been reached, and then they stop taking donations. New projects are also started daily. Additionally, project funding amounts are constantly changing. As a result, I need to fetch project data daily regularly from the API to prevent stale data from reaching my application.
I researched several ways of automating query jobs, both with
npm packages and Ruby
gems, but I finally came across Heroku Scheduler, which turned out to be much more straightforward than those other methods.
Heroku Scheduler is a free add-on that allows you to automatically run jobs at scheduled intervals (daily, hourly, etc).
To install Heroku Scheduler, simply run
heroku addons:create scheduler:standard
In Rails, the convention is to create a
rake task file in the
Below are the tasks that I have written in my
desc "This task is called by the Heroku scheduler add-on" task :fetch_projects => :environment do puts "Deleting Tables..." Project.delete_all UserStarredProject.delete_all puts "done." puts "Fetching Projects..." Project.fetch(nextProject: false) puts "done." end task :fetch_countries => :environment do puts "Deleting Countries..." Country.delete_all puts "Fetching Countries... " Country.create_all puts "done." end
As you can see, the
fetch_projects task is the one that I will be adding to Heroku Scheduler to run regularly. It completes the actions of deleting the existing
Projects and fetches new
Projects from the GlobalGiving API.
After making sure the task is working locally and pushing it to Heroku, I can test that it is also functioning correctly on Heroku by running
heroku run rake fetch_projects. If it's working right then it will work on Heroku Scheduler!
To get your rake task up on Heroku Scheduler, either open Scheduler in your app dashboard or the following command:
heroku addons:open scheduler.
On the Scheduler dashboard, you can add and edit jobs. Jobs can be scheduled to run daily, hourly, or every 10 minutes. Next, you enter the task that the scheduler should run. In my case it was
rake fetch_projects. Then just hit "Save Job", and it will start running when the next scheduled time is reached!
After I scheduled my
fetch_projects job, everything seemed to be going fine and the databases began to populate. However, I checked my email, and saw a message with the shocking subject: "[warning] Database disruption imminent, row limit exceeded for hobby-dev database on Heroku app". That didn't sound good! I did some digging, and sure enough, I had over 45,000 rows in my app's Postgres database, while the limit for Heroku's Hobby-dev plan is 10,000 rows. I could either 1) upgrade to a higher database plan or 2) reduce the number of rows in my database.
Since I still need to build some more features in EffectiveDonate before I can start sharing it broadly, I decided to go with option 2 for now, and eventually I can always upgrade my database plan. This required a slight refactor in my database structure to avoid a join table that was adding multiple rows for each of the thousands of
Project objects that I was creating.
Another useful tip I learned to prevent a scheduled job from continuing to run was a special CLI command. You can view all processes that are currently running on your Heroku app with
heroku ps. If your scheduled task is running, you will be able to see its associated process id. Then you can stop it using
heroku ps:stop <process id from heroku ps>, for example
heroku ps:stop run.8359. This is an invaluable tool to use if you see that your database is fast approaching the 10,000 row limit, then you can stop the rake task's runaway train!
As you can see, the Heroku Scheduler add-on is an extremely powerful tool that can automate API fetches, or any other task that needs to be run regularly in your app. I love the Scheduler's easy-to-use interface and how nicely it plays with Rails' rake tasks. Now that I have my row count consistently under 10,000, my scheduled job has been running smoothly and there haven't been any more alarming emails reaching my inbox :)