Weird things happen in prod
I don't care how much test coverage you have, or how many database transactions you use, or how many years you have been coding: weird things happen in prod.
Scripts that perform sanity checks on your databases will catch these oddities before they become a major issue. Or at least, that's the idea, and it has been true for the past two decades for me.
Any time you introduce business rules that can't (or not easily) be enforced directly by the database, add a basic check that scans for records that are not following those rules.
Example 1: Basic attribute values
Rule: When state == 'paid' then paid_at should not be null.
Simply write a query that selects all records where state == 'paid' and paid_at is null and send them to your alerting system.
Example 2: X-Tenancy Mismatch
Object X belongs_to Object Y and both should have an Owner Z
Maybe this check alerts PagerDuty since it could be disastrous.
Other Examples:
- All users belong to at least 1 organization
- All organizations have at least 1 user
- Users set to super admin only belong to Org Foo (maybe this one runs every 5min)
Additional Tips:
- Feel free to limit checks to records that have been created or updated in the past 48 hours
- Ignore archived records if applicable
- Run the checks manually before deploying to prevent a flood of errors for bad sanity check logic. Sometimes I forget about an edge case that is actually valid.
Top comments (0)