If you've ever administered or developed in a Salesforce org, you might have run into errors related to limits, something like Apex CPU time limit exceeded or REQUEST_LIMIT_EXCEEDED.
Since Salesforce runs on a shared, multi-tenant architecture, it enforces limits on things like Apex CPU time and the total number of REST, SOAP, or Bulk API calls an organization can make over a rolling 24-hour period.
The consequence of hitting these limits? Failed integrations, broken automation, and overall disruption in business processes. Not a great time for the platform on-call engineer.
How I used to do it
In the past, I built a custom application to monitor these limits in a Salesforce org and alert my team when usage crossed a threshold we configured in the backend. The goal was to proactively catch a problem before it became an outage.
We also integrated the application with ServiceNow so that an incident ticket was created with the right priority or severity level, and that would kick off another integration with PagerDuty to page our on-call engineer for the week.
This worked, but we had all these tools on different platforms: our monitoring was on one app, the incident creation and management was on ServiceNow, and our on-call engineer had to manually handle communication updates on both ServiceNow and Slack every couple of hours until the incident was resolved.
Recently, I found that Datadog covers most of that workflow.
- Its Salesforce integration collects limit metrics from your org so you can monitor usage and alert appropriate teams when it crosses a threshold.
- Its Incident Management declares incidents, tracks timeline events, and handles communication and documentation.
I wanted to see if I could connect the two to recreate my old setup on just one platform.
Let's try it
I began by signing up for a Developer Edition Salesforce org and a trial Datadog account.
First, by using the integration guide I created a new External Client App in my Salesforce org. By enabling it for OAuth I got my Consumer Key and Consumer Secret keys to set up the connection. On Datadog, I added my org in the Salesforce Integration tile and set a custom tag to attach to the events that come through.
Note: For the purpose of this demo I did not include platform events since Event Monitoring can't be enabled in Salesforce Developer Edition orgs.
Once the connection was created, it took a few minutes to see the limit metrics come through in the Metrics Summary tab: salesforce.limits.max (the maximum limit for this resource) and salesforce.limits.remaining (the remaining limit for this resource).
Both metrics report every Salesforce limit, so I filtered on the name:dailyapirequests tag to isolate the daily API limit.
Next, I created a new Monitor with a Threshold Alert detection type to trigger when a metric crosses a threshold value. I defined the metric using both salesforce.limits.remaining and salesforce.limits.max. Since Datadog reports remaining calls, not usage like my custom app had used, I set the alert to go off at 20% remaining.
So this was the full monitor query:
avg(last_15m):(avg:salesforce.limits.remaining{name:dailyapirequests} / avg:salesforce.limits.max{name:dailyapirequests}) * 100 <= 20
I also configured notifications and automations so the alert sent a custom message and triggered incident creation when the threshold was met.
Beforehand, I created an incident automation and set the Severity Level to SEV-2 (I set limit alerts to Priority 2 in my custom app) and assigned an incident commander. This would ideally be the on-call engineer for the week.
Now, for the simulation. I ran a custom script to call the Salesforce REST API on a loop so the remaining API calls gradually decrease over time, mocking a runaway integration with excessive requests.
At about 1,260 calls/hour, I left the script running overnight and the next morning the alert fired right at 8:28 AM, bright and early!
The Results
Taking a look at the monitor status, I saw that it triggered where I expected: 19.8% remaining, just below the 20% threshold. And at the same time, a new incident was created automatically.
The new incident had the information I was expecting: SEV-2 severity level, incident commander assigned, and a timeline generated from the events that happened.
I stopped the simulation script, filled out some details on the incident overview (what happened, impacts, root cause) to address it, and resolved it.
There's a Generate Postmortem option on the Post-Incident tab that I found pretty useful. Based on the template you choose, it automatically pulls the incident details and the timeline to generate a document. Normally, I would've created that by hand in a Google Doc or slide deck to send out to stakeholders.
Takeaways
It was nostalgic recreating a tool that I used to rely on every day. Of course, the Developer Edition Salesforce org in this demo had some limitations like not being able to set up Event Monitoring to allow access to real-time events and log files. I can imagine with it enabled I'd be able to access per-call event logs which can tell me which integration was draining the limit.
One feature I didn't use in this demo is Status Pages, which would've provided a shareable web page so that stakeholders have one place to follow updates. This addresses the pain point the on-call engineer had when it came to updating multiple channels of communication.
Another feature I'd like to try next is creating a forecast monitor so that Datadog can warn me that my org's usage is trending toward the limit before it even gets near the threshold.
Overall, I enjoyed not having to switch between configuring my threshold in a separate application and managing the incident and communication across others. Having the incident timeline and postmortem automatically generated for me means I can spend more time fixing the problem instead of documenting it. As an engineer who's been on call, I can see this making my next 2 AM page a little easier.
Disclaimer: The views in this post are my own and don’t reflect those of my current or past employers. This demo was built with a personal Salesforce Developer Edition org, a Datadog trial account, and publicly available documentation.
Top comments (0)