arnold infant

Posted on Aug 30

OCI Monitoring Alarms: Six Ways an Alarm Can Look Ready and Still Fail

#oracle #oci #cloud #monitoring

A practical review of metric timing, query scope, notification delivery, suppression, and operator response

While reviewing Oracle's alarm documentation for this article, one point kept coming back: the alarm definition is only one part of the operating path.

A valid alarm can still leave a gap. The query may use a one-minute interval for a metric that arrives hourly. A dimension may exclude the resource that matters. The notification topic may exist while a subscription remains unconfirmed. Even a well-written absence query can behave differently from what the support team expects.

These are six areas I would ask the technical owner to walk through before the team depends on an OCI Monitoring alarm.

1. Start with the metric's emission frequency

The interval in an alarm query is a time window for assessing metric data. It does not control how often the source service emits that data.

Oracle's troubleshooting guide gives a useful example. The StoredBytes metric in the oci_object_storage namespace is emitted once an hour, but the example alarm uses this query:

StoredBytes[1m].sum() > 800000000

The one-minute interval is shorter than the hourly emission frequency. In Oracle's example, the alarm fires at 3:00 and clears at 3:01, even though the last data point was still above the threshold. Changing the interval to at least one hour keeps the alarm firing until the next hourly data point is assessed.

That example is a good reason to begin the review with the source metric documentation. Note the namespace, metric name, emission frequency, query interval, and statistic. Then compare the expected FIRING and OK timing with the alarm history or an approved test.

Before moving on, ask the owner: can the source metric supply data at the pace the alarm query expects?

2. Read every dimension as a scope decision

Dimensions make a query precise, but precision is useful only when the selected scope is correct.

Oracle documents a case where an alarm is filtered to one availability domain while the resource meeting the alarm condition is in another. The expression is valid; the relevant resource has simply been filtered out.

For that reason, I would read the metric chart and the MQL together. The resource shown on the chart must be the resource the team intends to monitor. The compartment, metric namespace, resource group, and each dimension name and value also need to match that intent.

This check belongs before threshold tuning. There is little value in refining a threshold until the team knows which metric streams the query includes.

3. Absence alarms have their own timing

An absence alarm is useful when missing data is itself the condition. In OCI Monitoring, absent() returns 1 when the metric is absent for the entire interval and 0 when it is present.

The detail that deserves attention is what happens next. After continuous true values for the configured absence detection period, absent() stops returning values. The default absence detection period is two hours. Oracle allows a custom value from one minute (1m) to three days (3d or 72h).

Oracle also recommends groupBy in absence-alarm queries. A new OCI dimension can create an initially empty metric stream. Without grouping, that empty stream can cause an irrelevant trigger even while other streams are present.

Oracle's documented pattern for a compute instance is:

CpuUtilization[1m]{resourceId = "<resource_identifier>"}.groupBy(resourceId).absent()

Before using an absence alarm, record three decisions: why missing data represents a problem, which metric streams are grouped, and how long the absence must continue. The alarm history should also be read with the absence detection period and OCI's internal reset behavior in mind; an absence alarm can later return to OK when it no longer tracks the missing stream.

Where testing is permitted, use a controlled test in a safe lab or approved tenancy. Do not create a Production incident merely to prove an alarm.

4. Follow the message beyond the alarm

Selecting a Notifications topic is not the end of the delivery path. OCI publishes the alarm message to the topic, and the Notifications service sends it to the topic's subscriptions.

Some subscription protocols require confirmation. Until that happens, the subscription remains in PENDING status; after confirmation it becomes ACTIVE. Oracle also states that the confirmation URL is valid for three days.

For a readiness review, I would trace one message across the whole route:

the alarm points to the intended destination;
the topic contains the expected subscription;
any required confirmation is complete;
the intended recipient receives the message; and
the owner and escalation route are known.

The alarm body matters as well. Oracle recommends giving operators guidance and, where useful, a link to the relevant runbook. A message such as “high CPU” tells the recipient what happened. It still leaves unanswered who owns the response and what the first action is.

A controlled notification test, when allowed, is stronger than a screenshot of the topic configuration because it checks the route the message must actually travel.

5. Treat suppression as a maintenance control

Suppression needs an owner, a time window, and a reason. Otherwise, a planned maintenance step can leave an unexpected blind spot.

There is a wording difference in Oracle's current documentation that is worth noticing. The Create Alarm page describes the option as suppressing “evaluations and notifications.” The dedicated suppression pages describe it as temporarily stopping notifications. Because those descriptions are not identical, the safe review is to inspect the target alarm's behavior and its history after maintenance instead of assuming suppression affects notification delivery only.

The documented setup also has several limits:

the start and end times must each be within 90 days of the current time;
a dimension-specific suppression is available only for an alarm configured for split messages;
a dimension-specific suppression cannot be added to multiple alarms at the same time; and
the Console can apply suppressions to several selected alarms, but that is not a dimension-specific suppression.

After the maintenance window, check that the suppression has ended and review the alarm's state and history. Keep the change or work reference with the suppression decision so the next reviewer can see why it existed.

6. Choose trigger delay for the response you need

Trigger delay is the number of minutes for which the condition must be maintained before the alarm enters the FIRING state.

A longer delay can filter short-lived breaches, but it also postpones the alert. Oracle's troubleshooting guidance shows the other side: an alarm with a ten-minute trigger delay does not fire when a true evaluation at 1:30 becomes false at 1:32. The condition did not remain true for the full delay.

This setting needs a reason that the support owner understands. Compare the delay with the metric's emission frequency, expected latency, and the response time required for the condition. Then use the alarm history or an approved test to see whether the timing matches that decision.

What I would keep with the alarm review

The review record does not need to be large. I would keep:

the metric namespace, metric name, and emission frequency;
the MQL query and the intended resource scope;
the reason for the threshold or absence period and trigger delay;
the notification topic, subscription status, recipient, and escalation owner; and
any suppression decision, followed by the post-maintenance result.

Most gaps in this list come from small mismatches: timing, scope, delivery, or ownership. They become important only when the team needs the alarm to work.

The useful review follows the full path from metric emission to operator response. If one part of that path has not been checked, record it before the alarm is used for Production support.

DEV Community