At 02:15 the nightly job starts, and by 02:40 the broker service takes six seconds to answer. Nobody is paged and no static threshold fires, yet customers complain in the morning. That is the real problem behind batch job monitoring in Dynatrace: a batch is neither a service nor a host, it simply eats CPU, memory and I/O at the worst moment. This article proposes a way to decide what to alert on, what to leave to Davis and what to automate. The vendor describes a similar scenario in its two-part post; I start from the operational decision, not from its outline.
Method: I re-read the post and the cited documentation pages on 11 November 2026, but I did not replay this scenario on a live environment. Anything I could not confirm is marked « not verified ». The reasoning applies to Dynatrace SaaS; I did not check a Managed equivalent, so I promise none.
Why batch job monitoring in Dynatrace defeats classic alerts
A batch raises three difficulties. It is short-lived: the pod exists for an hour, then vanishes. It is periodic: its load is normal at 02:15 and abnormal at 14:00. Above all it is a cause, rarely the victim: the neighbouring service suffers. An alert on the batch’s own CPU therefore says little; you must tie the batch to the symptoms of the service it hurts.
The first operational question is what you want to know. Three answers are enough. Did the batch fail? Did it run unusually long? Did it degrade something else? The first two call for a periodic check, the third for Davis. The vendor’s post illustrates the first part with a workflow that reads job status every 15 minutes; it gives no threshold for « too long », so you must set it yourself.
Give Davis context with a start and an end event
Davis sees metrics and traces, not your job calendar. To let it connect a slowdown to a batch, announce the batch with an event. The event ingestion API (POST /api/v2/events/ingest) only requires eventType and title; the entitySelector attaches the event to the relevant service, otherwise it sticks to the environment. The API token needs the events.ingest scope.
Three details matter in this payload. The CUSTOM_INFO type is informational: it does not open a problem by itself. The timeout (in minutes, 15 by default, 360 at most) bounds the event’s lifetime; set it to the usual batch duration. Finally, non-dt.* properties are free, up to 100 entries: put the job name, run identifier, workload and namespace there. Send a second event at the end with the same identifier so you can measure durations.
Let Davis tie the slowdown to the batch
Once the event is in place, detection is no longer your job. In the vendor’s scenario, Davis spots the rise in latency and errors on the service before any ticket exists, and blames a batch that saturated CPU, memory and I/O next to it. My operational takeaway: do not set thresholds on the batch, tune detection on the business service and let the event supply the explanation. That cuts noise, a topic already covered in our article on Davis alert noise.
One limit: the event enriches context, it does not guarantee Davis will name the batch as root cause. Confirm this on a real incident or a load test before building automation on it.
Decide what a workflow may do on its own
This is the central decision, and it comes before any configuration. A workflow can start on a Davis problem, open a ticket, test a condition, then act. Use a simple grid:
- Harmless: open or comment a ticket, notify a channel. Automate straight away; see the Jira and ServiceNow workflow.
- Reversible: suspend a CronJob, postpone it. Automate if a condition checks the job is not critical to the accounting close.
- Irreversible: delete a running job, restart a database. Keep that for a human.
For a batch, suspending a CronJob has one advantage: it only affects the next schedule. A job already started carries on, so stopping the current run is a second gesture that you must decide separately. Two safeguards exist in the manifest before Dynatrace is involved: concurrencyPolicy: Forbid prevents overlaps and activeDeadlineSeconds caps the duration.
Reach the private cluster without opening a port
A workflow runs on the Dynatrace side; the Kubernetes API of a private cluster must not be exposed. EdgeConnect solves this: it opens an outbound WebSocket connection to Dynatrace AppEngine and needs no inbound connection. HTTP requests whose host matches a configured pattern are then executed inside your network. According to the documentation, that pattern is declared under Settings > General > External Requests > EdgeConnect, not in the YAML file.
The service account used against the Kubernetes API should have the bare minimum: the right to patch CronJobs in a single namespace. I did not read the page describing the exact action that calls the Kubernetes API from a workflow: that detail is not verified, so check it in the workflow editor.
How to check that it works
- Send the start event by hand against a test service and check it appears on its timeline. The 201 response lists a
statusper event: look forOK, notINVALID_ENTITY_TYPE. - Generate real load on a pre-production environment, not in production.
- Confirm the Davis problem names the batch and the workflow starts.
- Check that the CronJob
suspendfield flips totrue, and that the ticket is commented rather than duplicated. - Replay in reverse: a problem unrelated to the batch must trigger no suspension.
Best practices
- One unique run identifier, present in the start event, the end event and the ticket.
- A « dry run » mode: for the first two weeks the workflow writes in the ticket what it would do.
- An explicit exemption for closing batches, never suspended automatically.
- Workflows permissions (
automation:workflows:readandadmin) limited to the platform team; to get started, see the Workflows module of the SaaS course. - The EdgeConnect OAuth secret kept in a vault, never in the repository.
Frequently asked questions
Do I need an event for every batch?
Yes if you want Davis to be able to explain it. Without an event, the slowdown stays visible but its cause is not tied to the scheduled job.
Can Dynatrace stop a job by itself?
It can trigger a workflow that calls the Kubernetes API through EdgeConnect, provided you built and authorised it. Nothing stops without that chain.
Does this work on Managed?
The pages I read describe Workflows and EdgeConnect on the SaaS side. I did not verify a Managed equivalent, so treat the scenario as SaaS only.
Conclusion
Announce your batches with an event, watch the business service rather than the job, then automate only reversible gestures. Start with the dry run before letting a workflow suspend anything in production.
span.observer is an independent community blog, not affiliated with Dynatrace LLC. Brands mentioned belong to their owners. Tools change quickly: check commands and options in the official documentation.
Originally published on span.observer. French version: https://span.observer/fr/traitements-batch-dynatrace-davis-workflows/.



Top comments (0)