DEV Community

OneLifeVibe
OneLifeVibe

Posted on

Your Health Check Is Testing the Wrong Path: Probing Private Cross-Cloud Connectivity Honestly

Most cross-cloud outages are not discovered by users first. They are discovered late, because the health check kept reporting green while the real private path was already broken.

That happens for a simple reason. The probe is not travelling the path the application travels. It checks a public endpoint from the internet, or it checks a service from inside the same cloud, and then everyone assumes the link between clouds is fine. When the private route, the DNS view, or the firewall rule in the middle fails, the dashboard stays green and the application fails anyway.

Honest probing is not about adding more checks. It is about placing the checks where the traffic actually goes, and making each check fail for one clear reason at a time.

The three paths that get confused

A typical multi-cloud service has at least three different paths that can each be healthy on their own:

  1. The public path. A load balancer or public endpoint that anyone on the internet can reach.
  2. The intra-cloud path. A service calling another service inside the same cloud and the same region, often over a short, forgiving route.
  3. The private cross-cloud path. The route the work actually depends on: a private interconnect or VPN, a shared services network, a private DNS zone, and a set of firewall or security group rules that only apply to traffic coming from the other cloud.

A check against path 1 or path 2 tells you almost nothing about path 3. Yet in many estates, path 3 is the only one without a dedicated probe, because it is harder to place. The probe has to live in the source cloud, resolve the private name as the application resolves it, and connect over the private route, not out to the internet and back in.

If that probe does not exist, the first real test of the cross-cloud path is production traffic itself.

Probe from where the caller sits

Place a synthetic caller in each cloud that depends on the service, not just in the cloud that hosts it.

That caller should do the same steps the application does, in the same order:

  • Resolve the private service name using the same DNS view the application uses.
  • Open a connection to the resolved private address over the private route.
  • Complete the TLS handshake and check the certificate the application would see.
  • Make one small, safe, read-only request that exercises authentication without creating work or side effects.

Each step should be timed and reported separately. A single pass or fail hides the useful information. DNS took 2 seconds, TCP timed out, TLS failed, and the request returned 403 are four different incidents with four different owners. If the probe collapses them into one red dot, the on-call engineer still has to reproduce the journey by hand at 3 a.m.

A useful trick is to record the resolved address and the route identity with each probe result. When a probe starts resolving a public address instead of the private one, or starts succeeding through a backup route with much higher latency, that change is itself a signal, even while the probe still passes.

Do not let the probe take a shortcut

Probes quietly cheat in a few common ways.

The probe runs in a management network that has wider firewall access than the application network, so it passes through rules the application cannot use. The probe uses a cached DNS answer with a long TTL, so it keeps testing an old address after DNS has broken for everyone else. The probe talks to a health endpoint that is served locally in each cloud, so it never actually crosses the link it is supposed to test. Or the probe is allowed out to the public endpoint because the private route failed and the system silently fell back to the internet, masking the private failure and changing the security posture at the same time.

Fixing this means treating the probe like a real workload. Run it from the same subnet class, with the same DNS resolver, the same egress rules, and no special firewall exceptions. If the probe needs an exception to work, the application probably needs a deliberate rule too, and that rule should be written down, not hidden inside monitoring.

It also helps to add a negative check. Confirm that the private service is not reachable over the public path from the other cloud when policy says it should not be. A private path that works is good. A private service that also quietly works over the internet is a finding, not a convenience.

Alert on divergence, not just on failure

In a multi-cloud estate, the most informative event is often not a total failure. It is divergence.

One cloud can reach the service and another cannot. Latency from one source doubles while the other stays flat. DNS returns different addresses to different callers, and only one set is current. Those patterns point at the shared middle: routing, DNS propagation, or a rule change that was applied in one place and not the others.

So compare probe results across sources. Alert when sources disagree for more than a few minutes, even if a majority still passes. A service that is up for two clouds and down for the third is down, for the workloads in the third cloud. Averaging that away into a mostly healthy status page helps no one who is actually affected.

History matters here too. Keep probe latency and step timings long enough to compare this week with last week. Slow drift in cross-cloud latency often shows up in probe history days before it becomes a timeout, as routes change, tunnels renegotiate, or a provider path gets longer.

Keep the probe cheap, safe, and boring

A good synthetic request should be easy to allowlist and easy to explain to an auditor. Use a dedicated read-only endpoint or a harmless operation on an existing one. Give the probe its own workload identity with the minimum access it needs, rotate its credentials like any other workload, and log its calls separately so probe traffic is never mistaken for user traffic in an incident or an audit.

Set the frequency from the failure you want to catch, not from habit. Probing every few seconds from every cloud creates load and noise, and it can trip rate limits that then look like an outage. For most private services, a probe every 30 to 60 seconds per source cloud, with alerting that requires a short run of failures or a clear divergence, catches real breaks quickly without inventing false ones.

Finally, test the probe. Break the private route in a staging setup, expire a test certificate, or withdraw a DNS record, and confirm that the right step fails and the right alert fires with a readable message. A health check that has never been seen to fail is an assumption, not a check.

A short checklist to start

  • List the private cross-cloud calls a service depends on. Probe each one from the calling cloud.
  • Split each probe into DNS, connect, TLS, and request steps, with timings for each.
  • Run probes under the same network rules as the application. Remove probe-only shortcuts.
  • Record resolved address and latency with every result, and alert when sources diverge.
  • Add one negative check for a path that policy says should stay closed.
  • Break the path on purpose in staging and watch the probe fail correctly.

Teams that want a structured way to map these private paths, place probes consistently across AWS, Azure, Google Cloud, and Oracle estates, and keep the evidence tidy for review can look at the approach set out at CloudVerg, where VergRail and VergMesh are used for exactly this kind of cross-cloud connectivity work. The checklist above stands on its own either way. Start with the one call your service can least afford to lose, and make its health check tell the truth about the path it really takes.

Top comments (0)