Opening an AI API relay provider's website proves only that its web entry point is reachable. It does not prove that a specific model, route, and protocol can complete an API request.
At AI Relay Check, every result is scoped to a provider + model + protocol + timestamp. This keeps one failed model from being misreported as a provider-wide outage.
Current evidence scope
The public dataset currently contains 231 privacy-reviewed API samples: 168 passed and 63 failed. Failure sequences grouped by the same provider, model, and protocol form 13 observed failure episodes. The dataset includes HTTP 400, 404, 503, and timeout samples.
These are dated observations, not a guarantee of future behavior and not evidence about routes that were not tested.
Four checks that should not be collapsed into one score
| Evidence | What it establishes | What it does not establish |
|---|---|---|
| Website or console responds | The web service was reachable | A target model can generate |
Model ID appears in /v1/models
|
The key can see that model ID | Generation permission, identity, or stability |
| Minimal generation request passes | That key, route, protocol, and model worked at that time | Long-term reliability or billing accuracy |
| Account meter or ledger changes | The tested request produced an observed account charge | Pricing on other routes or subscription plans |
HTTP 400: request or compatibility mismatch
A 400 response may mean that the route rejects the model name, request fields, tool-call format, or another protocol-specific parameter. Start with a minimal request using a fixed model and protocol, then add parameters one at a time.
One 400 sample is not evidence that the entire provider is down.
HTTP 404: not necessarily a missing website
An API can return 404 while the provider homepage still returns 200. Common causes include a wrong endpoint, an unavailable model route, or a protocol path such as /v1/responses that the provider has not implemented.
Record the endpoint, model, and protocol together. Homepage status alone cannot diagnose an API 404.
HTTP 429: rate limit, quota, or account state
A 429 response can indicate request-rate limits, concurrency limits, route quota, or account allowance. The status code alone cannot reliably distinguish temporary throttling from insufficient balance.
Useful evidence includes the account-side usage record, the route's pricing rule, and a later independent retest.
HTTP 503: no usable upstream at that moment
A 503 response often means that the current route had no available upstream or that the gateway was temporarily unavailable. The next successful sample for the same model and protocol is useful recovery evidence.
However, next observed pass is not the exact recovery time. The interval between two tests remains unobserved.
A reproducible failure record
Each record should include:
- test timestamp;
- provider and route group;
- exact model ID;
- protocol;
- HTTP status and a privacy-reviewed error category;
- the next observed pass for the same scope.
Do not publish API keys, account identifiers, full request bodies, raw responses, or collection credentials.
Public evidence
- Failure and retest timeline
- Model compatibility matrix
- Ten-request billing reconciliation
- Privacy-reviewed failure sequence CSV
Conclusion
Website uptime, API reachability, model compatibility, and account billing are four different questions. A useful result preserves the conditions under which it was observed and relies on repeated samples, rather than turning one error into a permanent provider-wide verdict.
Disclosure: AI Relay Check has referral relationships with some tested providers. Referral status is excluded from test outcomes. Unverified model identity, cache behavior, pricing, and billing remain explicitly unverified.
Top comments (0)