How to test docs code examples that need API keys (without leaking secrets)

DocsAlive Team — Tue, 10 Nov 2026 19:21:38 +0000

Code examples that need credentials are difficult to test in a clean documentation scan. A valid run may depend on access the scanner should not assume it has. But a missing key does not automatically mean the documentation is broken: a reader who cannot tell where to put their key has a documentation problem.

A practical approach is to split credential-dependent examples into layers:

Check syntax.
Test request construction and response parsing offline.
Run the live call separately with an authorized test-mode or sandbox account.

Use a clearly named environment variable, explain how readers obtain and provide their own value, and report skipped live calls as gaps in coverage.

Make the credential boundary clear

An API example may promise several things at once: that a package installs, a client can be configured, a request has the right shape, a key is accepted, an account has access, and a response can be parsed. A failed live call does not immediately show which promise failed.

A clean documentation check may not have a key. A forked pull request may run without repository secrets. A sandbox account may also be unavailable or lack a needed permission. Those conditions should be visible in test results, but they do not automatically establish that the example is wrong.

Start by showing readers exactly how the credential enters the example. This complete Python program reads a fictional Acme SDK credential without embedding or printing a key:

import os

api_key = os.environ.get("ACME_API_KEY")
if api_key:
    print("A key is configured")
else:
    print("Set ACME_API_KEY to run the live example")

ACME_API_KEY and the Acme SDK are fictional names. The program reports only whether a value is present. If you use a placeholder such as YOUR_API_KEY or <API_KEY>, explain which value replaces it and where the reader supplies it. A realistic-looking fake key can be mistaken for a real credential and is harder to distinguish from one.

State whether the value is a test-mode key, which environment variable the example reads, and which account or permission the live call requires. A placeholder alone does not explain how to configure the example.

Test what you can offline

Separate local work from the external call. Depending on the SDK, client construction can be local even when the client eventually needs credentials. Request building and parsing a recorded response can often be tested without contacting the service. If the SDK validates or contacts the service during construction, treat construction as part of the live boundary.

The test layers answer different questions:

Layer	What it establishes	What it does not establish
Syntax only	The selected source parses	Imports, behavior, authentication, or service access work
Offline stub or recorded fixture	Local request and response handling work for the tested case	The real service accepts the key or currently returns that response
Live sandbox call	The tested account and service path work at that time	Every account, permission, or production condition works

Make the offline test meaningful: build the same request shape as the live client, return a representative response, and assert the fields the documentation promises. Keep fixtures sanitized and free of real credentials and private customer data.

For example, this single file uses only the Python 3.12 standard library. It starts a local HTTP stub, sends a request, parses a canned JSON response, checks the result, and shuts the server down. It contacts no external API. Save it as offline_example.py:

import json
import os
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer
from urllib.request import Request, urlopen


class StubHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/v1/greeting":
            self.send_error(404)
            return

        body = json.dumps({"message": "Hello from the stub"}).encode("utf-8")
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, format, *args):
        pass


def fetch_greeting(base_url):
    api_key = os.environ.get("ACME_API_KEY", "offline-placeholder")
    request = Request(
        base_url + "/v1/greeting",
        headers={"Authorization": "Bearer " + api_key},
    )
    with urlopen(request, timeout=5) as response:
        return json.load(response)


def main():
    server = HTTPServer(("127.0.0.1", 0), StubHandler)
    thread = threading.Thread(target=server.serve_forever, daemon=True)
    thread.start()
    try:
        host, port = server.server_address
        result = fetch_greeting(f"http://{host}:{port}")
        assert result["message"] == "Hello from the stub"
        print(result["message"])
    finally:
        server.shutdown()
        thread.join()
        server.server_close()


if __name__ == "__main__":
    main()

Run it with:

python3 offline_example.py

It prints:

Hello from the stub

The default value is an offline placeholder. The stub does not validate it, so the example does not prove that a service accepts any credential. A recorded fixture is another offline option: load a representative, sanitized response and assert the parsing result. A fixture can exercise parsing; a local stub can also exercise request construction and status handling. Neither replaces a live authentication check.

Skip live tests clearly

When the key is absent, make the live test’s skip visible. This shell script exits successfully with a clear message and makes no network request:

if [ -z "${ACME_API_KEY:-}" ]; then
  printf '%s\n' 'SKIP: ACME_API_KEY is not set'
  exit 0
fi

printf '%s\n' 'Credential is present; run the authorized live test here'

Replace the final message with your project’s live-test command after configuring an authorized sandbox. A successful job that skipped the live call is not evidence that the call passed.

For a pytest suite, this complete test file demonstrates the skip condition without making a real API call. Save it as test_live_example.py:

import os

import pytest


@pytest.mark.skipif(
    not os.environ.get("ACME_API_KEY"),
    reason="ACME_API_KEY is required for the authorized live test",
)
def test_live_configuration_is_present():
    assert os.environ.get("ACME_API_KEY")

When the variable is absent, pytest reports a skip. When it is present, this demonstration checks only that a value exists; replace the body with an authorized sandbox call and assertions to verify live behavior. Do not print the key in failure messages, command traces, HTTP logs, or test artifacts.

Set CI up around trust

Store the sandbox credential in the CI platform’s secret store and map it to a job-level environment variable for the live job. A practical workflow has an offline job for ordinary changes and a live job or step that runs only when the authorized secret is available. Report a skip when it cannot run.

GitHub’s secrets documentation says secrets cannot be referenced directly in an if: conditional. Map the secret to a job-level environment variable, then check that variable in the step condition. With the exception documented for GITHUB_TOKEN, secrets are not passed to runners for workflows triggered from forks. Keep syntax and offline tests available there, and arrange the live check in an authorized context rather than making it a required check on fork pull requests.

Mask sensitive values in logs, and check shell tracing, HTTP debugging, exception messages, and uploaded artifacts as well as normal test output. Never deliberately print keys.

Triage results by what they show

A needs-credentials finding says the scan cannot establish a live result without credentials. It is not, by itself, a docs-bug classification. Human review may still find a documentation problem, such as a placeholder readers cannot map to the required configuration.

A sandbox response of 401 Unauthorized or 403 Forbidden needs investigation. It can result from a missing or invalid key, insufficient account permissions, or a setup mismatch. Check the authorized account, permissions, endpoint, and documented setup; preserve sanitized evidence without recording the key.

Keep the coverage outcome explicit:

Offline checks passed and the live test passed.
Offline checks passed and the live test was skipped because no credential was available.
Offline checks passed and the live test failed with an authentication or service response.
Syntax failed before either behavioral layer could run.

These outcomes call for different reviews. If a release requires live verification, arrange an authorized run before treating that requirement as met.

DocsAlive detects certain secret reads and placeholders, classifies credential-dependent blocks, and does not run them with customer-provided test credentials.