Prism spent 30 minutes on Stripe's OpenAPI spec and never opened a port

Disclosure: I work on Mockzilla, which competes with Stoplight Prism.
Everything below was measured on my laptop, and there is a link to the longer
comparison at the end.

Prism is the default answer to "I need to mock this OpenAPI spec", and for most
specs it is the right one. Install it, point it at a document, every operation
in the file answers.

npm install -g @stoplight/prism-cli
prism mock openapi.yml

Then I gave it Stripe.

The run

Stripe publishes its API as an OpenAPI document. The copy I used is 6.0 MB, 414
paths and 587 operations. Large, but not unusual: GitHub's is 8.8 MB.

$ prism mock stripe.yml
[CLI] …  awaiting  Starting Prism…

Nothing followed it. Thirty minutes on it was still that one line, the
process pinned at 100% of a core with 216 MiB resident and nothing listening on
port 4010. That is where I stopped it.

Handing it JSON instead does not help. The same document as stripe.json, same
machine: no listener after fifteen minutes, at 274 MiB. There is no flag for
this and nothing was misconfigured.

Where the time goes

Not parsing. Python's YAML parser loads the same file in 5.3 seconds.

It goes on resolution. Stripe's document carries 1,286 component schemas with
3,700 $refs pointing into them, and Prism walks all of that before it binds a
port. A spec with a few hundred inline schemas never shows you this.

The file itself is servable. The mock server I work on opens a port on the same
document in 195 ms and answers the first request at 364 ms.

What it costs on a spec that does work

Twilio Verify: 29 paths, 53 operations, 173 KB of YAML. Installed natively,
three runs.

	Prism 5.16.0
Start to first response	509 to 531 ms
Memory, warm	191 to 199 MiB
What you install	210 npm packages, 68 MiB

Half a second and 200 MB is nothing on a laptop. A container that boots and
dies on every CI job pays it every time.

One process, one document

"A single Prism instance serves one OpenAPI document." Prism's documentation
says so, and what it recommends for a second document is a second process on a
second port with a reverse proxy in front of both:

services:
  proxy:
    image: caddy
    ports:
      - '8080:80'
    depends_on:
      - prism_1
      - prism_2
  prism_1:
    image: stoplight/prism:4
    command: mock -p 4010 --host 0.0.0.0 petstore-v2.yaml
  prism_2:
    image: stoplight/prism:4
    command: mock -p 4010 --host 0.0.0.0 petstore-v3.yaml

Most test suites talk to more than one vendor. Three of them is three Node
processes, three ports, a Caddyfile and something to supervise the lot, on
every developer machine and in every CI job. Each process holds its own copy of
its own document, so the 191 to 199 MiB above is per API rather than per
machine.

What Prism does better

Three things, and if you need the first one this post ends here.

Forcing a status code, per request. Prefer: code=404 on any call, or
?__code=404 if a header is awkward, and Prism serves that response out of the
document. Ask for a code the document does not declare and you still get the
status, with a Prism error body saying so. What I run has no per-request
equivalent.

Enforcing the document's security. Twilio Verify declares HTTP basic auth,
so Prism answers 401 with a WWW-Authenticate header until you send
credentials. Prism's behaviour is the contract-accurate one, and it catches a
client that forgot to send a token.

Older and stranger documents. OpenAPI 2.0, the version still called
Swagger, and Postman Collections. If your document has not been converted yet,
that decides it.

If your spec is the big kind

A mock server that reads the document at start-up and resolves all of it has a
ceiling, and Stripe's spec is above it. Worth knowing before you put prism
mock in a CI job and find out from a timeout.

I wrote up the full comparison with both tools running the same specs, the
generated data either side, the per-request controls, and the prices:

Stoplight Prism alternative: an OpenAPI mock server with a URL