DEV Community: ByteStay

Lint your Dockerfile and K8s manifests against Pod Security Standards in one call

ByteStay — Tue, 03 Nov 2026 14:00:05 +0000

Disclosure: we're ByteStay and we built Shipshape, the API in this post. There's a free plan and a no-signup demo, so you can check everything below yourself.

Take the nginx Deployment from the Kubernetes docs (deployment.yaml), the one most of us copied at some point. Run it through Shipshape and it gets a D, 42/100:

13 findings in total: 5 medium, 6 low and 2 info.
4 of the mediums are Pod Security Standards "restricted" checks: capabilities not dropped, privilege escalation allowed, runAsNonRoot not set, no seccomp profile.
The fifth medium is the missing memory limit.

That's not a criticism of the docs: the example is there to show a Deployment, not a hardened one. But it's what a lot of real manifests look like.

What Shipshape checks

You POST a Dockerfile or one or more Kubernetes manifests (YAML or JSON, multi-document is fine). You get back findings, a score and a pass/fail flag.

59 rules: 26 for Dockerfiles and 33 for Kubernetes.
Kubernetes:
- Pod Security Standards, both baseline and restricted.
- Resources and probes.
- :latest images.
- Service-account token automount.
- Secrets in env vars.
Dockerfiles:
- Root in the final stage, following FROM <stage> inheritance.
- Unpinned base images.
- curl | sh.
- apt/apk/pip cache hygiene.
- Secrets in ENV/ARG.
- ADD vs COPY.
- Shell-form CMD.
Every finding has a line number, the object and container it applies to, a fix, and the standard behind it: PSS, the NSA/CISA Kubernetes Hardening Guide, the CIS Docker Benchmark, or the hadolint id.

One finding from the nginx example, as the production API returns it:

{
  "id": "K8S-009",
  "severity": "medium",
  "title": "Capabilities do not drop ALL",
  "line": 16,
  "object": "Deployment/nginx-deployment",
  "container": "nginx",
  "fix": "Add `securityContext.capabilities.drop: [\"ALL\"]` and add back only NET_BIND_SERVICE if needed.",
  "reference": "PSS Restricted: Capabilities; NSA/CISA hardening",
  "pss": "restricted"
}

From D to A

Here's the same Deployment after working through the findings. The image is the unprivileged nginx build, so it listens on 8080 and runs as non-root.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: web
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: nginx
        image: nginxinc/nginx-unprivileged:1.27.2
        ports:
        - containerPort: 8080
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          capabilities:
            drop: ["ALL"]
        resources:
          requests:
            cpu: 100m
            memory: 64Mi
          limits:
            memory: 128Mi
        livenessProbe:
          httpGet: {path: /, port: 8080}
        readinessProbe:
          httpGet: {path: /, port: 8080}
        volumeMounts:
        - {name: tmp, mountPath: /tmp}
      volumes:
      - name: tmp
        emptyDir: {}

Result: A, 100/100. One info finding is left, "No CPU limit", and it's info on purpose: CPU limits cause throttling, and plenty of teams choose not to set them.

Grade and `passed` are different things

The score is overall hygiene: 100, minus 20 per high, 8 per medium and 3 per low (info is free). passed is your CI gate: it's false when any finding is at or above fail_on, which is high by default.

So they can disagree, and that's deliberate:

The nginx Deployment: D, but passed: true. Nothing in it is high, so it doesn't break the build by default.
Docker's own welcome-to-docker Dockerfile: B (80/100), but passed: false. It has one high finding: the final stage never sets USER, so the container runs as root.

Adding USER node before CMD takes it to A, 100/100 and passed: true.

On every pull request: the GitHub Action

We published an open-source GitHub Action for this. With your RapidAPI key stored as the secret RAPIDAPI_KEY, it's one step:

name: shipshape
on: pull_request
permissions:
  contents: read
  pull-requests: read
jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: Bytestay/shipshape-action@v1
        with:
          rapidapi-key: ${{ secrets.RAPIDAPI_KEY }}

What it does on each pull request:

It lints only the Dockerfiles and manifests the pull request changed. Helm templates and YAML that isn't Kubernetes are skipped locally, so a pull request usually costs 1 to 3 requests.
Findings show up as annotations on the diff: high as errors, medium as warnings, each with the fix.
The job summary has a table per file with the grade, the score and every finding.
The step fails on high by default (fail-on), or when a file grades worse than min-grade.
If the API is down or your quota runs out, you get a warning. Your build is never failed because of us.

The source is on GitHub (MIT).

The Action takes a RapidAPI key for now. Support for api.market keys is planned, but it isn't there yet.

Other CI systems

Prefer no third-party action, or using GitLab CI, Jenkins or a pre-commit hook? You only need curl and jq, one request per file. Here it is as a GitHub Actions step; the same commands work anywhere:

- name: Shipshape
  env:
    RAPIDAPI_KEY: ${{ secrets.RAPIDAPI_KEY }}
  run: |
    jq -Rs '{content: ., fail_on: "high"}' Dockerfile |
      curl -sS -X POST https://shipshape-dockerfile-and-kubernetes-lint-api.p.rapidapi.com/v1/dockerfile \
        -H "X-RapidAPI-Key: $RAPIDAPI_KEY" \
        -H "X-RapidAPI-Host: shipshape-dockerfile-and-kubernetes-lint-api.p.rapidapi.com" \
        -H "Content-Type: application/json" --data-binary @- > shipshape.json
    jq -r '.findings[] | "::\(if .severity == "high" then "error" elif .severity == "medium" then "warning" else "notice" end) file=Dockerfile,line=\(.line // 1)::\(.id) \(.title)"' shipshape.json
    jq -e '.passed' shipshape.json > /dev/null

The last line makes the step fail when passed is false. For manifests, use /v1/kubernetes, or use /v1/lint, which detects the type. To skip a rule you've decided to accept, add "ignore": ["DF-017"] to the body.

In your coding agent: the MCP server

If an AI coding agent writes your Dockerfiles and manifests, it can run the same check before handing them over.

The open-source Shipshape MCP server runs on your machine and uses your own key, from RapidAPI or api.market:

{
  "mcpServers": {
    "shipshape": {
      "command": "uvx",
      "args": ["--from", "bytestay-shipshape-mcp", "shipshape-mcp"],
      "env": { "RAPIDAPI_KEY": "your-rapidapi-key" }
    }
  }
}

Its lint tool takes the file's content or a local path, and returns the grade and every finding with its fix.
With an api.market key, set API_MARKET_KEY instead of RAPIDAPI_KEY.
It's listed in the official MCP Registry as io.github.Bytestay/shipshape-mcp.
Each tool call is one request on your plan.

Limits

Helm templates aren't parsed. Render them first (helm template … | …).
Cross-object checks are out of scope. NetworkPolicies, PodDisruptionBudgets and RBAC aren't checked.
Base images aren't inspected. If a third-party base image sets USER itself, we can't see it, and the finding says so.
Line numbers are best effort: the instruction's first line in Dockerfiles, the key's line in YAML.

Privacy

The file is parsed in memory and nothing is stored or fetched. Manifests and Dockerfiles are never logged.

Try it

No signup: the demo on shipshape.bytestay.pt runs the real API in your browser. Paste a Dockerfile or a manifest.
Free plan: on RapidAPI, Basic is $0 for 15 requests a month, with a hard limit, so it never runs up a bill. Paid plans start at $9 for 10,000 requests.
Docs and plans: RapidAPI listing.
Also on api.market: a 7-day free trial, once per customer. Shipshape on api.market.

If a rule fires where it shouldn't, tell us. False positives are the bugs we fix first.

Google-Extended isn't a crawler. Why our fetcher honours it anyway

ByteStay — Wed, 28 Oct 2026 15:10:33 +0000

Disclosure: we're ByteStay and we built Distill, the URL-to-Markdown API in this post. There's a free plan and a no-signup demo, so you can check everything below yourself.

Google-Extended shows up in robots.txt like any other user agent:

User-agent: Google-Extended
Disallow: /

But no bot ever sends it. Googlebot keeps fetching the page as before. The token only tells Google whether the content may be used for Gemini. Applebot-Extended does the same for Apple.

So a fetcher that announces itself as DistillBot could skip that group and be technically right: it isn't Google-Extended. We don't skip it. Here's why, with numbers.

What 37 robots.txt files say

On 27 October 2026 we downloaded the robots.txt of 37 big sites:

international news: BBC, The New York Times, The Guardian, the FT, Reuters, AP, The Verge and others;
Portuguese news: Público, Expresso, Observador, JN, DN, RTP, SAPO;
platforms: GitHub, Medium, Substack, Wikipedia.

We ran each one through the same parser Distill uses in production and asked about the home page (/):

36 of 37 let User-agent: * in.
29 of those 36 refuse at least one AI crawler by name.
7 name no AI crawler at all: Wikipedia, Forbes, Substack, The Independent, The Telegraph, DN and RTP.

How often each name is refused:

token	sites refusing it
`Bytespider`	27
`CCBot`	26
`ClaudeBot`	25
`Applebot-Extended`	25
`Google-Extended`	18
`GPTBot`	16

The second bullet is the one that matters. A scraper that reads only the * group would fetch 29 home pages whose owners said no to AI crawlers by name.

How Distill decides

The parser follows RFC 9309: the longest matching rule wins, * and $ wildcards work, and groups for the same agent are merged. Then:

If there's a DistillBot group, it decides alone. That's the site owner talking to us directly.
Otherwise the path must be allowed by * and by every group for a known AI crawler. There are 15 of them:
- OpenAI: GPTBot, ChatGPT-User, OAI-SearchBot
- Anthropic: ClaudeBot, Claude-User, Claude-SearchBot, anthropic-ai
- Google: Google-Extended
- Apple: Applebot-Extended
- Perplexity: PerplexityBot, Perplexity-User
- Meta: Meta-ExternalAgent
- Common Crawl: CCBot
- Cohere: cohere-ai
- ByteDance: Bytespider

This is the core of the production code:

own = self.groups.get(settings.BOT_TOKEN.lower())
if own is not None:
    return None if allowed(own, path) else settings.BOT_TOKEN
for token in ("*", *settings.AI_OPT_OUT_TOKENS):
    rules = self.groups.get(token.lower())
    if rules is not None and not allowed(rules, path):
        return token
return None

And this is what a user gets back for the BBC's news page, from the production API:

{
  "url": "https://www.bbc.co.uk/news",
  "status": "robots_txt",
  "detail": "robots.txt disallows this path for GPTBot",
  "markdown": "",
  "took_ms": 146
}

Across the sample, Distill reads 7 of the 37 home pages and refuses 30: the 29 above, plus Reuters, whose * group blocks /. That's the product working as intended.

So why Google-Extended?

The honest finding first: in our sample, Google-Extended never decided anything on its own. All 18 sites that refuse it also refuse other AI crawlers by name. Taking it off our list would change none of these 37 answers today.

We keep it anyway:

It's the clearest statement of intent there is. A site that writes User-agent: Google-Extended / Disallow: / is saying "index me, but don't feed me to an LLM". Distill fetches pages only to feed them to LLMs. That's exactly the use they refused.
The name of the bot is an accident. Usage tokens exist because Google and Apple don't run a separate AI crawler. We go by what the page will be used for, not by who knocks.
Owners can still let us in. A DistillBot group overrides everything else, so a site that blocks AI in general but is happy for Distill users to read its docs can say exactly that:

User-agent: DistillBot
Allow: /docs/
Disallow: /

User-agent: *
Disallow: /

What being strict costs

Some refused pages are ones the user may well have the right to read. When that happens:

The response says why: status: "robots_txt" and, in detail, the group that refused.
We don't retry another way. Our Terms forbid fetching through other means a page that refused DistillBot.
The same goes for noai in a <meta name="robots"> tag or an X-Robots-Tag header, even when the user sends us the HTML directly.

A few more rules from the same RFC:

robots.txt is checked on every host, including redirect targets, and cached for up to an hour.
If it answers 5xx or 429, we treat everything as disallowed and check again after 5 minutes.
A missing robots.txt (404) means no rules.

The fine print on these numbers

It's the home page only. Article paths can have different rules, so a site can refuse / and allow /articles/, or the other way round.
It's 37 big publishers and platforms, not the web. We didn't measure smaller sites, and they may look very different.
robots.txt files change. These are the files as served on 27 October 2026.

Try it

For site owners: how DistillBot reads robots.txt, and how to opt out or in.
No signup: the demo on distill.bytestay.pt runs the real API in your browser. Try a site you know and see what its robots.txt says.
Free plan: Basic is $0 for 15 requests a month, with a hard limit, so it never runs up a bill. Paid plans start at $9 for 10,000 requests.
Docs and plans: RapidAPI listing.

If you run a site and think we read your robots.txt wrong, tell us. That's a bug we want to fix first.

Turn any web page into clean Markdown for your LLM, without ignoring robots.txt

ByteStay — Tue, 27 Oct 2026 12:16:35 +0000

Disclosure: we're ByteStay and we built Distill, the API in this post. There's a free plan and a no-signup demo, so you can check everything below yourself.

If you feed web pages to an LLM, most of what you send isn't content. Navigation, footers, cookie banners, inline scripts, tracking pixels and sidebars all cost tokens and add noise to your embeddings.

A real example: the Python tutorial page on Data Structures is 111,008 bytes of HTML. The main content as Markdown is 28,453 characters, about 7,100 tokens, and keeps the headings, the code blocks and the 31 links.

That's roughly a quarter of the size, and none of what's gone is anything your model needed.

What Distill does

You send a URL, or HTML you already have. You get back:

the page's main content as Markdown: headings, lists, GFM tables, fenced code blocks with their language;
metadata: title, author, published date, language and site name;
counts: words, characters and an approximate token count;
the links list, with relative URLs already made absolute.

Here's the GitHub Blog's "What is Git?" post, as returned by the production API today (trimmed):

{
  "status": "ok",
  "title": "What is Git? Our beginner’s guide to version control",
  "author": "Kedasha Kerr",
  "published_date": "2024-05-27",
  "language": "en-US",
  "site_name": "The GitHub Blog",
  "markdown": "What is Git? Our beginner’s guide to version control\n\nLet’s get you started on your Git journey…",
  "truncated": true,
  "took_ms": 1825
}

Quick start

Through RapidAPI, with your key:

curl -G "https://distill-html-to-markdown-api.p.rapidapi.com/v1/markdown" \
  --data-urlencode "url=https://en.wikipedia.org/wiki/Markdown" \
  -d include_links=false -d max_chars=4000 \
  -H "X-RapidAPI-Key: YOUR_RAPIDAPI_KEY" \
  -H "X-RapidAPI-Host: distill-html-to-markdown-api.p.rapidapi.com"

The options you'll actually use:

max_chars cuts at a paragraph or word boundary, closes any open code fence and sets truncated: true;
include_links (default on) and include_images (default off);
include_tables (default on) keeps tables as GFM.

Already have the HTML, for example from your own crawler or a headless browser? POST it with a base_url, and the relative links come back resolved.

A small RAG ingest loop in Node.js

const url = new URL("https://distill-html-to-markdown-api.p.rapidapi.com/v1/markdown");
url.search = new URLSearchParams({ url: "https://docs.python.org/3/library/json.html", include_links: "false" });

const res = await fetch(url, {
  headers: {
    "X-RapidAPI-Key": process.env.RAPIDAPI_KEY,
    "X-RapidAPI-Host": "distill-html-to-markdown-api.p.rapidapi.com",
  },
});
if (!res.ok) throw new Error(`Distill ${res.status}`); // 401, 422, 429…
const page = await res.json();
if (page.status !== "ok") console.warn(page.status, page.detail); // robots_txt, blocked…
else chunks.push(...page.markdown.split("\n## ").map((s) => ({ text: s, url: page.final_url, title: page.title })));

Statuses, not exceptions

A page you can't use isn't an error in your code: it's an answer. Distill always returns HTTP 200 with a status:

status	what happened
`ok`	Markdown extracted
`no_content`	the page loaded, but there's no main content (often a JavaScript-built page)
`robots_txt`	robots.txt disallows the path
`opted_out`	the page says `noai`
`blocked`	the site's bot protection said no
`not_html`	PDF, image, JSON…
`http_error` / `unreachable`	the site errored or couldn't be reached

HTTP errors are kept for your mistakes (422 for a bad URL) and for limits (429).

What we refuse to do

This is the part we care about most. Scraping for AI has a bad reputation, mostly earned.

robots.txt is checked on every redirect hop, and if it answers 5xx we treat the path as disallowed (RFC 9309).
AI-crawler groups are honoured too. If a site blocks GPTBot, ClaudeBot, CCBot, Google-Extended or a dozen others, we don't fetch it either.
noai opt-outs in X-Robots-Tag or <meta name="robots"> are respected.
We identify ourselves as DistillBot/1.0 with a bot page. We never pretend to be a browser, and we never retry a bot wall from somewhere else.
At most 10 fresh fetches per minute per site, across all customers. Results are cached for an hour, so popular pages don't get hammered.

Limits, honestly

JavaScript isn't run. Pages that build their content in the browser come back as no_content. If you already render them, POST the HTML.
Pages are read up to 5 MB, with a 20-second total budget per request.
Extraction is heuristic (we build on trafilatura), so some layouts come out imperfect.

Try it

No signup: the demo on distill.bytestay.pt runs the real API in your browser.
Free plan: Basic is $0 for 15 requests a month, with a hard limit, so it never runs up a bill. Paid plans start at $9 for 10,000 requests.
Docs and plans: RapidAPI listing.

If a page comes out wrong, send us the URL: those reports are how the extraction gets better.

DEV Community: ByteStay

Lint your Dockerfile and K8s manifests against Pod Security Standards in one call

What Shipshape checks

From D to A

Grade and passed are different things

On every pull request: the GitHub Action

Other CI systems

In your coding agent: the MCP server

Limits

Privacy

Try it

Google-Extended isn't a crawler. Why our fetcher honours it anyway

What 37 robots.txt files say

How Distill decides

So why Google-Extended?

What being strict costs

The fine print on these numbers

Try it

Turn any web page into clean Markdown for your LLM, without ignoring robots.txt

What Distill does

Quick start

A small RAG ingest loop in Node.js

Statuses, not exceptions

What we refuse to do

Limits, honestly

Try it

Grade and `passed` are different things