DEV Community

VaultyPDF
VaultyPDF

Posted on Edited on

How to serve a Googlebot-certified sitemap.xml and robots.txt from a Motoko canister

Hey builders,

One of the biggest headaches when launching a public SaaS or utility dApp on the Internet Computer Protocol (ICP) is search engine indexing.

Most single-page React applications hosted on-chain either fail to serve a proper sitemap.xml or return uncertified responses that Googlebot rejects during pre-crawl HEAD checks.

While building VaultyPDF (an 18-tool on-chain PDF suite), I had to configure our Motoko actor (main.mo) to handle raw HTTP requests directly so Google could crawl and index all 18 tool routes cleanly. Here is how that setup works.

1. Handle Both GET and HEAD Requests

Before downloading your sitemap, Googlebot often sends a lightweight HEAD request to check the response headers. If your canister only handles GET requests, indexing fails. Make sure your http_request query function explicitly checks request.method == "HEAD" and returns an empty body with the proper XML headers.

2. Enforce Consensus Certification (upgrade = ?true)

For the IC HTTP gateway to serve a certified response that search crawlers trust, escalate your /sitemap.xml and /robots.txt routes to an update call by setting upgrade = ?true in your HttpResponse record:

let SITEMAP_HEADERS : [(Text, Text)] = [
  ("Content-Type", "application/xml; charset=utf-8"),
  ("Cache-Control", "public, max-age=86400"),
  ("X-Content-Type-Options", "nosniff"),
  ("Access-Control-Allow-Origin", "*"),
];

if (path.contains(#text "sitemap.xml")) {
  if (method == "HEAD") {
    return {
      status_code = 200;
      headers = SITEMAP_HEADERS;
      body = "".encodeUtf8();
      streaming_strategy = null;
      upgrade = ?true;
    };
  };
  return {
    status_code = 200;
    headers = SITEMAP_HEADERS;
    body = SITEMAP_XML.encodeUtf8();
    streaming_strategy = null;
    upgrade = ?true;
  };
};
Enter fullscreen mode Exit fullscreen mode

3. Point robots.txt Directly to Your Sitemap

Serve a matching /robots.txt route with Content-Type: text/plain; charset=utf-8 and upgrade = ?true so crawlers immediately discover every route in your dApp without manual submissions.


Want the complete pre-configured codebase?

Wiring up certified HTTP routing, Internet Identity login, on-chain blob storage, and multi-language React dashboards from scratch takes weeks.

If you want to skip the boilerplate setup for your next ICP project, I packaged the full 18-tool React 19 + Motoko codebase (including this certified SEO router and a built-in blog engine) into a starter template:

👉 **Grab the VaultyPDF React + Motoko Boilerplate on Gumroad

Have you run into indexing issues with single-page dApps on ICP? Let me know how you handled it in the comments!

Top comments (0)