DEV Community: acetrondi

Your site might rank on Google but still be invisible to ChatGPT. Learn how to test GPTBot, ClaudeBot, PerplexityBot, Bingbot, and other crawlers, detect hidden CDN blocks, and make your content AI-accessible in under five minutes.

acetrondi — Sun, 19 Jul 2026 05:17:08 +0000

acetrondi

Jul 19

Your CDN Might Be Blocking ChatGPT From Your Docs (And robots.txt Won't Show It)

#webdev #seo #ai #nextjs

2 min read

Your CDN Might Be Blocking ChatGPT From Your Docs (And robots.txt Won't Show It)

acetrondi — Sun, 19 Jul 2026 05:16:06 +0000

Linux / macOS

Status Code

curl -sI -A "GPTBot" https://yoursite.com/ | head -1

Windows PowerShell

Status Code

(Invoke-WebRequest -Uri "https://yoursite.com/" -UserAgent "GPTBot").StatusCode

Windows Command Prompt (CMD)

Full Headers

curl -I -A "GPTBot" https://yoursite.com/

Note: Replace yoursite.com with your domain

If that returns anything other than 200, AI engines cannot read your site and your robots.txt may look perfectly permissive while it happens. Audits of several thousand sites found roughly 27% block at least one major AI crawler, most of them unintentionally, at the CDN or WAF layer rather than in robots.txt.

Why this happens

Bot-protection rules ship with sensible-sounding defaults: block unknown user agents, rate-limit non-browser traffic, challenge anything without JavaScript. GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot all trip at least one of those on common configurations. Your marketing team sees a clean robots.txt; your edge quietly returns 403.

The five-minute audit

for UA in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot PerplexityBot Bingbot; do

printf "%-16s %s\n" "$UA" "$(curl -sI -A "$UA" https://yoursite.com/ | head -1)"
done

Then check your edge logs for 403s grouped by user agent that's where the truth lives.

The second failure: client-side rendering
Several AI crawlers don't execute JavaScript. If your H1 and opening paragraph arrive via hydration, they see an empty shell. Server-render anything you want quoted view-source: is the test, not DevTools.

Allowing crawlers explicitly (Next.js App Router)

import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {

  const disallow = ['/admin', '/api/']

  return {

    rules: [

      { userAgent: '*', allow: '/', disallow },

      { userAgent: 'GPTBot', allow: '/', disallow },

      { userAgent: 'OAI-SearchBot', allow: '/', disallow },

      { userAgent: 'ClaudeBot', allow: '/', disallow },

      { userAgent: 'PerplexityBot', allow: '/', disallow },

      { userAgent: 'Google-Extended', allow: '/', disallow },

    ],

    sitemap: 'https://yoursite.com/sitemap.xml',

  }

}

robots.txt is necessary but not sufficient the CDN rule is the one that actually bites.

Does being crawlable matter?

It's the precondition, not the strategy. Being retrieved is won with ordinary search ranking; being cited is won with extractable structure direct answers, explicit definitions, sourced statistics. But none of that runs if the crawler gets a 403 at the door.

I'm publishing a live experiment on this: 20 fixed questions asked to ChatGPT and Gemini daily, with raw answers, cited-source leaderboards and the retrieved-but-not-cited gap published openly connectingdots.live/experiment

Using Supabase SMS-Hook to Send Custom Authentication Messages in India

acetrondi — Thu, 10 Oct 2024 12:29:51 +0000

Given the limited support for local SMS providers in Supabase, the platform has introduced an auth hook that allows users to integrate their own local SMS or email providers for sending OTPs. This feature offers flexibility for those seeking to implement custom authentication solutions.
Follow these steps to set up a custom SMS hook for authentication in your Supabase project:

1. Access the Hooks Dashboard

Navigate to your Supabase project dashboard
Go to https://supabase.com/dashboard/project/_/auth/hooks

2. Initiate Hook Creation

Click the "Add a new hook" button
Select "Send SMS hook" from the options

3. Configure Hook Settings

Toggle on "Enable Send SMS hook"
For Hook type, choose "HTTPS"

4. Set Up the Endpoint

In the URL endpoint field, enter: https://project_ref_id.supabase.co/functions/v1/sms-hook
Replace project_ref_id with your actual Supabase project reference ID

5. Generate and Save the Secret

Click on "Generate secret"
Copy the generated secret and store it securely

6. Create the Hook

Click "Create" to complete the setup

Implementing the SMS Hook Edge Function in Supabase

After setting up the custom SMS hook in your Supabase dashboard, the next step is to create and deploy an edge function that will handle the SMS sending logic. Follow these steps to implement the sms-hook function:

Prerequisites

Supabase CLI installed
Docker Desktop installed

Steps

1. Set Up Your Local Environment

a. Log in to Supabase CLI:

   npx supabase login

b. List your Supabase projects:

   npx supabase projects list

c. Link your local setup to your Supabase project:

   npx supabase link --project-ref your-project-id

2. Create the SMS Hook Function

Create a new function named "sms-hook":

   npx supabase functions new sms-hook

3. Implement the Function Logic

Open the index.ts file located at functions/sms-hook/index.ts and replace its content with the following code:

   import { Webhook } from "https://esm.sh/standardwebhooks@1.0.0";
   import { readAll } from "https://deno.land/std/io/read_all.ts";
   import * as base64 from "https://denopkg.com/chiefbiiko/base64/mod.ts";

   const sendTextMessage = async (
     messageBody: string,
     toNumber: string,
   ): Promise<any> => {
     const username = Deno.env.get("TEXT_GURU_USERNAME");
     const password = Deno.env.get("TEXT_GURU_PASSWORD");
     const source = Deno.env.get("SMS_HEADER");
     const tempId = Deno.env.get("SMS_TEMPLATE_ID");
     const url: string =
       `https://www.textguru.in/api/v22.0/?username=${username}&password=${password}&source=${source}&dmobile=${toNumber}&dlttempid=${tempId}&message=${messageBody}`;
     console.log("url:", url);
     const response = await fetch(url, {
       method: "POST",
       headers: {
         "Content-Type": "application/x-www-form-urlencoded",
       },
     });
     return response;
   };

   Deno.serve(async (req) => {
     const payload = await req.text();
     // Paste the earlier copied base64 secret here and remove the first "v1,whsec_" from the copied base64 secret
     const base64_secret = Deno.env.get("SEND_SMS_HOOK_SECRET");
     const headers = Object.fromEntries(req.headers);
     const wh = new Webhook(base64_secret);
     try {
       const { user, sms } = wh.verify(payload, headers);
       const messageBody =
         `Dear Customer, Your Verification Code For Company name Is ${sms.otp}. Please do Not share this Code with anyone.`;
       const response = await sendTextMessage(
         messageBody,
         user.phone,
       );
       // You can edit further code as the response coming from your sms provider
       if (response.statusText === "OK") {
         return new Response(
           JSON.stringify({ message: "SMS sent successfully." }),
           {
             status: 200,
             headers: {
               "Content-Type": "application/json",
             },
           },
         );
       }
       return new Response(
         JSON.stringify({
           error: {
             http_code: response.status,
             message:
               `Failed to send SMS: ${response.message}. More info: ${response.more_info}`,
           },
         }),
         {
           status: response.status,
           headers: {
             "Content-Type": "application/json",
           },
         },
       );
     } catch (error) {
       return new Response(
         JSON.stringify({
           error: {
             http_code: 500,
             message: `Failed to send sms: ${JSON.stringify(error)}`,
           },
         }),
         {
           status: 500,
           headers: {
             "Content-Type": "application/json",
           },
         },
       );
     }
   });

4. Deploy the Function

Deploy the sms-hook function using the following command:

   npx supabase functions deploy sms-hook --no-verify-jwt

5. Enable Phone Authentication

Remember to enable the phone provider from Authentication Providers in your Supabase Dashboard.

Important Notes

Make sure to replace placeholders like your-project-id with your actual Supabase project details.
The SEND_SMS_HOOK_SECRET should be set as an environment variable, using the secret you generated earlier (remove the v1,whsec_ prefix).
You'll need to set up environment variables for TEXT_GURU_USERNAME, TEXT_GURU_PASSWORD, SMS_HEADER, and SMS_TEMPLATE_ID according to your SMS provider's requirements.
This implementation uses TextGuru as the SMS provider. You may need to adjust the sendTextMessage function if you're using a different provider.

By following these steps, you'll have successfully implemented and deployed a custom SMS hook for authentication in your Supabase project.

How to create API keys in Supabase for roles other than "anon" and "service"?

acetrondi — Sun, 04 Jun 2023 06:32:29 +0000

If you're reading this post, you may already be familiar with Supabase. However, for those who are new, let me provide a brief introduction.

Supabase is an alternative to Firebase that utilizes PostgreSQL as its database and offers various features, including authentication, real-time capabilities, and storage.

To get started with Supabase, follow these steps:

Obtain the JWT key from the Supabase dashboard or through this link.

Never disclose your Jwt secret/token in public
Create a role in your SQL editor:

CREATE ROLE your_role;
GRANT your_role TO authenticator;
-- grant role privileges here

3.Visit jwt.io and populate the payload field with the following information:

    {
      "iss": "supabase",
      "ref": "project ref id",
      "role": "your_role",
      "exp": 2001128702
    }

Replace ref with your project reference ID from Dashboard

4.In the "Verify Signature" field, enter the JWT Token/Secret obtained in step 1.

Note: Default algorithm for JWT token is "HS256"

//Header
{
  "alg": "HS256",
  "typ": "JWT"
}

5.The "Encoded/Token" field will display the newly generated token. Copy this token and include it in your REST API or client code and requests as Authorization: Bearer new_generated_token. This token will have all the privileges you gave to your_role role from sql editor in supabase dashboard.

Note: Please exercise caution when assigning sensitive data as this role will have the privileges you grant it.

Peace!