A robots.txt file is a plain text file at the root of your domain. Crawlers read it before they request anything else, which makes it powerful and very easy to break. Most crawl problems trace back to five small habits.
1. Shipping the staging rule to production. Staging sites usually block everything. If that file travels with a deploy, the live site inherits it:
User-agent: *
Disallow: /
That pair of lines tells every well-behaved crawler to stay out. Add a deploy check that fails when the production file contains a bare slash disallow.
2. Blocking CSS and JavaScript. Old advice said to hide asset folders. Search engines now render pages, and if they cannot fetch your stylesheets and scripts they may see a broken layout, or miss content that your front end builds in the browser. Let them through.
3. Using robots.txt to remove pages from search. Disallow stops crawling, not indexing. A blocked URL that other sites link to can still appear in results, with no description, because the crawler never gets to read a noindex tag on the page. If you want a page gone, allow crawling and serve a noindex directive, or return the correct status code.
4. Wildcard slips. A rule such as Disallow: /*? blocks every URL containing a query string. That can include filtered category pages or paginated archives you actually want found. Write the narrowest pattern that does the job, then test it against a list of real URLs.
5. Forgetting that paths are prefix-based and case-sensitive. Disallow: /shop also blocks /shop-sale and /shopping-guide. If you only meant the folder, end the path with a slash. /Shop/ and /shop/ are different paths as far as the file is concerned.
A routine that prevents most of this
- Keep the file in version control so every change has an author and a date.
- Keep a short list of URLs that must always be crawlable, and test them after each release.
- Open the live file in a browser after every deploy. It takes seconds.
- Remember the file is public. Listing a private path to hide it only advertises it.
None of this needs special tooling, only discipline and a habit of checking. If your sites serve several markets or languages and you would like a second pair of eyes on crawl rules, redirects and rendering, technical SEO support from a team that works with businesses across the UAE and the GCC is a reasonable place to start.
Written with AI assistance for the Gorilla Vibe marketing team.
Top comments (0)