DEV Community

a.infosecflavour
a.infosecflavour

Posted on

1

Jr Penetration Tester- Content Discovery- robots.txt

Closing the door of the first room, we're going to Discover(y) the Content of the second room. 🚪
We found a paper containing a question: Task 2- What is the directory in the robots.txt that isn't allowed to be viewed by web crawlers?

Accessing http://_machine_IP/robots.txt_, the message below shows up:

message

We're writing down the answer ✍️ /staff-portal. Just out of curiosity, let's see what's behind http://machine_IP/staff-portal. Here's the result:

robots

Remember the very first exercise of Intro to Offensive Security? We used gobuster to retrieve the hidden pages. Try to use the same command, to see if robots.txt can be found.

Top comments (0)

Billboard image

The Next Generation Developer Platform

Coherence is the first Platform-as-a-Service you can control. Unlike "black-box" platforms that are opinionated about the infra you can deploy, Coherence is powered by CNC, the open-source IaC framework, which offers limitless customization.

Learn more