Subdomains and robots.txt

reefoid · 24 Mar 2011 at 16:04

I have a subdomain (dev.domain.com) whose root is /public_html/dev. If I want to exclude bots from this subdomain, do I just need to disallow access to the /public_html/dev directory? Or is there something else I need to do as its a subdomain?

Thanks

gord · 24 Mar 2011 at 16:26

Subdomains will use their own robots.txt found in the root of the sub domain.

So if you have a robots.txt in /dev then it will use that.

reefoid · 24 Mar 2011 at 16:37

Aah, I see. So I just disallow all in there. Thanks.

Hammer100 · 27 Mar 2011 at 13:34

You may know this already but just thought I would mention in case. If you are disallowing robots to index certain areas you may want to hide your robots.txt for displaying in a browser as if someone see's locations disallowed they may have a snoop.
We this for a location that is only of benefit to our clients so our robots.txt file disallows the location and also our txt file is no accessible if anyone tries to view it in a browser.