Hey all,
So I've been reading up on the use of robots.txt to prevent crawling etc... and I'm trying to get my head around the use of this if you have multiple sites.
Example:
Say you have your main hosting "www.myhostingdomain.co.uk" but then within that you have 3 separate websites within 3 folders:
awesomewebsite1a.co.uk in folder website1a
awesomewebsite2a.co.uk in folder website2a
awesomewebsite3a.co.uk in folder website3a
To get the above sites to map to the correct addresses you can set up domain mapping, but what would this mean in terms of needing to have the robots.txt in the root directory?
Logically I would think that because of the mapping, the root directory of awesomewebsite1a.co.uk would be the top level of the folder website1a, and so the mapping would go in here, meaning it would be like this:
awesomewebsite1a.co.uk = website1a/robots.txt
awesomewebsite2a.co.uk = website2a/robots.txt
awesomewebsite3a.co.uk = website3a/robots.txt
Is this correct? or would the robots.txt need to go within the root folder of "www.myhostingdomain.co.uk" and then have every setting for robots within that one file.
It sounds almost a bit redundant to have separate files as I'm writing this, but I'm thinking about it from the perspective of 3 separate owners of sites managing their own stuff, rather than having to contact their host to make changes.
As always, questions, assistance and general abuse welcome
Dave.
So I've been reading up on the use of robots.txt to prevent crawling etc... and I'm trying to get my head around the use of this if you have multiple sites.
Example:
Say you have your main hosting "www.myhostingdomain.co.uk" but then within that you have 3 separate websites within 3 folders:
awesomewebsite1a.co.uk in folder website1a
awesomewebsite2a.co.uk in folder website2a
awesomewebsite3a.co.uk in folder website3a
To get the above sites to map to the correct addresses you can set up domain mapping, but what would this mean in terms of needing to have the robots.txt in the root directory?
Logically I would think that because of the mapping, the root directory of awesomewebsite1a.co.uk would be the top level of the folder website1a, and so the mapping would go in here, meaning it would be like this:
awesomewebsite1a.co.uk = website1a/robots.txt
awesomewebsite2a.co.uk = website2a/robots.txt
awesomewebsite3a.co.uk = website3a/robots.txt
Is this correct? or would the robots.txt need to go within the root folder of "www.myhostingdomain.co.uk" and then have every setting for robots within that one file.
It sounds almost a bit redundant to have separate files as I'm writing this, but I'm thinking about it from the perspective of 3 separate owners of sites managing their own stuff, rather than having to contact their host to make changes.
As always, questions, assistance and general abuse welcome
Dave.