robots.txt file

jbloggs

jbloggs

jbloggs

Soldato
Joined
4 Sep 2005
Posts
6,803
Location
UK
I have a family website with a registered URL, pointing to the web space it is on, however, I do not want Search Engine bots to be able to get onto the web space in an attempt to gather information about it for the purpose of indexing the website on Search Engines. I wish to keep it private, just for family members only.

I have heard that I can use a "robots.txt" file which will stop the search engines picking the site up.

I presume that a "robots.txt" file is simply a "Notepad" text file with the appropriate information in it, and placed on the website. What information would I need to put into this "Notepad" text file and where would I place it on the website?

I realise I could password it, but would prefer not too.

Many thanks for your help. :)
 
Create then upload a text file in notepad (or similar) called robots.txt into your root directory, with:

Code: 
User-agent: *
Disallow: /

contained in it. Each search engine uses its own user-agent tag and the star signifies "all". The Disallow could be any directory e.g. Disallow: /pictures/. However just "/" means any directory contained on the server.

A word of warning: Bots which have been made for malicious purposes clearly do not obey these rules and often use robots.txt files as a method to look for sensitive materials or vulnerable areas of websites.

Cheers
 
You must log in or register to reply here.
Back
Top Bottom