Piwigo - Preventing spider / robot / Google indexing your site

[Captain Chaos]

Hi all

Thread title says it all really.

What do you all use (well those that use Piwigo) to stop google etc indexing your site?

I've installed the piwigo plugin 'meta' and added:- <meta name="robots" content="noindex,nofollow">

I believe this will then add this to each page and thus stop the indexing?

Just after confirmation really as I dont want all my stuff ending up on google

 

[Sp00n]

Robots.txt + meta tags.

If the bots still aren't respecting those rules then I tend to drop connections by agent id.

 

[Captain Chaos]

Well im an uber noob when it comes to all this stuff.

where would i put robots.txt and how would i know if the bots are not respecting da 'rules lol.

Only really started getting my head round this server stuff last week so has been a bit of a learning curve.

Got ubuntu on with firewall setup, fail2ban seems to be doing its job and piwigo is running ok thus far. Just want to make sure my stuff on piwigo stays there lol.

 

[AHarvey]

where would i put robots.txt

"You must apply the following saving conventions so that Googlebot and other web crawlers can find and identify your robots.txt file: You must save your robots.txt code as a text file, You must place the file in the highest-level directory of your site (or the root of your domain)"

 

[Captain Chaos]

so I would just create a txt document, call it robots.txt with the following info in it:-

User-agent: *
Disallow: /

I would then put it here?

/var/www/html/robots.txt

 

[AHarvey]

yep

 

[Captain Chaos]

Done and thanks,

For anyone doing a search down the line and finds they cannot add files (permision denied)

Just run the following command to give you access (change yourusername to your username of course!)

sudo chown yourusername.users /var/www/html


Once done I used the following website to check my URL was blocking said bots:-

https://en.onpage.org/free-tools/robots-txt/

It seems it is working fine!