Thread: Archive.org
View Single Post
Old 08-23-2015, 03:39 PM   #13
CWatkinsNash
IOC Chief Archivist
CWatkinsNash ought to be getting tired of karma fortunes by now.CWatkinsNash ought to be getting tired of karma fortunes by now.CWatkinsNash ought to be getting tired of karma fortunes by now.CWatkinsNash ought to be getting tired of karma fortunes by now.CWatkinsNash ought to be getting tired of karma fortunes by now.CWatkinsNash ought to be getting tired of karma fortunes by now.CWatkinsNash ought to be getting tired of karma fortunes by now.CWatkinsNash ought to be getting tired of karma fortunes by now.CWatkinsNash ought to be getting tired of karma fortunes by now.CWatkinsNash ought to be getting tired of karma fortunes by now.CWatkinsNash ought to be getting tired of karma fortunes by now.
 
CWatkinsNash's Avatar
 
Posts: 3,950
Karma: 53868218
Join Date: Dec 2010
Location: Fruitland Park, FL, USA
Device: Meebook M7, Paperwhite 2021, Fire HD 8+, Fire HD 10+, Lenovo Tab P12
Quote:
Originally Posted by jgaiser View Post
archive.com honors robot.txt. Don't want them to access your pages? Use it.

http://www.robotstxt.org/robotstxt.html
Also: https://archive.org/about/exclude.php
CWatkinsNash is offline   Reply With Quote