Thread: Archive.org
View Single Post
Old 08-23-2015, 04:08 PM   #16
thewitt
Zealot
thewitt ought to be getting tired of karma fortunes by now.thewitt ought to be getting tired of karma fortunes by now.thewitt ought to be getting tired of karma fortunes by now.thewitt ought to be getting tired of karma fortunes by now.thewitt ought to be getting tired of karma fortunes by now.thewitt ought to be getting tired of karma fortunes by now.thewitt ought to be getting tired of karma fortunes by now.thewitt ought to be getting tired of karma fortunes by now.thewitt ought to be getting tired of karma fortunes by now.thewitt ought to be getting tired of karma fortunes by now.thewitt ought to be getting tired of karma fortunes by now.
 
Posts: 129
Karma: 1135446
Join Date: Nov 2014
Location: Miami, Florida
Device: iBooks
Quote:
Originally Posted by jgaiser View Post
archive.org honors robot.txt. Don't want them to access your pages? Use it.

http://www.robotstxt.org/robotstxt.html

Don't believe it. All my sites have included exclusions and requested removal. No dice.
thewitt is offline   Reply With Quote