Robots!

tartag · 01-13-2013, 11:27 PM

Hey all, searched the forum and didn't find anything. If i leave Calibre running so that I can access the content server, does the url run the risk of getting indexed by a search engine bot? wasnt sure if there is a robots.txt or anything..?
Cheers.
tartag

kovidgoyal · 01-13-2013, 11:35 PM

As long as you dont put links up to your content server on the public internet, how would a search engine even know your server exists? And if you are amking your server available over the public internet you really should protect it with a password, in which case a spider would not be able to index it, even if it knew of its existence.

phiscribe · 12-31-2014, 09:11 AM

I didn't put a link to my server anywhere and I am getting crawled.

The only way I can think it got out was I am using a free dns service, https://duckdns.org/.

That or Google is sucking stuff out of my email.

A robots.txt with User-agent: * Disallow: / would be nice.

I guess I can try a password.

phiscribe · 12-31-2014, 09:20 AM

I added <meta name="robots" content="noindex" />

Hopefully that will do it.

kovidgoyal · 12-31-2014, 09:59 AM

You really should not expose the server via the public internet without a password. You could get in trouble if you are hosting any copyrighted work.

drjenkins · 12-31-2014, 10:46 AM

Even if you are hosting only public domain works if your server gets a lot of traffice it could get you into trouble with your ISP if your TOS prohibits running servers. A small group of friends with the password normally won't generate enough traffic to draw the attention of your ISP.

phiscribe · 12-31-2014, 01:39 PM

I just added a password. I am very suspicious of google plundering my email. That or the duckdns passing along stuff to search providers. Maybe my ISP is up to something.

I only emailed my daughters the address. Not sure how the search providers got involved. Opening it just made it so they could get in through their phone.

It was showing

+http://www.alexa.com/site/help/webmasters; crawler@alexa.com
IPv4Scan (+http://ipv4scan.com

Anyway, password up. Thanks

01-13-2013, 11:27 PM	#1
tartag Junior Member Posts: 1 Karma: 10 Join Date: Jan 2013 Device: none	Robots! Hey all, searched the forum and didn't find anything. If i leave Calibre running so that I can access the content server, does the url run the risk of getting indexed by a search engine bot? wasnt sure if there is a robots.txt or anything..? Cheers. tartag

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
People with Kindles are like robots in another world	Barcey	General Discussions	116	06-10-2012 07:30 AM
Calling all robots to action!	Chi Cygni	General Discussions	27	06-03-2012 04:40 PM
Robots on the loose!	abaldu8246	Self-Promotions by Authors and Publishers	0	07-05-2011 12:16 AM

01-13-2013, 11:35 PM	#2
kovidgoyal creator of calibre Posts: 43,858 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	As long as you dont put links up to your content server on the public internet, how would a search engine even know your server exists? And if you are amking your server available over the public internet you really should protect it with a password, in which case a spider would not be able to index it, even if it knew of its existence.

12-31-2014, 09:11 AM	#3
phiscribe Junior Member Posts: 3 Karma: 10 Join Date: Dec 2014 Device: Next7P12-8G	I didn't put a link to my server anywhere and I am getting crawled. The only way I can think it got out was I am using a free dns service, https://duckdns.org/. That or Google is sucking stuff out of my email. A robots.txt with User-agent: * Disallow: / would be nice. I guess I can try a password.

12-31-2014, 09:20 AM	#4
phiscribe Junior Member Posts: 3 Karma: 10 Join Date: Dec 2014 Device: Next7P12-8G	I added <meta name="robots" content="noindex" /> Hopefully that will do it.

12-31-2014, 09:59 AM	#5
kovidgoyal creator of calibre Posts: 43,858 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	You really should not expose the server via the public internet without a password. You could get in trouble if you are hosting any copyrighted work.

12-31-2014, 10:46 AM	#6
drjenkins Addict Posts: 250 Karma: 1702156 Join Date: Nov 2010 Device: Kindle Voyage	Even if you are hosting only public domain works if your server gets a lot of traffice it could get you into trouble with your ISP if your TOS prohibits running servers. A small group of friends with the password normally won't generate enough traffic to draw the attention of your ISP.

12-31-2014, 01:39 PM	#7
phiscribe Junior Member Posts: 3 Karma: 10 Join Date: Dec 2014 Device: Next7P12-8G	I just added a password. I am very suspicious of google plundering my email. That or the duckdns passing along stuff to search providers. Maybe my ISP is up to something. I only emailed my daughters the address. Not sure how the search providers got involved. Opening it just made it so they could get in through their phone. It was showing +http://www.alexa.com/site/help/webmasters; crawler@alexa.com IPv4Scan (+http://ipv4scan.com Anyway, password up. Thanks

Advert

Advert