MobileRead Forums - View Single Post - bulk metadata download through command line

MelBr · 09-01-2013, 12:49 PM

woodapple, even if you manage to import 200,000 books into calibre, are you sure calibre will perform well enough on your system with that big of a library? You say your files are rared to save space so your system might not even handle 20% of your library.

And what BetterRed said about getting banned & blacklisted by sites is absolutely true. Few months back, I got blacklisted by a publisher's site when I was looking up ISBN numbers and was scraping title & authors information. Even though I put 10-15 sec random wait time between queries, I got banned after 6 hrs. I was able to rename about 85% of files so it wasn’t that bad. Heck, even Google.com will ban you for much less than that and start giving you a captcha to solve before you are able to use their search and to remove captcha, you'll have to file a request and wait few days (trust me, I had to do that after google blocked me for doing few dozen autmated image searches).

cybmole, if they're books of fiction, I completely agree but if they're non-fiction, then having a lot of books can be extremely useful when you're synthesizing information and need to reference, cross-check and verify facts etc.

I do full-text search all the time on my library and it's proven extremely useful to me. You never know what you're gonna find when you do that.

09-01-2013, 12:49 PM	#10
MelBr Zealot Posts: 105 Karma: 414068 Join Date: Feb 2013 Device: iPad Pro, Kobo Aura One	woodapple, even if you manage to import 200,000 books into calibre, are you sure calibre will perform well enough on your system with that big of a library? You say your files are rared to save space so your system might not even handle 20% of your library. And what BetterRed said about getting banned & blacklisted by sites is absolutely true. Few months back, I got blacklisted by a publisher's site when I was looking up ISBN numbers and was scraping title & authors information. Even though I put 10-15 sec random wait time between queries, I got banned after 6 hrs. I was able to rename about 85% of files so it wasn’t that bad. Heck, even Google.com will ban you for much less than that and start giving you a captcha to solve before you are able to use their search and to remove captcha, you'll have to file a request and wait few days (trust me, I had to do that after google blocked me for doing few dozen autmated image searches). cybmole, if they're books of fiction, I completely agree but if they're non-fiction, then having a lot of books can be extremely useful when you're synthesizing information and need to reference, cross-check and verify facts etc. I do full-text search all the time on my library and it's proven extremely useful to me. You never know what you're gonna find when you do that. Last edited by MelBr; 09-01-2013 at 12:56 PM.