Looking for tips/tricks to help speed things up~

Mashkai · 08-22-2016, 06:23 PM

So, I found myself with some spare time this week and thought I would finally start sorting my books/libraries again. I started with the smallest one needing to be sorted (approx. 6500 books). Each book already having its correct ISBN number which seems to help things a bit in my experience. In chunks of 300-500 books each, I began downloading metadata only. Judging by the current download times and the completed downloads, I estimate it will take approx. 33-36 hrs. until completed. With this being the smallest batch, I am hesitant to attempt the larger libraries.

Thus, I was rather hoping to discover some tips/tricks to help me speed this up whether that be through settings, plugins, or whatever else.

Thank you in advance to anyone who replies!

kovidgoyal · 08-22-2016, 10:16 PM

Metadata downloading is deliberately slow so as to be courteous to the people that run the servers from which the information is scraped. Not to mention that many providers will try to block your IP address if they think you are running a bot.

sealbeater · 08-30-2016, 01:42 AM

Quote:

Originally Posted by kovidgoyal

Metadata downloading is deliberately slow so as to be courteous to the people that run the servers from which the information is scraped. Not to mention that many providers will try to block your IP address if they think you are running a bot.

Just a question, how does the metadata slowdown work? Is it a fixed amount of time between queries and if so, how long or does it have some other sort of delay built in?

As for the bot thing, I use proxychains for just such a thing.

kovidgoyal · 08-30-2016, 01:45 AM

THere is no bandwidth throttling if that's what you are asking. IIRC (you'd have to check the code to be sure) the slowdown is simply done by not running multiple requests to the same domain in parallel and each individual metadata plugin might implement additional slowdowns itself if needed.

08-22-2016, 06:23 PM	#1
Mashkai Junior Member Posts: 5 Karma: 10 Join Date: Aug 2014 Device: none	Looking for tips/tricks to help speed things up~ So, I found myself with some spare time this week and thought I would finally start sorting my books/libraries again. I started with the smallest one needing to be sorted (approx. 6500 books). Each book already having its correct ISBN number which seems to help things a bit in my experience. In chunks of 300-500 books each, I began downloading metadata only. Judging by the current download times and the completed downloads, I estimate it will take approx. 33-36 hrs. until completed. With this being the smallest batch, I am hesitant to attempt the larger libraries. Thus, I was rather hoping to discover some tips/tricks to help me speed this up whether that be through settings, plugins, or whatever else. Thank you in advance to anyone who replies!

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[Tips & Tricks] Some useful tips and tricks	kguil	Marvin	0	06-25-2016 05:18 AM
Some Koreader tips and tricks	WS64	Kobo Reader	29	08-01-2014 11:48 AM
Odyssey Some tips and tricks	WS64	Bookeen	10	06-06-2013 02:53 PM
Request: Tips and Tricks	nlundberg	Calibre	9	03-21-2013 04:32 PM
Looking for: Tips and Tricks	PeterT	Kobo Reader	2	11-29-2012 11:44 PM

08-22-2016, 10:16 PM	#2
kovidgoyal creator of calibre Posts: 45,299 Karma: 27111240 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Metadata downloading is deliberately slow so as to be courteous to the people that run the servers from which the information is scraped. Not to mention that many providers will try to block your IP address if they think you are running a bot.

08-30-2016, 01:45 AM	#4
kovidgoyal creator of calibre Posts: 45,299 Karma: 27111240 Join Date: Oct 2006 Location: Mumbai, India Device: Various	THere is no bandwidth throttling if that's what you are asking. IIRC (you'd have to check the code to be sure) the slowdown is simply done by not running multiple requests to the same domain in parallel and each individual metadata plugin might implement additional slowdowns itself if needed.

Advert