Why so slow on metadata

lemonboy · 08-26-2010, 08:52 PM

I have the 0.7.15 calibre with a load of lit files and windows vista.

If I try to load metadata from the web, it takes an inordinate amount of time.

I have an isbndb account.

I can take the title and search on amazon and find the isbn before calibre decides it cant find anything.

When I then put in the isbn found from amazon it takes about 2 mins to find the metadata.

Why does this take so long, if I try to search for multiple files the problem is multiplied and effectively makes the multiple lookup function impossible.

When carrying out a multiple search, the gui only tells you how many files are being searched, it gives no indication of progress, therefore if it is locked up you do not know.

Is the lookup using some form of scraping on Amazon, if so what code is it using and where is the function (in what language). I'm sure this can be much more effective - as I said earlier, I can actually do this quicker with a manual search in amazon. If someone can tell me where to look in the code, I can surely try to make this more efficient.

kovidgoyal · 08-26-2010, 09:01 PM

probably because you have a proxy setup

lemonboy · 08-26-2010, 09:05 PM

No, no proxy. Direct internet connection. As I said I can do a manual amazon search quicker - to at least find the isbn

Thanks for the quick response.

Starson17 · 08-26-2010, 09:13 PM

Quote:

Originally Posted by lemonboy

I have the 0.7.15 calibre with a load of lit files and windows vista.

I run Vista

Quote:

If I try to load metadata from the web, it takes an inordinate amount of time.

Not on mine

Quote:

I have an isbndb account.

Me too.

Quote:

When I then put in the isbn found from amazon it takes about 2 mins to find the metadata.

Mine takes about 3 seconds with the ISBN and about 13 seconds to find 20 possible matches when only author/title are listed.

Code:

Why does this take so long, if I try to search for multiple files the problem is multiplied and effectively makes the multiple lookup function impossible.

I multiple search very often.

Quote:

Is the lookup using some form of scraping on Amazon, if so what code is it using and where is the function (in what language). I'm sure this can be much more effective - as I said earlier, I can actually do this quicker with a manual search in amazon. If someone can tell me where to look in the code, I can surely try to make this more efficient.

I can tell you where in the code to look, but you'd be better off finding what's borked on your machine first.

kovidgoyal · 08-26-2010, 09:26 PM

are you sure you have no proxy? calibre reads proxy settings from the registry (the same settings that are used by internet explorer) if you dont use internet explorer you may never realize you have the proxy settings set.

Malware in windows often sets proxy settings to compromise your machine.

lemonboy · 08-26-2010, 09:32 PM

Quote:

Originally Posted by Starson17

I run Vista

I can tell you where in the code to look, but you'd be better off finding what's borked on your machine first.

I've tried it on four machines with vista, xp and windows 7, all with similar performance, maybe they are all borked. If there is something in the default setup of calibre I've got wrong, can you point me in the right direction. I don't think it's the internet connection as I can look up the info manually pretty quickly (by manual standards), but I do connect via a local router so if there is a particular port (TCP/UDP) I need to open can you let me know (although it does seem to get there eventually).

kovidgoyal · 08-26-2010, 10:59 PM

no calibre uses standard http requests for metadata download.

DoctorOhh · 08-26-2010, 11:55 PM

Quote:

Originally Posted by lemonboy

I've tried it on four machines with vista, xp and windows 7, all with similar performance, maybe they are all borked.

The odds of all 4 being "borked" is remote, as long as IE connects OK then calibre should connect fine too.

Starson17 · 08-27-2010, 08:39 AM

Quote:

Originally Posted by dwanthny

The odds of all 4 being "borked" is remote, as long as IE connects OK then calibre should connect fine too.

I don't see a confirmation that any of the 4 machines have been tested with Internet Explorer. If it's the proxy problem suggested by Kovid I suppose they could all have been infected the same way.

DoctorOhh · 08-27-2010, 04:03 PM

Quote:

Originally Posted by Starson17

I don't see a confirmation that any of the 4 machines have been tested with Internet Explorer. If it's the proxy problem suggested by Kovid I suppose they could all have been infected the same way.

I agree with you that's why I brought up IE as a verification point. I thought he might try it and return the results here.

08-26-2010, 08:52 PM	#1
lemonboy Junior Member Posts: 3 Karma: 10 Join Date: Aug 2010 Device: none	Why so slow on metadata I have the 0.7.15 calibre with a load of lit files and windows vista. If I try to load metadata from the web, it takes an inordinate amount of time. I have an isbndb account. I can take the title and search on amazon and find the isbn before calibre decides it cant find anything. When I then put in the isbn found from amazon it takes about 2 mins to find the metadata. Why does this take so long, if I try to search for multiple files the problem is multiplied and effectively makes the multiple lookup function impossible. When carrying out a multiple search, the gui only tells you how many files are being searched, it gives no indication of progress, therefore if it is locked up you do not know. Is the lookup using some form of scraping on Amazon, if so what code is it using and where is the function (in what language). I'm sure this can be much more effective - as I said earlier, I can actually do this quicker with a manual search in amazon. If someone can tell me where to look in the code, I can surely try to make this more efficient.

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
PRS-600 Slow-down	lininjim	Sony Reader	3	07-06-2010 09:13 AM
Get Well Slow	neilmarr	General Discussions	46	04-12-2010 08:27 PM
Slow to use?	Mac Sidewinder	Sigil	9	08-07-2009 07:48 PM
calibre - very slow conversion, very slow on PRS	cremofix	Calibre	3	06-10-2009 04:21 PM
Why is it so slow?	jlc	Sony Reader	13	06-11-2007 09:50 AM

08-26-2010, 09:01 PM	#2
kovidgoyal creator of calibre Posts: 43,826 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	probably because you have a proxy setup

08-26-2010, 09:05 PM	#3
lemonboy Junior Member Posts: 3 Karma: 10 Join Date: Aug 2010 Device: none	No, no proxy. Direct internet connection. As I said I can do a manual amazon search quicker - to at least find the isbn Thanks for the quick response.

08-26-2010, 09:26 PM	#5
kovidgoyal creator of calibre Posts: 43,826 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	are you sure you have no proxy? calibre reads proxy settings from the registry (the same settings that are used by internet explorer) if you dont use internet explorer you may never realize you have the proxy settings set. Malware in windows often sets proxy settings to compromise your machine.

08-26-2010, 10:59 PM	#7
kovidgoyal creator of calibre Posts: 43,826 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	no calibre uses standard http requests for metadata download.

Advert

Advert