Calibre: download metada result in hung worker

[Maverynthia]

I was getting some metadata and though it was because i had 136 books queued, but I took it down to 3 and it still got hung. It still works for an individual book and it would sometimes work for other clusters of books. (I was getting both covers and metadata.)

Starting job: Download metadata for 3 books
Job: "Download metadata for 3 books" failed with error:
Traceback (most recent call last):
File "site-packages\calibre\gui2\threaded_jobs.py", line 83, in start_work
File "site-packages\calibre\gui2\metadata\bulk_download.py", line 236, in download
File "site-packages\calibre\utils\ipc\simple_worker.py", line 160, in fork_job
File "site-packages\calibre\utils\ipc\simple_worker.py", line 68, in communicate
WorkerError: Worker appears to have hung

Called with args: ([786, 782, 783], u'S:\\TEMP\\calibre_qrk2sj\\jc_edd_metadata_bulk.l og', <calibre.db.legacy.LibraryDatabase object at 0x0483D5F0>, True, True, None) {u'notifications': <Queue.Queue instance at 0x08FCA850>, u'abort': <threading._Event object at 0x08EF8C30>, u'log': <calibre.utils.logging.GUILog object at 0x08EF8110>}

[kovidgoyal]

Try disabling your antivirus, antivirus programs often think that a program that is connecting to lots of websites, like the calibre download worker is malicious

[Maverynthia]

Don't have any antivirus. So it can't be that. It was working yesterday on all the MOBIs I was doing from an Amazon back up. These are all EPUBs.

[kovidgoyal]

Well something on your computer is causing the process to hang, if it isn't the antivirus, then it is likely some other security software.

[Maverynthia]

I have no other security software on my computer mainly BECAUSE it's so intrusive and doesn't allow things to work right.
I found further information, it seems to have hung after 32% and I am guessing it was on this book:


****************************** Amazon.com Covers ******************************
Request extra headers: [('User-agent', 'Mozilla/5.0 (Windows NT 6.2; rv:9.0.1) Gecko/20100101 Firefox/9.0.1')]
Failed to download valid cover
Took 2.6400001049 seconds
No cached cover found, running identify
Failed to make details query: '/Dark-Earth-Jason-Halstead/dp/1451572964/ref=sr_1_1?ie=UTF8&qid=1380082285&sr=8-1'
Traceback (most recent call last):
File "site-packages\calibre\ebooks\metadata\sources\amazon.py ", line 226, in get_details
File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_mechanize.py", line 199, in open_novisit
File "site-packages\mechanize-0.2.4-py2.7.egg\mechanize\_mechanize.py", line 216, in _mech_open
BrowserStateError: can't fetch relative reference: not viewing any document

No cover found

************************************************** ******************************

****************************** Open Library Covers ******************************
Request extra headers: [('User-agent', 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)')]
Failed to download valid cover
Took 0.483999967575 seconds
No cover for ISBN: '9781451572964' found

************************************************** ******************************

It seems to hang when it can find no "valid" covers. I processed it manually and it got the metadata fine, however the ONLY covers it could find were "Google Image Search" covers. No others, possibly causing the open library and/or Amazon cover search to shrug and go "what do I do now?"

I also ran a Metadata only and it processed correctly out at 10 minutes:
The identify phase took 8.72 seconds
The longest time (8.485000) was taken by: Google
Merging results from different sources and finding earliest publication dates from the worldcat.org service
We have 9 merged results, merging took: 2.48 seconds


So it's the covers that are causing Calibre to hang. I want to say it's Amazon but I shut it off and it still hangs.

[Sabardeyn]

Bit of a wild guess here...

Cannot recall which topic it was exactly ATM, but in one of the plugin topics (I think) in the last couple of days, there was a discussion that came down to Amazon having changed their website, thus disrupting the plugin. Perhaps the change has also interfered with calibre's built-in functionality?

(Of course, now that I'm trying to find the topic, I cannot locate it again.)

[BetterRed]

Originally Posted by Sabardeyn

Bit of a wild guess here...

Cannot recall which topic it was exactly ATM, but in one of the plugin topics (I think) in the last couple of days, there was a discussion that came down to Amazon having changed their website, thus disrupting the plugin. Perhaps the change has also interfered with calibre's built-in functionality?

(Of course, now that I'm trying to find the topic, I cannot locate it again.)

this one maybe - https://www.mobileread.com/forums/sho...ghlight=amazon

BR

[theducks]

Originally Posted by Sabardeyn

Bit of a wild guess here...

Cannot recall which topic it was exactly ATM, but in one of the plugin topics (I think) in the last couple of days, there was a discussion that came down to Amazon having changed their website, thus disrupting the plugin. Perhaps the change has also interfered with calibre's built-in functionality?

(Of course, now that I'm trying to find the topic, I cannot locate it again.)

The (expired) timeout should have returned a not found.
That is if a reasonable timeout value was still used (? Preferences: Behavior: default network timeout: 5 seconds)

[kovidgoyal]

The only way a hang can happen is if some metadata download plugin is hanging and no other plugin is finding covers. I suggest you disable/uninstall any third party metadata download plugins you have, as far I recall, all the builtin plugins use timeouts on network requests, so they cannot hang indefinitely, cant speak for third party plugins.

And update your calibre install to be able to get metadata from amazon again, as pointed out earlier in this thread.

[Maverynthia]

I found out the rouge plugin. It was Google Image search. It seems to now toss up EOFs:

calibre, version 1.4.0
ERROR: Download failed: Failed to download any covers, click "Show details" for details.

Traceback (most recent call last):
File "site-packages\calibre\utils\ipc\simple_worker.py", line 50, in run
File "site-packages\calibre\utils\ipc\__init__.py", line 19, in eintr_retry_call
EOFError


I disabled the Google Image plugin and all the other ones (Amazon, B&N, Goodreads, etc.) went fine. No problems.
I enabled ONLY the Google Image plugin and got the same EOF/hung response. I even took the returned images down from 5 to 3 to 1 and it still all hung up. Which sucks since Google Images is the one that returns the nice ~1000+ res covers.
I have a feeling it wasn't hanging before on the MOBIs as they were all from Amazon and thus Amazon would at least toss a cover out. On these ePubs, some of them had no metadata OR relevant/valid covers at all other than what the EPUB provided and thus Google Image search was left by it's lonesome or something shrugging it's shoulders looking around for an adult to solve the problem.

Google Image even choked on the single metadata "Download Cover", but it just tossed up and error and no images other than the current were provided.

[Sabardeyn]

Originally Posted by BetterRed

this one maybe - https://www.mobileread.com/forums/sho...ghlight=amazon

BR

Yes, thank you, that was the discussion I was trying to reference but could not re-located. While it isn't applicable here, it does seem to have helped us find the culprit.


Maverynthia,
Huh that is kind of odd because I was just using the Search the Internet plugin to look up a few covers yesterday. That plugin searched Google for images without issue so the two plugins must submit different requests to Google. I have not previously used Google Image Search.


Theducks,
The default calibre values listed under Metadata Download are:
Max time to wait after first match is found: 30
Max time to wait after first cover is found: 60
Which exceed your 5 sec suggestion. So unless the timing under Behaviour has been altered (decreased) they should suffice.


Kovid,
I just (re-)noticed that once you set a cover there is no way to delete it, returning to a null state. Can a "Delete Cover" function be added to the context menu of the cover image portion of the Edit Metadata dialog? Or is there some other method to return to a cover=null state?

I know on a few occasions I've discovered I've made an error with the cover image (different ISBN's cover usually) and I would prefer to have no image rather than a wrong image. Searching for cover:false is possible, cover:inappropriate is not.

[BetterRed]

Originally Posted by Sabardeyn

Can a "Delete Cover" function be added to the context menu of the cover image portion of the Edit Metadata dialog? Or is there some other method to return to a cover=null state?

These don't work for you ?

BR

[theducks]

@Sabardeyn
I was referring to the overall network setting

I did not notice the ones you mentioned

[Sabardeyn]

Originally Posted by BetterRed

These don't work for you ?

No, when I don't press them, they don't work for me!

I've been mentally screening that entire group of buttons out from the Edit Metadata dialog. I think that goes back to when Download Metadata was first introduced; getting bombarded with all those (frequently inappropriate) covers, tags, etc just made me blank the buttons out. Too much extra editing and re-editing required!

Don't think I ever knew you could delete the cover from the Book Details though.

Thanks for letting me know.

[kovidgoyal]

@Maverynthia: Google Images works fine for me. It is the most heavyweight plugin, since it uses a complete browser instance, which runs in a separate worker process, the error that you posted means the worker process is not responding. The only way I know of for that to happen is an antivirus to block the process for suspicious activity, other than that, I'm out of ideas. THe code itself times out if google does not respond: https://github.com/kovidgoyal/calibr..._images.py#L95