Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 07-19-2016, 12:19 PM   #1
blitzker
Junior Member
blitzker began at the beginning.
 
Posts: 5
Karma: 10
Join Date: Jul 2016
Device: kindle
Possible bug causing Amazon captcha?

Hey everyone. I recently downloaded Calibre 2.62. Within a day or so, I started getting the following captcha error when downloading metadata from Amazon:

Quote:
Found 0 results
Downloading from Amazon.com took 19.1949999332
get_details failed for url: [REDACTED]
Traceback (most recent call last):
File "site-packages\calibre\ebooks\metadata\sources\amazon.py ", line 288, in run
File "site-packages\calibre\ebooks\metadata\sources\amazon.py ", line 301, in get_details
File "site-packages\calibre\ebooks\metadata\sources\amazon.py ", line 306, in parse_details
CaptchaError: Amazon returned a CAPTCHA page, probably because you downloaded too many books. Wait for some time and try again.
I'm not doing anything different than what I've always done (e.g. making excessive queries), so I was wondering if it might be an issue with Calibre. One thing I notice is that in the "request extra headers" portion of the log, the user-agent is now always the same. In older versions, didn't the user-agent randomly change every time a query is made? Is it possible that Amazon is throwing a captcha because it's getting too many requests with the same user-agent?

Thank you!
blitzker is offline   Reply With Quote
Advert
Old 07-19-2016, 12:41 PM   #2
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 20,470
Karma: 20259386
Join Date: Aug 2009
Location: (The original) Silicon Valley, USA
Device: Astak Pocket Pro, K4NT,Galaxy Tab 2
did you tamper with any 'tweaks' that override the deliberately slowed down requests?

Been using Calibre for years and never triggered that. Even when I first started by documenting (empty books) my 900+ paper library, Baen CD's (after importing PRC's)
theducks is offline   Reply With Quote
Old 07-19-2016, 12:45 PM   #3
blitzker
Junior Member
blitzker began at the beginning.
 
Posts: 5
Karma: 10
Join Date: Jul 2016
Device: kindle
No, I haven't tampered with anything. Don't you think it's strange that the user-agent is the same with each request? I think that could definitely throw some flags.
blitzker is offline   Reply With Quote
Old 07-19-2016, 01:34 PM   #4
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 30,865
Karma: 7728435
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
I deliberately made the user agent the same because with random user agents captcha pages were becoming far more common (nowadays it is hard to come up with many common stable user agents, given evergreen browsers).

But, if you want to go back to random user agents, it's a two line change running calibre from source. Change line 804 in amazon.py to return whatever you like.
kovidgoyal is offline   Reply With Quote
Old 07-19-2016, 01:49 PM   #5
SteveB23
Junior Member
SteveB23 began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Jul 2016
Device: Kindle Apps (Desktop, Android phone & tablet)
I'm seeing exactly the same error. It started a couple of days ago, and Amazon won't allow me to query either book metadata or cover images. Like blitzker, I have not made any tweaks to my Calibre config.

Quote:
I deliberately made the user agent the same because with random user agents captcha pages were becoming far more common (nowadays it is hard to come up with many common stable user agents, given evergreen browsers).
Was this a recent change?

Quote:
But, if you want to go back to random user agents, it's a two line change running calibre from source. Change line 804 in amazon.py to return whatever you like.
Is this a configuration or script change I can make locally, or does it have to be done at the source level?

Thanks!
SteveB23 is offline   Reply With Quote
Advert
Old 07-19-2016, 01:58 PM   #6
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 30,865
Karma: 7728435
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
It was over a month ago, https://github.com/kovidgoyal/calibr...df3034752468e8

And no it has to be a source level change. Though I'm pretty sure that you will find going back to random UAs will not help (it might help for a little while, but then you will be back to getting captcha pages).
kovidgoyal is offline   Reply With Quote
Old 07-19-2016, 02:06 PM   #7
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 30,865
Karma: 7728435
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
I changed it to a more common IE 11 variant, that might help: https://github.com/kovidgoyal/calibr...5202bf567d0abc
kovidgoyal is offline   Reply With Quote
Old 07-19-2016, 02:10 PM   #8
blitzker
Junior Member
blitzker began at the beginning.
 
Posts: 5
Karma: 10
Join Date: Jul 2016
Device: kindle
Thanks, Kovid. Would it be possible to give users the option to solve the captcha when Amazon presents one?
blitzker is offline   Reply With Quote
Old 07-19-2016, 02:12 PM   #9
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 30,865
Karma: 7728435
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
No, way too much work.
kovidgoyal is offline   Reply With Quote
Old 07-19-2016, 02:39 PM   #10
blitzker
Junior Member
blitzker began at the beginning.
 
Posts: 5
Karma: 10
Join Date: Jul 2016
Device: kindle
Okay, I just reverted to Calibre 2.58. I'm no longer getting a captcha error.
blitzker is offline   Reply With Quote
Old 07-20-2016, 07:51 PM   #11
AnthonyBe
Junior Member
AnthonyBe began at the beginning.
 
Posts: 5
Karma: 10
Join Date: Oct 2013
Device: Surface RT, Lumia 930
Hmmm... was wondering why all my metadata searches today were only returning results from Google and none from Amazon.

Am using Calibre 2.62
AnthonyBe is offline   Reply With Quote
Old 07-21-2016, 08:21 AM   #12
AnthonyBe
Junior Member
AnthonyBe began at the beginning.
 
Posts: 5
Karma: 10
Join Date: Oct 2013
Device: Surface RT, Lumia 930
How about breaking the UA out as a variable that can be over-ridden in the Calibre preferences?

Yes, not a permanent solution, but at least when it does happen, it allows advanced user to make a change that will provide some relief.
AnthonyBe is offline   Reply With Quote
Old 07-21-2016, 08:41 AM   #13
amazon author
Cracking Coding Interview
amazon author began at the beginning.
 
amazon author's Avatar
 
Posts: 33
Karma: 10
Join Date: Mar 2015
Device: iPhone
Upgrade to latest calibre.
amazon author is offline   Reply With Quote
Old 07-21-2016, 10:01 AM   #14
blitzker
Junior Member
blitzker began at the beginning.
 
Posts: 5
Karma: 10
Join Date: Jul 2016
Device: kindle
Quote:
Originally Posted by AnthonyBe View Post
How about breaking the UA out as a variable that can be over-ridden in the Calibre preferences?
Better yet: a setting to re-enable random UAs. I suspect any solution involving single UAs is ultimately going to be problematic.
blitzker is offline   Reply With Quote
Old 07-22-2016, 05:10 PM   #15
Quella
Junior Member
Quella began at the beginning.
 
Posts: 1
Karma: 10
Join Date: Jul 2016
Device: Kindle Paperwhite
I am also experiencing the same issue with Amazon Meta Data or Covers with a Captcha error when I look at the details. When I downgraded to 2.58 as suggested above I was also able to do 40+ book without issue. Is there a way to upgrade to the latest and bring over the amazon.py portion from the 2.58 codebase and apply it to the 2.62/63 version? Maybe a quick fix until a more permanent solution is found?
Quella is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Aussies launch anti-CAPTCHA petition. Is it time to kill CAPTCHAs? Alexander Turcic Lounge 30 08-16-2013 07:32 PM
Bug in Kobo processing of epub files causing hang in "Processing content" BensonBear Kobo Reader 21 12-21-2012 05:47 AM
Get books bug for Amazon UK rustleg Library Management 1 10-21-2012 01:25 PM


All times are GMT -4. The time now is 06:39 AM.


MobileRead.com is a privately owned, operated and funded community.