Possible bug causing Amazon captcha?

blitzker · 07-19-2016, 01:19 PM

Hey everyone. I recently downloaded Calibre 2.62. Within a day or so, I started getting the following captcha error when downloading metadata from Amazon:

Quote:

Found 0 results
Downloading from Amazon.com took 19.1949999332
get_details failed for url: [REDACTED]
Traceback (most recent call last):
File "site-packages\calibre\ebooks\metadata\sources\amazon.py ", line 288, in run
File "site-packages\calibre\ebooks\metadata\sources\amazon.py ", line 301, in get_details
File "site-packages\calibre\ebooks\metadata\sources\amazon.py ", line 306, in parse_details
CaptchaError: Amazon returned a CAPTCHA page, probably because you downloaded too many books. Wait for some time and try again.

I'm not doing anything different than what I've always done (e.g. making excessive queries), so I was wondering if it might be an issue with Calibre. One thing I notice is that in the "request extra headers" portion of the log, the user-agent is now always the same. In older versions, didn't the user-agent randomly change every time a query is made? Is it possible that Amazon is throwing a captcha because it's getting too many requests with the same user-agent?

Thank you!

theducks · 07-19-2016, 01:41 PM

did you tamper with any 'tweaks' that override the deliberately slowed down requests?

Been using Calibre for years and never triggered that. Even when I first started by documenting (empty books) my 900+ paper library, Baen CD's (after importing PRC's)

blitzker · 07-19-2016, 01:45 PM

No, I haven't tampered with anything. Don't you think it's strange that the user-agent is the same with each request? I think that could definitely throw some flags.

kovidgoyal · 07-19-2016, 02:34 PM

I deliberately made the user agent the same because with random user agents captcha pages were becoming far more common (nowadays it is hard to come up with many common stable user agents, given evergreen browsers).

But, if you want to go back to random user agents, it's a two line change running calibre from source. Change line 804 in amazon.py to return whatever you like.

SteveB23 · 07-19-2016, 02:49 PM

I'm seeing exactly the same error. It started a couple of days ago, and Amazon won't allow me to query either book metadata or cover images. Like blitzker, I have not made any tweaks to my Calibre config.

Quote:

I deliberately made the user agent the same because with random user agents captcha pages were becoming far more common (nowadays it is hard to come up with many common stable user agents, given evergreen browsers).

Was this a recent change?

Quote:

But, if you want to go back to random user agents, it's a two line change running calibre from source. Change line 804 in amazon.py to return whatever you like.

Is this a configuration or script change I can make locally, or does it have to be done at the source level?

Thanks!

kovidgoyal · 07-19-2016, 02:58 PM

It was over a month ago, https://github.com/kovidgoyal/calibr...df3034752468e8

And no it has to be a source level change. Though I'm pretty sure that you will find going back to random UAs will not help (it might help for a little while, but then you will be back to getting captcha pages).

kovidgoyal · 07-19-2016, 03:06 PM

I changed it to a more common IE 11 variant, that might help: https://github.com/kovidgoyal/calibr...5202bf567d0abc

blitzker · 07-19-2016, 03:10 PM

Thanks, Kovid. Would it be possible to give users the option to solve the captcha when Amazon presents one?

kovidgoyal · 07-19-2016, 03:12 PM

No, way too much work.

blitzker · 07-19-2016, 03:39 PM

Okay, I just reverted to Calibre 2.58. I'm no longer getting a captcha error.

AnthonyBe · 07-20-2016, 08:51 PM

Hmmm... was wondering why all my metadata searches today were only returning results from Google and none from Amazon.

Am using Calibre 2.62

AnthonyBe · 07-21-2016, 09:21 AM

How about breaking the UA out as a variable that can be over-ridden in the Calibre preferences?

Yes, not a permanent solution, but at least when it does happen, it allows advanced user to make a change that will provide some relief.

amazon author · 07-21-2016, 09:41 AM

Upgrade to latest calibre.

blitzker · 07-21-2016, 11:01 AM

Quote:

Originally Posted by AnthonyBe

How about breaking the UA out as a variable that can be over-ridden in the Calibre preferences?

Better yet: a setting to re-enable random UAs. I suspect any solution involving single UAs is ultimately going to be problematic.

Quella · 07-22-2016, 06:10 PM

I am also experiencing the same issue with Amazon Meta Data or Covers with a Captcha error when I look at the details. When I downgraded to 2.58 as suggested above I was also able to do 40+ book without issue. Is there a way to upgrade to the latest and bring over the amazon.py portion from the 2.58 codebase and apply it to the 2.62/63 version? Maybe a quick fix until a more permanent solution is found?

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Aussies launch anti-CAPTCHA petition. Is it time to kill CAPTCHAs?	Alexander Turcic	Lounge	30	08-16-2013 08:32 PM
Bug in Kobo processing of epub files causing hang in "Processing content"	BensonBear	Kobo Reader	21	12-21-2012 06:47 AM
Get books bug for Amazon UK	rustleg	Library Management	1	10-21-2012 02:25 PM

07-19-2016, 01:41 PM	#2
theducks Well trained by Cats Posts: 31,267 Karma: 61916422 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	did you tamper with any 'tweaks' that override the deliberately slowed down requests? Been using Calibre for years and never triggered that. Even when I first started by documenting (empty books) my 900+ paper library, Baen CD's (after importing PRC's)

07-19-2016, 01:45 PM	#3
blitzker Junior Member Posts: 5 Karma: 10 Join Date: Jul 2016 Device: kindle	No, I haven't tampered with anything. Don't you think it's strange that the user-agent is the same with each request? I think that could definitely throw some flags.

07-19-2016, 02:34 PM	#4
kovidgoyal creator of calibre Posts: 45,609 Karma: 28549044 Join Date: Oct 2006 Location: Mumbai, India Device: Various	I deliberately made the user agent the same because with random user agents captcha pages were becoming far more common (nowadays it is hard to come up with many common stable user agents, given evergreen browsers). But, if you want to go back to random user agents, it's a two line change running calibre from source. Change line 804 in amazon.py to return whatever you like.

07-19-2016, 02:58 PM	#6
kovidgoyal creator of calibre Posts: 45,609 Karma: 28549044 Join Date: Oct 2006 Location: Mumbai, India Device: Various	It was over a month ago, https://github.com/kovidgoyal/calibr...df3034752468e8 And no it has to be a source level change. Though I'm pretty sure that you will find going back to random UAs will not help (it might help for a little while, but then you will be back to getting captcha pages).

07-19-2016, 03:06 PM	#7
kovidgoyal creator of calibre Posts: 45,609 Karma: 28549044 Join Date: Oct 2006 Location: Mumbai, India Device: Various	I changed it to a more common IE 11 variant, that might help: https://github.com/kovidgoyal/calibr...5202bf567d0abc

07-19-2016, 03:10 PM	#8
blitzker Junior Member Posts: 5 Karma: 10 Join Date: Jul 2016 Device: kindle	Thanks, Kovid. Would it be possible to give users the option to solve the captcha when Amazon presents one?

07-19-2016, 03:12 PM	#9
kovidgoyal creator of calibre Posts: 45,609 Karma: 28549044 Join Date: Oct 2006 Location: Mumbai, India Device: Various	No, way too much work.

07-19-2016, 03:39 PM	#10
blitzker Junior Member Posts: 5 Karma: 10 Join Date: Jul 2016 Device: kindle	Okay, I just reverted to Calibre 2.58. I'm no longer getting a captcha error.

07-20-2016, 08:51 PM	#11
AnthonyBe Junior Member Posts: 6 Karma: 10 Join Date: Oct 2013 Location: Sydney, Australia Device: Kindle Voyage	Hmmm... was wondering why all my metadata searches today were only returning results from Google and none from Amazon. Am using Calibre 2.62

07-21-2016, 09:21 AM	#12
AnthonyBe Junior Member Posts: 6 Karma: 10 Join Date: Oct 2013 Location: Sydney, Australia Device: Kindle Voyage	How about breaking the UA out as a variable that can be over-ridden in the Calibre preferences? Yes, not a permanent solution, but at least when it does happen, it allows advanced user to make a change that will provide some relief.

07-21-2016, 09:41 AM	#13
amazon author Cracking Coding Interview Posts: 33 Karma: 10 Join Date: Mar 2015 Device: iPhone	Upgrade to latest calibre.

07-22-2016, 06:10 PM	#15
Quella Junior Member Posts: 1 Karma: 10 Join Date: Jul 2016 Device: Kindle Paperwhite	I am also experiencing the same issue with Amazon Meta Data or Covers with a Captcha error when I look at the details. When I downgraded to 2.58 as suggested above I was also able to do 40+ book without issue. Is there a way to upgrade to the latest and bring over the amazon.py portion from the 2.58 codebase and apply it to the 2.62/63 version? Maybe a quick fix until a more permanent solution is found?