Possible bug causing Amazon captcha?

[eschwartz]

You can change whatever you want using the directions here: https://manual.calibre-ebook.com/develop.html

[SteveB23]

The problem went away for me when I upgraded to 2.63. However, I've only downloaded metadata for about a dozen books since then, so maybe there's a limit that I haven't hit yet.

[Oddball]

After updating to 2.7.1 i started getting the Amazon captcha error too, can only do 50 at a time and have to wait 8 hours.

[kovidgoyal]

There were no changes to amazon metadata download in 2.71. Indeed the last change (apart froma trivial removal of blank lines) to it was on Aug 28. https://github.com/kovidgoyal/calibr...rces/amazon.py

[AnselmD]

1. I run into this issue at my ubuntu 16.04 LTS 64bit.
2. After that i used my Windows 7 32bit and it works correctly.
3. Switching to my my Windows 7 64bit (32bit calibre) i am getting the error too. Changing to calibre 64bit gives the same error.


Internet environment is ip6 / ip4 over ds-lite

Any idea why it does work with my windows 7 32bit?

[AnselmD]

The windows 7 32bit was configured to use US Amazon, switching to de and i have the same trouble.

If this difficulty is bound to IP Address, i will have trouble, because my ip6 address is bound for a long time.

Does anyone have tried to workaround with proxy?

[Oddball]

i have the current version 2.77, did 125, today (10 hours later could not do one). but when i switched amazon uk ..no error. getting errors with ISBNDB also. i wonder where that info is kept at. They have to be tracking somehow. USUALLY, once i got the error i would wait 7 or 8 hours and could go do more. I find that interesting AnselmD that it worked under win7 32 but not under win7x64. it SHOULDNT be trouble with ip, since my ip hasnt changed in 6 months.

[sherman]

I started having issues with this a few weeks ago.

What seemed to 'solve' it for me was to remove any amazon identifiers (such as asin etc) from the 'Ids' field before conducting a metadata search.

[AnselmD]

Originally Posted by Oddball

i have the current version 2.77, did 125, today (10 hours later could not do one). but when i switched amazon uk ..no error. getting errors with ISBNDB also. i wonder where that info is kept at. They have to be tracking somehow. USUALLY, once i got the error i would wait 7 or 8 hours and could go do more. I find that interesting AnselmD that it worked under win7 32 but not under win7x64. it SHOULDNT be trouble with ip, since my ip hasnt changed in 6 months.

At my 32bit i wrongly used amazon_uk, therefore i did not get an error.

If i get the error, i (sometimes) can open the IE at my windows PC and it ask me for the Captures and if i solved them, the errors within calibre (which is meanwhile running at my ubuntu PC) vanished. With Firefox and Chrome i do not get the capture question, i think it is because calibre uses an IE UA string (see Change UA string for amazon metadata download to IE 11 on win 7 · kovidgoyal/calibre@e73972e · GitHub
https://github.com/kovidgoyal/calibr...5202bf567d0abc)

Therefore i think switching between different UA strings can help. Personally i would prefer not to have an IE UA at my ubuntu PC, because i am not able to solve the Captures there.
I would like to have some wait time or random-wait time (like in wget) to be more amazon friendly. Maybe an additional option to wait some more time after a fixed number of books.

On the other side rebooting my router/cable modem and getting a new IPv4 address helps too. I would like to pause the metadata batch download within calibre, in the meanwhile, but this is not possible. Aborting it does not help, because calibre abort the complete job and not only the downloading part.

[kovidgoyal]

Just use a user agent switcher for firefox/chrome to send an IE user agent for amaon. But generally speaking, given that amazon is doing bot detection, there is going to be no robust way to use amazon for bulk metadata downloads. Every trick calibre implements to try to get around it can be countered leading to a never ending arms race.

[AnselmD]

Originally Posted by kovidgoyal

Just use a user agent switcher for firefox/chrome to send an IE user agent for amaon. But generally speaking, given that amazon is doing bot detection, there is going to be no robust way to use amazon for bulk metadata downloads.

Originally Posted by kovidgoyal

Every trick calibre implements to try to get around it can be countered leading to a never ending arms race.

I agree, but nevertheless it is only fair not to make bulk metadata download with full speed. So a general way to slow down the access of every metadata downloader plugin makes it "fair". As the description of wget says: "lightens the server load by making the requests less frequent."

[kovidgoyal]

bulk metadata download already is throttled to reduce server load.

[TSRV]

I am getting the Amazon Captcha problem as well. Running version 2.79.0.

A day ago I fetched metadata for a whole bunch of books (maybe 300 or so), but after that none of the books I fetched would return metadata. I didn't fetch anything for ~16 hours and it still doesn't work.

I changed the setting in the preferences of which Amazon website to use (tried various other countries); and it still shows up in the log that is provided as Amazon.com not working.

The same book fetched at Amazon.com in my web browser works fine.

This is really quite annoying because I had never gotten this error before, and now it looks like I can't fetch any metadata anymore. I tried the other sources in the metadata preferences but none of them actually work for the books I am searching for (these are not obscure books).

****************************** Amazon.com ******************************
Request extra headers: [('User-agent', u'Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko')]
Found 0 results
Downloading from Amazon.com took 1.10199999809
Plugin Amazon.com failed
Traceback (most recent call last):
File "site-packages\calibre\ebooks\metadata\sources\identify. py", line 48, in run
File "site-packages\calibre\ebooks\metadata\sources\amazon.py ", line 1163, in identify
File "site-packages\calibre\ebooks\metadata\sources\amazon.py ", line 1073, in parse_results_page
CaptchaError: Amazon returned a CAPTCHA page, probably because you downloaded too many books. Wait for some time and try again.

************************************************** ******************************

****************************** Google ******************************
Request extra headers: [('User-agent', u'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36')]
Found 0 results
Downloading from Google took 1.4279999733

************************************************** ******************************
The identify phase took 1.63 seconds
The longest time (1.428000) was taken by: Google
Merging results from different sources and finding earliest publication dates from the worldcat.org service
We have 0 merged results, merging took: 0.00 seconds

[SteveB23]

As of yesterday afternoon (>24 hours) I am completely unable to get ANY Amazon metadata, whether it's an AZW3 book I bought at Amazon or an E-PUB I've downloaded (but which I know is sold by Amazon). I used to be able to sneak around the CAPTCHA by replacing the "mobi-asin:" in the ID field with "amazon:" but even that no longer works. Running Calibre 2.79 on Windows 10.

[PeterT]

Just wondering why you think it's a calibre bug, when it's the website (Amazon) deciding to requiring a captcha to validate a human is accessing the data as opposed to an automated process?