Problem: download metadata from german Amazon site

[ratanplan]

Since 2.38 (maybe even before, but I noticed with 2.38) calibre doesn't find the metadata-entries from the (german) Amazon.de-Site (I've checked: the book is there). I didn't change any settings. Did calibre change in any way, so that I have to change some settings for the metadata download from the german amazon-website? (I find the results from this site very useful, because the metadata are often complete. Normally I use entries in title and/or author for the search.)

Edit: oops, Didn't want to post under "editor". Please mods move to the right place.

[kovidgoyal]

You have the amazon plugin configured to download from amazon.de, not amazon.com?

[ratanplan]

Yes, I did (just made a control, before this answer).

[kovidgoyal]

Then make sure your books dont have an identifier that looks like amazon:something or asin:something. Change it to amazon_de:something or delete it.

[ratanplan]

There were no identifiers in each book. This field was empty at every time, I tried to download metadata.
And I got results, but now every time only from google and the isbn and not from amazon_de (in previous versions I got often results from all three).

[kovidgoyal]

Then you need to provide the log from the download process. All the book I tested with amazon.de work fine.

[ratanplan]

Here's the log:

Spoiler:

Running identify query with parameters:
{u'authors': [u'Arto Paasilinna'], u'timeout': 30, u'title': u'Zehn z\xe4rtliche Kratzb\xfcrsten', u'identifiers': {}}
Using plugins: Google, Amazon.com, Edelweiss, Overdrive
The log from individual plugins is below

****************************** Edelweiss ******************************
Request extra headers: [('User-agent', 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.0.249.78 Safari/532.5')]
Found 0 results
Downloading from Edelweiss took 1.94600009918
Using query URL: http://edelweiss.abovethetreeline.co...owseType=title

************************************************** ******************************

****************************** Google ******************************
Request extra headers: [('User-agent', 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.0.249.78 Safari/532.5')]
Found 1 results
Downloading from Google took 1.64899992943


---
Title : Zehn zärtliche Kratzbürsten
Author(s) : Arto Paasilinna
Publisher : BASTEI LÜBBE
Tags : Fiction, General
Languages : deu
Published : 2015-07-16T15:04:36.332000+00:00
Identifiers : isbn:9783732512706, google:U7mqCQAAQBAJ
Comments : Geschäftsführer Rauno Rämekorpi feiert seinen sechzigsten Geburtstag. Gratulanten gibt es viele, und so häufen sich Blumensträuße und Geschenke. Schade nur, dass seine Frau auf Blumen allergisch ist. Die Blütenpracht muss also weg, jedoch keinesfalls auf den Müll! So beginnt Rauno eine Fahrt durch Helsinki, und mit einem Strauß in der Hand stattet er seinen verflossenen Liebschaften einen Besuch ab ...

************************************************** ******************************

****************************** Overdrive ******************************
Request extra headers: [('User-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1')]
Found 0 results
Downloading from Overdrive took 1.69100022316
Initial query is Arto Paasilinna
Cross reference query is Kratzbürsten

************************************************** ******************************

****************************** Amazon.com ******************************
Request extra headers: [('User-agent', 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1')]
Found 0 results
Downloading from Amazon.com took 2.60599994659
Failed to make details query: 'http://www.amazon.de/Zehn-z%C3%A4rtliche-Kratzb%C3%BCrsten-Arto-Paasilinna/dp/3404922867/ref=sr_1_1/279-0869157-8612732?s=books&ie=UTF8&qid=1443107086&sr=1-1'
Traceback (most recent call last):
File "site-packages\calibre\ebooks\metadata\sources\amazon.py ", line 28, in parse_details_page
File "site-packages\mechanize-0.2.5-py2.7.egg\mechanize\_mechanize.py", line 199, in open_novisit
File "site-packages\mechanize-0.2.5-py2.7.egg\mechanize\_mechanize.py", line 230, in _mech_open
File "site-packages\mechanize-0.2.5-py2.7.egg\mechanize\_opener.py", line 193, in open
File "site-packages\mechanize-0.2.5-py2.7.egg\mechanize\_urllib2_fork.py", line 344, in _open
File "site-packages\mechanize-0.2.5-py2.7.egg\mechanize\_urllib2_fork.py", line 332, in _call_chain
File "site-packages\mechanize-0.2.5-py2.7.egg\mechanize\_urllib2_fork.py", line 1142, in http_open
File "site-packages\mechanize-0.2.5-py2.7.egg\mechanize\_urllib2_fork.py", line 1116, in do_open
File "httplib.py", line 1074, in getresponse
File "httplib.py", line 415, in begin
File "httplib.py", line 379, in _read_status
BadStatusLine: ''

get_details failed for url: 'http://www.amazon.de/Zehn-z%C3%A4rtliche-Kratzb%C3%BCrsten-Arto-Paasilinna/dp/3404922867/ref=sr_1_1/279-0869157-8612732?s=books&ie=UTF8&qid=1443107086&sr=1-1'
Traceback (most recent call last):
File "site-packages\calibre\ebooks\metadata\sources\amazon.py ", line 282, in run
File "site-packages\calibre\ebooks\metadata\sources\amazon.py ", line 289, in get_details
TypeError: 'NoneType' object is not iterable

Failed to make details query: 'http://www.amazon.de/z%C3%A4rtliche-Kratzb%C3%BCrsten-Taschenbuch-Paasilinna-Pirschel/dp/B0053TIJ74/ref=sr_1_3/279-0869157-8612732?s=books&ie=UTF8&qid=1443107086&sr=1-3'
Traceback (most recent call last):
File "site-packages\calibre\ebooks\metadata\sources\amazon.py ", line 28, in parse_details_page
File "site-packages\mechanize-0.2.5-py2.7.egg\mechanize\_mechanize.py", line 199, in open_novisit
File "site-packages\mechanize-0.2.5-py2.7.egg\mechanize\_mechanize.py", line 230, in _mech_open
File "site-packages\mechanize-0.2.5-py2.7.egg\mechanize\_opener.py", line 193, in open
File "site-packages\mechanize-0.2.5-py2.7.egg\mechanize\_urllib2_fork.py", line 344, in _open
File "site-packages\mechanize-0.2.5-py2.7.egg\mechanize\_urllib2_fork.py", line 332, in _call_chain
File "site-packages\mechanize-0.2.5-py2.7.egg\mechanize\_urllib2_fork.py", line 1142, in http_open
File "site-packages\mechanize-0.2.5-py2.7.egg\mechanize\_urllib2_fork.py", line 1116, in do_open
File "httplib.py", line 1074, in getresponse
File "httplib.py", line 415, in begin
File "httplib.py", line 379, in _read_status
BadStatusLine: ''

get_details failed for url: 'http://www.amazon.de/z%C3%A4rtliche-Kratzb%C3%BCrsten-Taschenbuch-Paasilinna-Pirschel/dp/B0053TIJ74/ref=sr_1_3/279-0869157-8612732?s=books&ie=UTF8&qid=1443107086&sr=1-3'
Traceback (most recent call last):
File "site-packages\calibre\ebooks\metadata\sources\amazon.py ", line 282, in run
File "site-packages\calibre\ebooks\metadata\sources\amazon.py ", line 289, in get_details
TypeError: 'NoneType' object is not iterable

Failed to make details query: 'http://www.amazon.de/z%C3%A4rtliche-Kratzb%C3%BCrsten-Pirschel-Regine-Paasilinna/dp/B003GQ0NP2/ref=sr_1_2/279-0869157-8612732?s=books&ie=UTF8&qid=1443107086&sr=1-2'
Traceback (most recent call last):
File "site-packages\calibre\ebooks\metadata\sources\amazon.py ", line 28, in parse_details_page
File "site-packages\mechanize-0.2.5-py2.7.egg\mechanize\_mechanize.py", line 199, in open_novisit
File "site-packages\mechanize-0.2.5-py2.7.egg\mechanize\_mechanize.py", line 230, in _mech_open
File "site-packages\mechanize-0.2.5-py2.7.egg\mechanize\_opener.py", line 193, in open
File "site-packages\mechanize-0.2.5-py2.7.egg\mechanize\_urllib2_fork.py", line 344, in _open
File "site-packages\mechanize-0.2.5-py2.7.egg\mechanize\_urllib2_fork.py", line 332, in _call_chain
File "site-packages\mechanize-0.2.5-py2.7.egg\mechanize\_urllib2_fork.py", line 1142, in http_open
File "site-packages\mechanize-0.2.5-py2.7.egg\mechanize\_urllib2_fork.py", line 1116, in do_open
File "httplib.py", line 1074, in getresponse
File "httplib.py", line 415, in begin
File "httplib.py", line 379, in _read_status
BadStatusLine: ''

get_details failed for url: 'http://www.amazon.de/z%C3%A4rtliche-Kratzb%C3%BCrsten-Pirschel-Regine-Paasilinna/dp/B003GQ0NP2/ref=sr_1_2/279-0869157-8612732?s=books&ie=UTF8&qid=1443107086&sr=1-2'
Traceback (most recent call last):
File "site-packages\calibre\ebooks\metadata\sources\amazon.py ", line 282, in run
File "site-packages\calibre\ebooks\metadata\sources\amazon.py ", line 289, in get_details
TypeError: 'NoneType' object is not iterable

************************************************** ******************************
The identify phase took 2.67 seconds
The longest time (2.606000) was taken by: Amazon.com
Merging results from different sources and finding earliest publication dates from the worldcat.org service
We have 1 merged results, merging took: 0.37 seconds

[kovidgoyal]

That log indicates the amazon servers are erroring out for you. They are returning empty http responses. Trying that tile and author works for me with amazon.de

[ratanplan]

Very strange. So I assume, I have to live with that...

[ratanplan]

Maybe one step in the direction for the solution:

Today I added a book (epub) and I got a result from amazon_de.
I looked at the url and I realised that the site had the "old" amazon style.

For you to look at it, if you want to:
Result "old style" (Amazon uses this for the Kindle Edition): http://www.amazon.de/Vollmeisen-Roma...=UTF8&qid=&sr=

"new style" (Amazon uses this for the pocketbook): http://www.amazon.de/Vollmeisen-Roma...mm_pap_title_0

Is it possible, that this is causing my problems? And if so, do you have an idea, how I can solve the problem?

Edit: the problem exists for me even in 2.39

[kovidgoyal]

There is no difference in those URLs except that they refer to different products, one is to the ebook and the other the paper book.

You need to figure out why you are getting bad server responses from amazon -- the most likely culprit is an antivirus program

[ratanplan]

Originally Posted by kovidgoyal

...the most likely culprit is an antivirus program

I will check that.

[ratanplan]

You're great. Thanks you for your advice and patience.
That's indeed the reason. If I deactivate my KIS 2016, I get results from amazon_de.
Now I have to find out, which settings I have to change in KIS 2016... (so this seems to be a new problem for me )

Edit: I found the settings in KIS 2016 (to controll programms), so now KIS 2016 should "trust" calibre in every way.