[Metadata Source Plugin] Amazon.com not fetching all descriptions

StillReading · 01-09-2016, 07:21 AM

I've noticed that "Download metadata" in Calibre 2.48.0 Windows 64-bit doesn't work like in previous versions. I can't tell with which version the problem started.

Effect: "Download metadata" isn't downloading the complete description text. The description texts on Amazon are divided into three categories: "Short Description", "Review", and "About the Author". In previous versions, Calibre would fetch all three texts. Now it only fetches the text under "Short Description". This happens with the pugin set to "English" and "German". Examples are Abaddon's Gate: Book 3 of the Expanse (English, ISBN 9780748122981) and 23 Lügen, die sie uns über den Kapitalismus erzählen (German, ISBN 9783641041403). The plugin was set to the book's language (country) for each test.

A fix would be much appreciated, as copying/pasting and manually editing the descriptions is very time consuming (; .

kovidgoyal · 01-09-2016, 08:31 AM

Amazon are now putting that content into a iframe that is populated dynamically via javascript. The silly part is that the actual content is present as a URL-encoded javascript variable in a script tag.

https://github.com/kovidgoyal/calibr...8ba0cf5c5bf828

StillReading · 01-09-2016, 08:41 AM

Wow, thanks for the quick fix! Looking forward to the new version.

In addition: would you be willing or would it be possible/feasible to implement some additional parsing? One small "error" that crops up often with Amazon descriptions (and that I'm correcting manually) is spaces before end tags (like "<p>BlaBlaBla </p>). It would be nice if Calibre could get rid of these, but I can live with manual editing of course.

kovidgoyal · 01-09-2016, 08:56 AM

Sorry, I'm not willing to take on responsibility for cleaning amazon's markup, that way lies madness.

StillReading · 01-09-2016, 09:02 AM

Yeah, I can see what you mean by that (; . Not a big deal, I could probably even (half-) automate my comment editing with a RegEx search/replace operation. Thanks again for fixing the original problem.

01-09-2016, 07:21 AM	#1
StillReading Enthusiast Posts: 30 Karma: 10 Join Date: Sep 2014 Device: none	[Metadata Source Plugin] Amazon.com not fetching all description texts I've noticed that "Download metadata" in Calibre 2.48.0 Windows 64-bit doesn't work like in previous versions. I can't tell with which version the problem started. Effect: "Download metadata" isn't downloading the complete description text. The description texts on Amazon are divided into three categories: "Short Description", "Review", and "About the Author". In previous versions, Calibre would fetch all three texts. Now it only fetches the text under "Short Description". This happens with the pugin set to "English" and "German". Examples are Abaddon's Gate: Book 3 of the Expanse (English, ISBN 9780748122981) and 23 Lügen, die sie uns über den Kapitalismus erzählen (German, ISBN 9783641041403). The plugin was set to the book's language (country) for each test. A fix would be much appreciated, as copying/pasting and manually editing the descriptions is very time consuming (; . Last edited by StillReading; 01-09-2016 at 07:27 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[Metadata Source Plugin] Skoob	rodrigoccurvo	Plugins	11	06-13-2019 07:44 PM
[Metadata Source Plugin] Question: Amazon.com	silbaer	Calibre	13	12-31-2015 01:12 PM
[Metadata Source Plugin] Amazon.CN	fated	Plugins	0	11-20-2014 05:59 PM
Amazon Metadata source plugin not working	Stormvision	Plugins	3	05-03-2013 09:20 AM
[Metadata Source Plugin] Amazon.it	nandocuci	Plugins	2	05-18-2011 03:36 AM

01-09-2016, 08:31 AM	#2
kovidgoyal creator of calibre Posts: 45,664 Karma: 28549046 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Amazon are now putting that content into a iframe that is populated dynamically via javascript. The silly part is that the actual content is present as a URL-encoded javascript variable in a script tag. https://github.com/kovidgoyal/calibr...8ba0cf5c5bf828

01-09-2016, 08:41 AM	#3
StillReading Enthusiast Posts: 30 Karma: 10 Join Date: Sep 2014 Device: none	Wow, thanks for the quick fix! Looking forward to the new version. In addition: would you be willing or would it be possible/feasible to implement some additional parsing? One small "error" that crops up often with Amazon descriptions (and that I'm correcting manually) is spaces before end tags (like "<p>BlaBlaBla </p>). It would be nice if Calibre could get rid of these, but I can live with manual editing of course.

01-09-2016, 08:56 AM	#4
kovidgoyal creator of calibre Posts: 45,664 Karma: 28549046 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Sorry, I'm not willing to take on responsibility for cleaning amazon's markup, that way lies madness.

01-09-2016, 09:02 AM	#5
StillReading Enthusiast Posts: 30 Karma: 10 Join Date: Sep 2014 Device: none	Yeah, I can see what you mean by that (; . Not a big deal, I could probably even (half-) automate my comment editing with a RegEx search/replace operation. Thanks again for fixing the original problem.