Amazon Support

[earthq]

Hi,

I have seen in closed tickets about this due to Amazon licensing terms. I guess we are dealing with this:
https://affiliate-program.amazon.com...=assoc-api-faq

In particular, it states:
"You will not use the Product Advertising API, Data Feed, or Product Advertising Content with any site or application, or in any other manner, that does not have the principal purpose of advertising and marketing the Amazon Site and driving sales of products and services on the Amazon Site."

It seems that they do not have an "informational" WS, but an adveritising WS, that is in fact the new name of the API.

However, what about using html scraping?
For example, retrieving http://www.amazon.com/exec/obidos/ASIN/{ISBN} returns details of a book, including title, author, cover, isbn and abstract.

[kovidgoyal]

I suppose it would be technically legal, though very slow. And what information would you get that you don't already get from the current metadata sources?

[earthq]

The information will be the same. The point is that Amazon has a huge books database and there are several books with metadata only present there.
It may be too much effort for a few books only in Amazon that could be entered manually, specially when scraping may get broken from time to time ...
Or maybe there is some users interest on this support...
Or perhaps it would open metadata providers as plugins... For example, what about metadata for comics, training books and other material that do not have ISBN?

Thanks.

[kovidgoyal]

Making the metadata providers plugin based is certainly possible, but until there are some significant non ISBN based sources of metadata, I don't much see the point.

[ldolse]

If it grabbed the full size cover I often find the amazon covers are higher resolution than the library thing covers. A lot of books have a summary or review when none is present in isbndb.

I just tried that URL and saw it was the normal product page, not some easily parsed data structure, so agree outside of possibly covers it's probably not worth the effort to write a scraper.

Just a note with covers, they dynamically modify a source cover image with a bunch of arguments. So while the thumbnail of a cover one scrapes would look like this:

http://ecx.images-amazon.com/images/I/41DcKN0STkL._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA240_SH20_OU01_.jpg

The full resolution image url is gotten by deleting the junk between the first '.' and the '.jpg':

http://ecx.images-amazon.com/images/I/41DcKN0STkL.jpg

Amazon also might be more reliable for covers than library thing, not only does librarything often have low-res, but it's also always timing out on me.

[kovidgoyal]

The problem with using Amazon for covers instead of library thing is that you need some way to map the book ISBN to the "canonical" ISBN that amazon uses. LibraryThing's ISBN API does that automatically. If you can figure out what the algorithm for that mapping is, I'll be happy to use Amazon for covers.

[ldolse]

I don't think it's a canonical ISBN, based on the info on Amazon's pages, there are two different ISBN's. Here's the product details for the book with those image URL's:

# Reading level: Young Adult
# Paperback: 544 pages
# Publisher: Little, Brown Young Readers (September 6, 2006)
# Language: English
# ISBN-10: 0316015849
# ISBN-13: 978-0316015844
# Product Dimensions: 8 x 5.5 x 1.6 inches

You'll note there are two ISBN's types listed there. ISBNDB (always??) uses the ISBN-13 ISBN, where as the amazon url format that the original poster highlights only supports ISBN-10.

I didn't bother reading this fully as it's a bit late for me in my part of the world, but it looks to me like to get ISBN-10 from ISBN-13 you need to drop off the first three digits and the hyphen. You then drop off the 10th digit and use the wikipedia algorithm to calculate what the final digit of the original 10 digit ISBN-10 should be.

Algorithm here:
http://en.wikipedia.org/wiki/Isbn#ISBN-10

Not entirely sure I got that all right, but I did read through it a second time and it seems like that might work...

[ldolse]

Actually, another thought on this - Amazon of course is selling the 'current' incarnation of a book. The problem with ISBNDB is that it lists all possible incarnations, and the current one is not neccessarily at the top. I do find with library thing that if I manually put in the ISBN-13 that I find via the amazon search results page that I have better luck with a higher quality cover. If I just randomly pick one of the ISBN's from ISBNdb then I'm more likely to get lowres/no cover.

So in retrospect, without solving the problem of figuring out what the most recently published ISBN is I'm not sure that Amazon would make that much of a difference.

[Jellby]

Originally Posted by ldolse

Not entirely sure I got that all right, but I did read through it a second time and it seems like that might work...

From my superficial knowledge of ISBN, that should work.

[kovidgoyal]

Hmm I recently changed calibre to always use the ISBN13 (if available) from the isbndb results (will be in the next beta). I suppose one way to do it is to find the isbns of all versions of a book and the search through each of them for a match on amazon, but that does seem like a lot of work for a rather small benefit.

[ldolse]

I agree, it's not pretty. Sounds like something for the back burner unless some other compelling discovery allows for a more elegant implementation.

[ldolse]

Ah, think I found something interesting here. From this ISBNDB URL:
http://isbndb.com/data-intro.html

example data message:

<?xml version="1.0" encoding="UTF-8"?>
<ISBNdb server_time="2005-07-29T03:02:22">
 <BookList total_results="1">
  <BookData book_id="paul_laurence_dunbar" isbn="0766013502">
   <Title>Paul Laurence Dunbar</Title>
   <TitleLong>Paul Laurence Dunbar: portrait of a poet</TitleLong>
   <AuthorsText>Catherine Reef</AuthorsText>
   <PublisherText publisher_id="enslow_publishers">
    Berkeley Heights, NJ: Enslow Publishers, c2000.
   </PublisherText>
   <Summary>
    A biography of the poet who faced racism and devoted himself
    to depicting the black experience in America.
   </Summary>
   <Notes>
    "Works by Paul Laurence Dunbar": p. 113-114.
    Includes bibliographical references (p. 124) and index.
   </Notes>
   <UrlsText></UrlsText>
   <AwardsText></AwardsText>
   <Prices>
    <Price store_id="alibris" is_in_stock="1" is_new="0"
           check_time="2005-07-29T01:18:18" price="14.92"/>
    <Price store_id="amazon" is_in_stock="1" is_new="1"
           check_time="2005-07-29T01:18:20" price="26.60" />
   </Prices>
  </BookData>
 </BookList>
</ISBNdb>

Note the last sections checking amazon for stock. If it's new & in stock at Amazon that's the most current version that would have the best cover to scrape.

Looks like you need to add the 'prices' argument to the isbndb query:

/api/books.xml?access_key=Z&results=prices&index1=isbn&value1=0061031321

More info here:
http://isbndb.com/docs/api/51-books.html

[kovidgoyal]

Hmm but calibre also gets metadata from google books and allows the user to select which ISBN they want. You'd still need a way to get from arbitrary ISBN to amazon cover URL

[ldolse]

I'm guessing Google books API allows for something similar... I know that linking into ways to sell the books was key for appeasing the publishers when Google announced that project. I'll dig into it and see what I can find.

[ldolse]

Ok, well google seems out for the moment on that particular feature, but it seems to me that this pricing feature of isbndb would still be quite useful. It easily allows the returned ISBNs to be ranked based on current availability. ISBNs being both new and in stock would be the most desirable as they would be most likely to have high quality covers along with useful things like access to reviews, ratings etc (not needed today but maybe in the future).

The GUI could put them at the top of the list and potentially format them in some way to indicate that those are preferred choices. New & in stock could have highest priority, in stock next, followed by ISBNs that return no availability info(google being in this latter category). I would be nice to have a checkbox or some option that allows the cover to automatically be retrieved when confirming the ISBN.


As far as the Amazon bit goes, figured out how to use the Amazon API effectively to solve that problem. Turns out that you need to have a developer ID to access the Amazon API, but that's free and trivially gotten. Forcing end users to sign up for that should theoretically leave any of the original poster's concerns about the license agreement to the end user and not to Calibre.

Once you have the developer ID you take the ISBN-10 and submit it to Amazon using the examples below:

This is a working example from a tutorial on kokogiak.com and uses authentication information published there:

http://xml-us.amznxslt.com/onca/xml?Service=AWSECommerceService&SubscriptionId=19267494ZR5A8E2CGPR2&AssociateTag=kokogiak7-20&IdType=ISBN&ItemId=0316038377&Operation=ItemLookup&ResponseGroup=Medium,ItemAttributes,OfferFull&SearchIndex=Books

The actual URL needed once a user signs up for a developer account looks like this (non-functioning example):

http://xml-us.amznxslt.com/onca/xml?Service=AWSECommerceService&AWSAccessKeyId=<key_deleted>&IdType=ISBN&ItemId=0316038377&Operation=ItemLookup&ResponseGroup=Medium,ItemAttributes,OfferFull&SearchIndex=Books

Note that with the free developer account only the AWSAccessKeyId is required vs. AssociateTag & SubscriptionId in the working example URL.

The different bits of this are:

1. AWSAccessKeyId=<key_here> - use ID provided by amazon after free signup, go to http://aws.amazon.com/ to sign up.
2. IdType=ISBN - Set this to search by ISBN-10, though other options are possible
3. ItemId=0316038377 - ISBN-10 here
4. ResponseGroup=Medium - additional arguments possible as illustrated above, medium seems to cover the basic requirements
5. SearchIndex=Books - search category required

Once that's submitted it spits back an xml file. One of the nodes is called '<LargeImage>', this is the full res image URL. There are other nodes with Editorial reviews and product descriptions as well. Using the Kokogiak credentials gave a bit of extra data, not sure if that's possible with a free account, but it's not critical for what we're discussing here.


Regarding Google, it appears that they have some ability to grab prices and availability, but I haven't been able to find anything in their public API documentation. Here's an example though:
http://books.google.com/books?id=MMlxzMNkE_0C