Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 06-21-2009, 07:28 PM   #1
earthq
Enthusiast
earthq began at the beginning.
 
Posts: 30
Karma: 10
Join Date: Jun 2009
Device: none
Amazon Support

Hi,

I have seen in closed tickets about this due to Amazon licensing terms. I guess we are dealing with this:
https://affiliate-program.amazon.com...=assoc-api-faq

In particular, it states:
"You will not use the Product Advertising API, Data Feed, or Product Advertising Content with any site or application, or in any other manner, that does not have the principal purpose of advertising and marketing the Amazon Site and driving sales of products and services on the Amazon Site."

It seems that they do not have an "informational" WS, but an adveritising WS, that is in fact the new name of the API.

However, what about using html scraping?
For example, retrieving http://www.amazon.com/exec/obidos/ASIN/{ISBN} returns details of a book, including title, author, cover, isbn and abstract.
earthq is offline   Reply With Quote
Old 06-21-2009, 08:29 PM   #2
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,415
Karma: 4961459
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
I suppose it would be technically legal, though very slow. And what information would you get that you don't already get from the current metadata sources?
kovidgoyal is offline   Reply With Quote
Old 06-22-2009, 12:42 AM   #3
earthq
Enthusiast
earthq began at the beginning.
 
Posts: 30
Karma: 10
Join Date: Jun 2009
Device: none
The information will be the same. The point is that Amazon has a huge books database and there are several books with metadata only present there.
It may be too much effort for a few books only in Amazon that could be entered manually, specially when scraping may get broken from time to time ...
Or maybe there is some users interest on this support...
Or perhaps it would open metadata providers as plugins... For example, what about metadata for comics, training books and other material that do not have ISBN?

Thanks.
earthq is offline   Reply With Quote
Old 06-22-2009, 02:34 AM   #4
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,415
Karma: 4961459
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Making the metadata providers plugin based is certainly possible, but until there are some significant non ISBN based sources of metadata, I don't much see the point.
kovidgoyal is offline   Reply With Quote
Old 06-22-2009, 05:36 AM   #5
ldolse
Wizard
ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.
 
Posts: 1,337
Karma: 123455
Join Date: Apr 2009
Location: Malaysia
Device: PRS-650, iPhone
If it grabbed the full size cover I often find the amazon covers are higher resolution than the library thing covers. A lot of books have a summary or review when none is present in isbndb.

I just tried that URL and saw it was the normal product page, not some easily parsed data structure, so agree outside of possibly covers it's probably not worth the effort to write a scraper.

Just a note with covers, they dynamically modify a source cover image with a bunch of arguments. So while the thumbnail of a cover one scrapes would look like this:
Code:
http://ecx.images-amazon.com/images/I/41DcKN0STkL._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA240_SH20_OU01_.jpg
The full resolution image url is gotten by deleting the junk between the first '.' and the '.jpg':
Code:
http://ecx.images-amazon.com/images/I/41DcKN0STkL.jpg
Amazon also might be more reliable for covers than library thing, not only does librarything often have low-res, but it's also always timing out on me.
ldolse is offline   Reply With Quote
Old 06-22-2009, 11:23 AM   #6
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,415
Karma: 4961459
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
The problem with using Amazon for covers instead of library thing is that you need some way to map the book ISBN to the "canonical" ISBN that amazon uses. LibraryThing's ISBN API does that automatically. If you can figure out what the algorithm for that mapping is, I'll be happy to use Amazon for covers.
kovidgoyal is offline   Reply With Quote
Old 06-22-2009, 11:38 AM   #7
ldolse
Wizard
ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.
 
Posts: 1,337
Karma: 123455
Join Date: Apr 2009
Location: Malaysia
Device: PRS-650, iPhone
I don't think it's a canonical ISBN, based on the info on Amazon's pages, there are two different ISBN's. Here's the product details for the book with those image URL's:
Code:
# Reading level: Young Adult
# Paperback: 544 pages
# Publisher: Little, Brown Young Readers (September 6, 2006)
# Language: English
# ISBN-10: 0316015849
# ISBN-13: 978-0316015844
# Product Dimensions: 8 x 5.5 x 1.6 inches
You'll note there are two ISBN's types listed there. ISBNDB (always??) uses the ISBN-13 ISBN, where as the amazon url format that the original poster highlights only supports ISBN-10.

I didn't bother reading this fully as it's a bit late for me in my part of the world, but it looks to me like to get ISBN-10 from ISBN-13 you need to drop off the first three digits and the hyphen. You then drop off the 10th digit and use the wikipedia algorithm to calculate what the final digit of the original 10 digit ISBN-10 should be.

Algorithm here:
http://en.wikipedia.org/wiki/Isbn#ISBN-10

Not entirely sure I got that all right, but I did read through it a second time and it seems like that might work...
ldolse is offline   Reply With Quote
Old 06-22-2009, 11:46 AM   #8
ldolse
Wizard
ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.
 
Posts: 1,337
Karma: 123455
Join Date: Apr 2009
Location: Malaysia
Device: PRS-650, iPhone
Actually, another thought on this - Amazon of course is selling the 'current' incarnation of a book. The problem with ISBNDB is that it lists all possible incarnations, and the current one is not neccessarily at the top. I do find with library thing that if I manually put in the ISBN-13 that I find via the amazon search results page that I have better luck with a higher quality cover. If I just randomly pick one of the ISBN's from ISBNdb then I'm more likely to get lowres/no cover.

So in retrospect, without solving the problem of figuring out what the most recently published ISBN is I'm not sure that Amazon would make that much of a difference.
ldolse is offline   Reply With Quote
Old 06-22-2009, 11:49 AM   #9
Jellby
frumious Bandersnatch
Jellby ought to be getting tired of karma fortunes by now.Jellby ought to be getting tired of karma fortunes by now.Jellby ought to be getting tired of karma fortunes by now.Jellby ought to be getting tired of karma fortunes by now.Jellby ought to be getting tired of karma fortunes by now.Jellby ought to be getting tired of karma fortunes by now.Jellby ought to be getting tired of karma fortunes by now.Jellby ought to be getting tired of karma fortunes by now.Jellby ought to be getting tired of karma fortunes by now.Jellby ought to be getting tired of karma fortunes by now.Jellby ought to be getting tired of karma fortunes by now.
 
Jellby's Avatar
 
Posts: 5,991
Karma: 4346919
Join Date: Jan 2008
Location: Spaniard in Sweden
Device: Cybook Orizon, Kobo Aura
Quote:
Originally Posted by ldolse View Post
Not entirely sure I got that all right, but I did read through it a second time and it seems like that might work...
From my superficial knowledge of ISBN, that should work.
Jellby is offline   Reply With Quote
Old 06-22-2009, 11:51 AM   #10
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,415
Karma: 4961459
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Hmm I recently changed calibre to always use the ISBN13 (if available) from the isbndb results (will be in the next beta). I suppose one way to do it is to find the isbns of all versions of a book and the search through each of them for a match on amazon, but that does seem like a lot of work for a rather small benefit.

Last edited by kovidgoyal; 06-22-2009 at 12:13 PM.
kovidgoyal is offline   Reply With Quote
Old 06-22-2009, 12:17 PM   #11
ldolse
Wizard
ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.
 
Posts: 1,337
Karma: 123455
Join Date: Apr 2009
Location: Malaysia
Device: PRS-650, iPhone
I agree, it's not pretty. Sounds like something for the back burner unless some other compelling discovery allows for a more elegant implementation.
ldolse is offline   Reply With Quote
Old 06-22-2009, 10:29 PM   #12
ldolse
Wizard
ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.
 
Posts: 1,337
Karma: 123455
Join Date: Apr 2009
Location: Malaysia
Device: PRS-650, iPhone
Ah, think I found something interesting here. From this ISBNDB URL:
http://isbndb.com/data-intro.html

example data message:
Code:
<?xml version="1.0" encoding="UTF-8"?>
<ISBNdb server_time="2005-07-29T03:02:22">
 <BookList total_results="1">
  <BookData book_id="paul_laurence_dunbar" isbn="0766013502">
   <Title>Paul Laurence Dunbar</Title>
   <TitleLong>Paul Laurence Dunbar: portrait of a poet</TitleLong>
   <AuthorsText>Catherine Reef</AuthorsText>
   <PublisherText publisher_id="enslow_publishers">
    Berkeley Heights, NJ: Enslow Publishers, c2000.
   </PublisherText>
   <Summary>
    A biography of the poet who faced racism and devoted himself
    to depicting the black experience in America.
   </Summary>
   <Notes>
    "Works by Paul Laurence Dunbar": p. 113-114.
    Includes bibliographical references (p. 124) and index.
   </Notes>
   <UrlsText></UrlsText>
   <AwardsText></AwardsText>
   <Prices>
    <Price store_id="alibris" is_in_stock="1" is_new="0"
           check_time="2005-07-29T01:18:18" price="14.92"/>
    <Price store_id="amazon" is_in_stock="1" is_new="1"
           check_time="2005-07-29T01:18:20" price="26.60" />
   </Prices>
  </BookData>
 </BookList>
</ISBNdb>
Note the last sections checking amazon for stock. If it's new & in stock at Amazon that's the most current version that would have the best cover to scrape.

Looks like you need to add the 'prices' argument to the isbndb query:
Code:
/api/books.xml?access_key=Z&results=prices&index1=isbn&value1=0061031321
More info here:
http://isbndb.com/docs/api/51-books.html
ldolse is offline   Reply With Quote
Old 06-22-2009, 11:45 PM   #13
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 25,415
Karma: 4961459
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Hmm but calibre also gets metadata from google books and allows the user to select which ISBN they want. You'd still need a way to get from arbitrary ISBN to amazon cover URL
kovidgoyal is offline   Reply With Quote
Old 06-22-2009, 11:51 PM   #14
ldolse
Wizard
ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.
 
Posts: 1,337
Karma: 123455
Join Date: Apr 2009
Location: Malaysia
Device: PRS-650, iPhone
I'm guessing Google books API allows for something similar... I know that linking into ways to sell the books was key for appeasing the publishers when Google announced that project. I'll dig into it and see what I can find.
ldolse is offline   Reply With Quote
Old 06-23-2009, 09:41 AM   #15
ldolse
Wizard
ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.
 
Posts: 1,337
Karma: 123455
Join Date: Apr 2009
Location: Malaysia
Device: PRS-650, iPhone
Ok, well google seems out for the moment on that particular feature, but it seems to me that this pricing feature of isbndb would still be quite useful. It easily allows the returned ISBNs to be ranked based on current availability. ISBNs being both new and in stock would be the most desirable as they would be most likely to have high quality covers along with useful things like access to reviews, ratings etc (not needed today but maybe in the future).

The GUI could put them at the top of the list and potentially format them in some way to indicate that those are preferred choices. New & in stock could have highest priority, in stock next, followed by ISBNs that return no availability info(google being in this latter category). I would be nice to have a checkbox or some option that allows the cover to automatically be retrieved when confirming the ISBN.


As far as the Amazon bit goes, figured out how to use the Amazon API effectively to solve that problem. Turns out that you need to have a developer ID to access the Amazon API, but that's free and trivially gotten. Forcing end users to sign up for that should theoretically leave any of the original poster's concerns about the license agreement to the end user and not to Calibre.

Once you have the developer ID you take the ISBN-10 and submit it to Amazon using the examples below:

This is a working example from a tutorial on kokogiak.com and uses authentication information published there:
Code:
http://xml-us.amznxslt.com/onca/xml?Service=AWSECommerceService&SubscriptionId=19267494ZR5A8E2CGPR2&AssociateTag=kokogiak7-20&IdType=ISBN&ItemId=0316038377&Operation=ItemLookup&ResponseGroup=Medium,ItemAttributes,OfferFull&SearchIndex=Books
The actual URL needed once a user signs up for a developer account looks like this (non-functioning example):
Code:
http://xml-us.amznxslt.com/onca/xml?Service=AWSECommerceService&AWSAccessKeyId=<key_deleted>&IdType=ISBN&ItemId=0316038377&Operation=ItemLookup&ResponseGroup=Medium,ItemAttributes,OfferFull&SearchIndex=Books
Note that with the free developer account only the AWSAccessKeyId is required vs. AssociateTag & SubscriptionId in the working example URL.

The different bits of this are:
  1. AWSAccessKeyId=<key_here> - use ID provided by amazon after free signup, go to http://aws.amazon.com/ to sign up.
  2. IdType=ISBN - Set this to search by ISBN-10, though other options are possible
  3. ItemId=0316038377 - ISBN-10 here
  4. ResponseGroup=Medium - additional arguments possible as illustrated above, medium seems to cover the basic requirements
  5. SearchIndex=Books - search category required

Once that's submitted it spits back an xml file. One of the nodes is called '<LargeImage>', this is the full res image URL. There are other nodes with Editorial reviews and product descriptions as well. Using the Kokogiak credentials gave a bit of extra data, not sure if that's possible with a free account, but it's not critical for what we're discussing here.


Regarding Google, it appears that they have some ability to grab prices and availability, but I haven't been able to find anything in their public API documentation. Here's an example though:
http://books.google.com/books?id=MMlxzMNkE_0C

Last edited by ldolse; 06-23-2009 at 10:12 AM.
ldolse is offline   Reply With Quote
Reply

Tags
book cover, isbn-13

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Would it be possible for Amazon to re-compile Amazon Reader App for Pocketbook OS dev brecklundin PocketBook 3 09-22-2010 08:04 PM
Battery Issues and No Support from Support BJonas Astak EZReader 11 05-29-2010 02:31 AM
Is Astak support on vacation or just stopped support? dmikov Astak EZReader 9 01-22-2010 01:43 PM
Amazon expands language support in Kindle Store Nate the great News 58 01-20-2010 04:46 AM
Amazon Support Home Runs Again! delphidb96 Amazon Kindle 8 12-26-2009 05:18 PM


All times are GMT -4. The time now is 04:18 AM.


MobileRead.com is a privately owned, operated and funded community.