Curious how Calibre "gets" new books

[Dionysus]

Can one access Calibre's API to search?

Or can anyone offer any information on the subject of getting book information from Amazon? I've looked into Amazon's API and searching for a book returns the "Digital List Price" and never the "Kindle Price".

When asking Amazon about it, they say that they do not provide this in their API (there is a forum post dated Dec. 6, 2010 that they stick to). However, Calibre correctly gets the Kindle price of a book, therefore they have found a way around this.

Is it just a matter of parsing the page? or is their some other API stuff that exists?

Thanks for listening/reading!

[kovidgoyal]

web scraping. i.e. parsing the page

[Dionysus]

I don't understand?? I was not asking about DRM removal... I'm talking about searching Amazon. You manage to get the correct prices, I was just curious how since Amazon's API does not support the return via API of "Kindle Price".

[Dionysus]

Or are you telling me that you parse the page?

[kiwidude]

Originally Posted by Dionysus

Or are you telling me that you parse the page?

That is exactly what Kovid is saying - the plugins for calibre parse the same sort of Amazon web pages that you would see in a web browser, but using xpath queries to extract the relevant data from the html. Hence why when Amazon changes their web page layout you see people complaining about their metadata downloads not working...

[Dionysus]

Thank you. Then I know what to do I was wondering if there was some special API bits that I wasn't privy too.

I appreciate your comments!

[Dionysus]

It's funny, but when I parse the Amazon search page, I get all the results back but "Pricing information not available". The same page when viewed in a browser returns pricing, but when viewed via a file_get_contents does not!

[PaulPosition]

Dionysus - Pricing information may be available only based on information stored in a cookie (like country on a "real user browser"), it may only be available when you've followed some internal link (with maybe a session number attached, or a 'hidden' GET or POST field), or the website may check for user-agents and serve only recognized names such as Explorer or Chrome...

Most Calibre plugins code is available. If I'm not mistaken it's Python and not PHP, but having a look at one or two of the metadata-grabbing ones could be worth it for you.

[Dionysus]

Thanks Paul (and to all the rest of you) for your answer. I appreciate it. And hello from a fellow Montrealer

[Dionysus]

Well Paul, you were right. I added this code :

ini_set('user_agent', $_SERVER['HTTP_USER_AGENT']);

And after that prices were displayed. However, they are not the right prices so I am betting you're also right about the country not being passed. But I'm still trying to figure out how I could do that.

Thanks for your help and if you know the answer to this, I'd appreciate it.

[kiwidude]

Why don't you just look at calibre's amazon store plugin(s) source code? Since that as you already know has the necessary logic to retrieve prices.

[Dionysus]

Actually, I would if I knew where I should look. I'm on a mac and I don't know where the plugins are. I'll do some searching online to find out where.

Thanks for reminding me.

[kiwidude]

Just download the calibre source, and all the store plugins (including the 6 amazon ones) are within:
src\calibre\gui2\store\stores