Using Overdrive for covers?

ldolse · 02-20-2011, 07:07 AM

Glad to see the new sub-forum.

Wanted to see what people thought of an idea for a new cover source, and get any ideas to improve on it if it seems feasible.

I've recently discovered that the highest quality covers on the 'net are hosted by Libraries around the world, and on Overdrive's servers which back the libraries. The resolutions are substantially higher/less compressed than what comes from Amazon/LT.

I was thinking this could provide a couple potential benefits:

Increased quality
Cover downloading could be distributed across many different libraries, as all libraries seem to use the same query format, it would just be a matter of maintaining a list of base URLs. I particularly like this as it would help prevent Calibre from being viewed as a DDOS source from more third parties.

There are a few stumbling blocks:

Adobe uses it's own ID separate from ISBN, and it needs to be determined before getting the cover
Libraries only host covers for things they have in their collection, although adobe itself could be used as a source of last resort

Knowing the Adobe ID really the only accurate way to get a cover, but I haven't found any good ways to get it aside from searching a Library's Overdrive database. This Overdrive id only matches the ebook edition's ISBN, searching for an alternate ISBN will fail. For some reason advanced search works better with individual libraries than it does directly against Adobe servers....

Here's some info on how this works and examples of how the same cover could be found across multiple systems with the Adobe identifier. Using 'On the Road' by Jack Kerouac as an example:

To get the Overdrive ID one can just do an advanced search on an individual library's collection (Chicago in this case):
http://overdrive.chipublib.org/82DC6...BANGSearch.dll

The request for all the libraries is always to BANGSearch.dll, though the hostname and url prefix chanes with each library. The actual query is in the request content body:

Code:

Title=On+the+Road&Creator=Jack+Kerouac&Keyword=&ISBN=&Format=&Language=&Publisher=&Subject=&Award=&CollDate=&PerPage=10&Sort=SortBy%3Dtitle

These arguments seem to be uniform across all Overdrive based libraries.

Scraping the result contents (unfortunately I don't see any way to do this without scraping) gives three unique Adobe Overdrive IDs:
ContentDetails.htm?ID=6609D1A7-98A4-4B41-B9A2-94BEE83CF861
ContentDetails.htm?ID=7FEC0594-4FAD-4CBC-83B3-7DAEBCC5B900
ContentDetails.htm?ID=D8414773-1865-4C6B-AEB3-F2CB05C84F16

The third one is the epub book, the first two are audiobooks. Ideally one could prioritize epub/mobi/pdf covers over audiobook covers. The Overdrive advanced search GUI allows formats to be specified, but it doesn't allow 'OR'. Not sure if it allows it under the hood, I haven't yet tried to construct my own queries.

We'll use the epub's ID - D8414773-1865-4C6B-AEB3-F2CB05C84F16 - with this the cover can be retrieved from any library with the book in it's collection.

Overdrive's own servers:

http://images.contentreserve.com/Ima...6%7dImg100.jpg

Cleveland Public Library:

http://emedia.clevnet.org/AB554A4C-7...3-F2CB05C84F16

Chicago Public Library:

http://overdrive.chipublib.org/82DC6...3-F2CB05C84F16

In all those cases the rest of the URL stays the same, just change the Overdrive ID and you get the specified book cover.

Next hurdle would be libraries with large collections to increase the likelihood of a result. I haven't found many libraries with more than 20,000 titles. Singapore, Cleveland, and Seattle are all around that mark for epub/mobi/pdf. Chicago, though I used it as an example isn't great with ~6000.

Openlibrary also maintains Overdrive IDs for ebook editions, but not sure how universally they do it, and thus far it seems like more scraping would be required to get to the desired result, which isn't ideal:
http://openlibrary.org/books/OL24273691M/On_the_Road

chaley · 02-20-2011, 07:53 AM

Sounds like you have looked quite deeply into the question. I don't have strong feelings one way or the other about its usefulness, but then I add books to my library very slowly and don't use much of the metadata downloading tools.

There seems to be a fair amount in common between what you are proposing and kiwidude's LibraryThing plugin. Both require finding obscure internal IDs, both require scraping, and both require storing some additional information about the book. Perhaps he will come along and enlighten us.

ldolse · 02-20-2011, 08:25 AM

I like the convenience of Calibre just downloading the cover, but generally try to insert the best cover I can find, which usually isn't LT/Amazon. Just trying to save myself the steps of manually searching if I can.

I'm not doing massive bulk downloads myself either, but based on the discussions I've seen around LibraryThing I'm guessing there are a fair number of users that are doing exactly that.

Side note - I figured out why search.overdrive.com wasn't working with advanced search - their application expects spaces to be replaced by '+' in author names/book titles, which you can see in my example previously with the library query. Overdrive's own advanced search is broken and escaping the spaces with %20, which causes the lookup to fail.

kovidgoyal · 02-20-2011, 01:42 PM

calibre already uses openlibrary with ISBNs for covers. You can handle special identifiers by simply having an in memory cache that maps ISBNs to the identifiers. See the amazon plugin for an example which maps isbns to asin and stores the result in a global cache.

Currently cover download is on a first result basis, i.e. as soon as a particular source returns a cover, calibre uses it, it does not wait to get covers from all sources and then pick the best one. Doing so is possible in principle but it means that the download will become much slower.

ldolse · 02-20-2011, 09:05 PM

Does Calibre try OpenLibrary first, then Amazon? Would it be more straightforward to rank the cover download plugins so that higher quality sources are tried before lower quality ones? Or are you saying all potential sources are tried at once?

Unfortunately any ISBN won't map to the Overdrive ID, it has to be the specific ISBN of the Overdrive edition. That said, I think the scraping could still be done and mapped as you described, I'll check out the Amazon code.

kiwidude · 02-20-2011, 09:15 PM

Quote:

Originally Posted by chaley

There seems to be a fair amount in common between what you are proposing and kiwidude's LibraryThing plugin. Both require finding obscure internal IDs, both require scraping, and both require storing some additional information about the book. Perhaps he will come along and enlighten us.

I think you meant my Goodreads plugin rather than LibraryThing

As Kovid has responded there is little I can add. I'm in a holding pattern on any further changes to the Goodreads covers and metadata downloads plugins until Kovid is done with all his metadata API rewrite stuff. No rush since what is there still seems to work in the meantime. Absolutely there are a number of similar issues though.

JSWolf · 02-20-2011, 09:52 PM

Most covers on Overdrive are at 510x680. Incorrect aspect ratio. So no, Overdrive is a really poor source for covers. MS Reader specifications say the cover should be 510x60 to fully fill the MS Reader window. But I find this just wrong as a lot of covers to look off like that.

Find a source that uses correct aspect ratio covers.

kovidgoyal · 02-21-2011, 12:16 AM

Quote:

Originally Posted by ldolse

Does Calibre try OpenLibrary first, then Amazon? Would it be more straightforward to rank the cover download plugins so that higher quality sources are tried before lower quality ones? Or are you saying all potential sources are tried at once?

It's simultaneous. Ranking is a possibility, but I'd be uncomfortable hardcoding the ranking, instead make it use configurable, so the user can decide what source she would prefer.

ldolse · 02-21-2011, 01:25 AM

I don't think aspect ratio is that big a deal, as Calibre's default settings makes that scale dynamically anyway. No offense, JSWolf, but I think you're among a relatively small group of users that pays attention to that, and anyway individual metadata plugins can be disabled.

That said, I agree with Kovid that allowing users to configure their ranking would make more sense. I'm not sure I'm ready to create the GUI required for allowing users to configure that, but I think I feel comfortable getting started on an Overdrive plugin at least.

Are there things coming with the metadata API rewrite that I should wait for? Not entirely sure what the plans are there.

theducks · 02-21-2011, 10:23 AM

Quote:

Originally Posted by ldolse

I don't think aspect ratio is that big a deal, as Calibre's default settings makes that scale dynamically anyway. No offense, JSWolf, but I think you're among a relatively small group of users that pays attention to that, and anyway individual metadata plugins can be disabled.

That said, I agree with Kovid that allowing users to configure their ranking would make more sense. I'm not sure I'm ready to create the GUI required for allowing users to configure that, but I think I feel comfortable getting started on an Overdrive plugin at least.

Are there things coming with the metadata API rewrite that I should wait for? Not entirely sure what the plans are there.

I will take just about any aspect ratio, as long as it is maintained during re-scale. (I have been watching Letter boxed, Laserdisc movies since 1980. I don't mind black bars

)

cybmole · 02-21-2011, 11:16 AM

there are many sites which host high quality cd, dvd, pc game etc. covers, with uploads from their user community. I guess book covers will evolve similarly, in time.

kovidgoyal · 02-21-2011, 11:18 AM

The new framework is being developed in metadata/sources

Cover download is relatively simple, so it should be easy to port your code once the new framework is ready.

JSWolf · 02-21-2011, 10:46 PM

Quote:

Originally Posted by theducks

I will take just about any aspect ratio, as long as it is maintained during re-scale. (I have been watching Letter boxed, Laserdisc movies since 1980. I don't mind black bars

)

But will you take incorrect aspect ratio. I prefer the correct aspect ratio. In fact, a bug was recently fixed wit regard to aspect ratio that I mentioned.

JSWolf · 02-21-2011, 10:47 PM

Quote:

Originally Posted by cybmole

there are many sites which host high quality cd, dvd, pc game etc. covers, with uploads from their user community. I guess book covers will evolve similarly, in time.

There is a website for book covers. But the problem is they are in JPG and over compressed. The artifacts are very easy to se which makes them useless.

ldolse · 02-23-2011, 10:23 PM

What's the best way to debug metadata download plugins? Is there a simple way to do it from the CLI, or do I need just execute the metadata query from the GUI and watch what happens?

02-20-2011, 07:07 AM	#1
ldolse Wizard Posts: 1,337 Karma: 123455 Join Date: Apr 2009 Location: Malaysia Device: PRS-650, iPhone	Using Overdrive for covers? Glad to see the new sub-forum. Wanted to see what people thought of an idea for a new cover source, and get any ideas to improve on it if it seems feasible. I've recently discovered that the highest quality covers on the 'net are hosted by Libraries around the world, and on Overdrive's servers which back the libraries. The resolutions are substantially higher/less compressed than what comes from Amazon/LT. I was thinking this could provide a couple potential benefits: Increased quality Cover downloading could be distributed across many different libraries, as all libraries seem to use the same query format, it would just be a matter of maintaining a list of base URLs. I particularly like this as it would help prevent Calibre from being viewed as a DDOS source from more third parties. There are a few stumbling blocks: Adobe uses it's own ID separate from ISBN, and it needs to be determined before getting the cover Libraries only host covers for things they have in their collection, although adobe itself could be used as a source of last resort Knowing the Adobe ID really the only accurate way to get a cover, but I haven't found any good ways to get it aside from searching a Library's Overdrive database. This Overdrive id only matches the ebook edition's ISBN, searching for an alternate ISBN will fail. For some reason advanced search works better with individual libraries than it does directly against Adobe servers.... Here's some info on how this works and examples of how the same cover could be found across multiple systems with the Adobe identifier. Using 'On the Road' by Jack Kerouac as an example: To get the Overdrive ID one can just do an advanced search on an individual library's collection (Chicago in this case): http://overdrive.chipublib.org/82DC6...BANGSearch.dll The request for all the libraries is always to BANGSearch.dll, though the hostname and url prefix chanes with each library. The actual query is in the request content body: Code: Title=On+the+Road&Creator=Jack+Kerouac&Keyword=&ISBN=&Format=&Language=&Publisher=&Subject=&Award=&CollDate=&PerPage=10&Sort=SortBy%3Dtitle These arguments seem to be uniform across all Overdrive based libraries. Scraping the result contents (unfortunately I don't see any way to do this without scraping) gives three unique Adobe Overdrive IDs: ContentDetails.htm?ID=6609D1A7-98A4-4B41-B9A2-94BEE83CF861 ContentDetails.htm?ID=7FEC0594-4FAD-4CBC-83B3-7DAEBCC5B900 ContentDetails.htm?ID=D8414773-1865-4C6B-AEB3-F2CB05C84F16 The third one is the epub book, the first two are audiobooks. Ideally one could prioritize epub/mobi/pdf covers over audiobook covers. The Overdrive advanced search GUI allows formats to be specified, but it doesn't allow 'OR'. Not sure if it allows it under the hood, I haven't yet tried to construct my own queries. We'll use the epub's ID - D8414773-1865-4C6B-AEB3-F2CB05C84F16 - with this the cover can be retrieved from any library with the book in it's collection. Overdrive's own servers: http://images.contentreserve.com/Ima...6%7dImg100.jpg Cleveland Public Library: http://emedia.clevnet.org/AB554A4C-7...3-F2CB05C84F16 Chicago Public Library: http://overdrive.chipublib.org/82DC6...3-F2CB05C84F16 In all those cases the rest of the URL stays the same, just change the Overdrive ID and you get the specified book cover. Next hurdle would be libraries with large collections to increase the likelihood of a result. I haven't found many libraries with more than 20,000 titles. Singapore, Cleveland, and Seattle are all around that mark for epub/mobi/pdf. Chicago, though I used it as an example isn't great with ~6000. Openlibrary also maintains Overdrive IDs for ebook editions, but not sure how universally they do it, and thus far it seems like more scraping would be required to get to the desired result, which isn't ideal: http://openlibrary.org/books/OL24273691M/On_the_Road Last edited by ldolse; 03-01-2011 at 05:34 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[Covers Plugin] Goodreads Covers Deprecated	kiwidude	Plugins	13	04-17-2011 05:09 PM
Is all Overdrive the same?	CWatkinsNash	General Discussions	3	12-28-2010 04:01 PM
Covers, covers and damn statistics (wait, I got that wrong).	Moejoe	Writers' Corner	86	11-29-2010 08:34 PM
Stop Using Overdrive	Fat Abe	General Discussions	19	09-11-2010 08:30 PM
Overdrive Overseas	Honch	Which one should I buy?	3	12-08-2009 08:21 AM

02-20-2011, 07:53 AM	#2
chaley Grand Sorcerer Posts: 11,738 Karma: 6997045 Join Date: Jan 2010 Location: Notts, England Device: Kobo Libra 2	Sounds like you have looked quite deeply into the question. I don't have strong feelings one way or the other about its usefulness, but then I add books to my library very slowly and don't use much of the metadata downloading tools. There seems to be a fair amount in common between what you are proposing and kiwidude's LibraryThing plugin. Both require finding obscure internal IDs, both require scraping, and both require storing some additional information about the book. Perhaps he will come along and enlighten us.

02-20-2011, 08:25 AM	#3
ldolse Wizard Posts: 1,337 Karma: 123455 Join Date: Apr 2009 Location: Malaysia Device: PRS-650, iPhone	I like the convenience of Calibre just downloading the cover, but generally try to insert the best cover I can find, which usually isn't LT/Amazon. Just trying to save myself the steps of manually searching if I can. I'm not doing massive bulk downloads myself either, but based on the discussions I've seen around LibraryThing I'm guessing there are a fair number of users that are doing exactly that. Side note - I figured out why search.overdrive.com wasn't working with advanced search - their application expects spaces to be replaced by '+' in author names/book titles, which you can see in my example previously with the library query. Overdrive's own advanced search is broken and escaping the spaces with %20, which causes the lookup to fail.

02-20-2011, 01:42 PM	#4
kovidgoyal creator of calibre Posts: 43,850 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	calibre already uses openlibrary with ISBNs for covers. You can handle special identifiers by simply having an in memory cache that maps ISBNs to the identifiers. See the amazon plugin for an example which maps isbns to asin and stores the result in a global cache. Currently cover download is on a first result basis, i.e. as soon as a particular source returns a cover, calibre uses it, it does not wait to get covers from all sources and then pick the best one. Doing so is possible in principle but it means that the download will become much slower.

02-20-2011, 09:05 PM	#5
ldolse Wizard Posts: 1,337 Karma: 123455 Join Date: Apr 2009 Location: Malaysia Device: PRS-650, iPhone	Does Calibre try OpenLibrary first, then Amazon? Would it be more straightforward to rank the cover download plugins so that higher quality sources are tried before lower quality ones? Or are you saying all potential sources are tried at once? Unfortunately any ISBN won't map to the Overdrive ID, it has to be the specific ISBN of the Overdrive edition. That said, I think the scraping could still be done and mapped as you described, I'll check out the Amazon code.

02-20-2011, 09:52 PM	#7
JSWolf Resident Curmudgeon Posts: 73,931 Karma: 128903250 Join Date: Nov 2006 Location: Roslindale, Massachusetts Device: Kobo Libra 2, Kobo Aura H2O, PRS-650, PRS-T1, nook STR, PW3	Most covers on Overdrive are at 510x680. Incorrect aspect ratio. So no, Overdrive is a really poor source for covers. MS Reader specifications say the cover should be 510x60 to fully fill the MS Reader window. But I find this just wrong as a lot of covers to look off like that. Find a source that uses correct aspect ratio covers.

02-21-2011, 01:25 AM	#9
ldolse Wizard Posts: 1,337 Karma: 123455 Join Date: Apr 2009 Location: Malaysia Device: PRS-650, iPhone	I don't think aspect ratio is that big a deal, as Calibre's default settings makes that scale dynamically anyway. No offense, JSWolf, but I think you're among a relatively small group of users that pays attention to that, and anyway individual metadata plugins can be disabled. That said, I agree with Kovid that allowing users to configure their ranking would make more sense. I'm not sure I'm ready to create the GUI required for allowing users to configure that, but I think I feel comfortable getting started on an Overdrive plugin at least. Are there things coming with the metadata API rewrite that I should wait for? Not entirely sure what the plans are there.

02-21-2011, 11:16 AM	#11
cybmole Wizard Posts: 3,720 Karma: 1759970 Join Date: Sep 2010 Device: none	there are many sites which host high quality cd, dvd, pc game etc. covers, with uploads from their user community. I guess book covers will evolve similarly, in time.

02-21-2011, 11:18 AM	#12
kovidgoyal creator of calibre Posts: 43,850 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	The new framework is being developed in metadata/sources Cover download is relatively simple, so it should be easy to port your code once the new framework is ready.

02-23-2011, 10:23 PM	#15
ldolse Wizard Posts: 1,337 Karma: 123455 Join Date: Apr 2009 Location: Malaysia Device: PRS-650, iPhone	What's the best way to debug metadata download plugins? Is there a simple way to do it from the CLI, or do I need just execute the metadata query from the GUI and watch what happens?

Advert

Advert