Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Development

Notices

Reply
 
Thread Tools Search this Thread
Old 02-20-2011, 07:07 AM   #1
ldolse
Wizard
ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.
 
Posts: 1,337
Karma: 123455
Join Date: Apr 2009
Location: Malaysia
Device: PRS-650, iPhone
Using Overdrive for covers?

Glad to see the new sub-forum.

Wanted to see what people thought of an idea for a new cover source, and get any ideas to improve on it if it seems feasible.

I've recently discovered that the highest quality covers on the 'net are hosted by Libraries around the world, and on Overdrive's servers which back the libraries. The resolutions are substantially higher/less compressed than what comes from Amazon/LT.

I was thinking this could provide a couple potential benefits:
  • Increased quality
  • Cover downloading could be distributed across many different libraries, as all libraries seem to use the same query format, it would just be a matter of maintaining a list of base URLs. I particularly like this as it would help prevent Calibre from being viewed as a DDOS source from more third parties.

There are a few stumbling blocks:
  • Adobe uses it's own ID separate from ISBN, and it needs to be determined before getting the cover
  • Libraries only host covers for things they have in their collection, although adobe itself could be used as a source of last resort

Knowing the Adobe ID really the only accurate way to get a cover, but I haven't found any good ways to get it aside from searching a Library's Overdrive database. This Overdrive id only matches the ebook edition's ISBN, searching for an alternate ISBN will fail. For some reason advanced search works better with individual libraries than it does directly against Adobe servers....

Here's some info on how this works and examples of how the same cover could be found across multiple systems with the Adobe identifier. Using 'On the Road' by Jack Kerouac as an example:

To get the Overdrive ID one can just do an advanced search on an individual library's collection (Chicago in this case):
http://overdrive.chipublib.org/82DC6...BANGSearch.dll

The request for all the libraries is always to BANGSearch.dll, though the hostname and url prefix chanes with each library. The actual query is in the request content body:
Code:
Title=On+the+Road&Creator=Jack+Kerouac&Keyword=&ISBN=&Format=&Language=&Publisher=&Subject=&Award=&CollDate=&PerPage=10&Sort=SortBy%3Dtitle
These arguments seem to be uniform across all Overdrive based libraries.

Scraping the result contents (unfortunately I don't see any way to do this without scraping) gives three unique Adobe Overdrive IDs:
ContentDetails.htm?ID=6609D1A7-98A4-4B41-B9A2-94BEE83CF861
ContentDetails.htm?ID=7FEC0594-4FAD-4CBC-83B3-7DAEBCC5B900
ContentDetails.htm?ID=D8414773-1865-4C6B-AEB3-F2CB05C84F16

The third one is the epub book, the first two are audiobooks. Ideally one could prioritize epub/mobi/pdf covers over audiobook covers. The Overdrive advanced search GUI allows formats to be specified, but it doesn't allow 'OR'. Not sure if it allows it under the hood, I haven't yet tried to construct my own queries.


We'll use the epub's ID - D8414773-1865-4C6B-AEB3-F2CB05C84F16 - with this the cover can be retrieved from any library with the book in it's collection.

Overdrive's own servers:Cleveland Public Library:Chicago Public Library:In all those cases the rest of the URL stays the same, just change the Overdrive ID and you get the specified book cover.


Next hurdle would be libraries with large collections to increase the likelihood of a result. I haven't found many libraries with more than 20,000 titles. Singapore, Cleveland, and Seattle are all around that mark for epub/mobi/pdf. Chicago, though I used it as an example isn't great with ~6000.



Openlibrary also maintains Overdrive IDs for ebook editions, but not sure how universally they do it, and thus far it seems like more scraping would be required to get to the desired result, which isn't ideal:
http://openlibrary.org/books/OL24273691M/On_the_Road

Last edited by ldolse; 03-01-2011 at 05:34 AM.
ldolse is offline   Reply With Quote
Old 02-20-2011, 07:53 AM   #2
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 5,644
Karma: 1137414
Join Date: Jan 2010
Location: France
Device: Many android devices
Sounds like you have looked quite deeply into the question. I don't have strong feelings one way or the other about its usefulness, but then I add books to my library very slowly and don't use much of the metadata downloading tools.

There seems to be a fair amount in common between what you are proposing and kiwidude's LibraryThing plugin. Both require finding obscure internal IDs, both require scraping, and both require storing some additional information about the book. Perhaps he will come along and enlighten us.
chaley is offline   Reply With Quote
 
Enthusiast
Old 02-20-2011, 08:25 AM   #3
ldolse
Wizard
ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.
 
Posts: 1,337
Karma: 123455
Join Date: Apr 2009
Location: Malaysia
Device: PRS-650, iPhone
I like the convenience of Calibre just downloading the cover, but generally try to insert the best cover I can find, which usually isn't LT/Amazon. Just trying to save myself the steps of manually searching if I can.

I'm not doing massive bulk downloads myself either, but based on the discussions I've seen around LibraryThing I'm guessing there are a fair number of users that are doing exactly that.

Side note - I figured out why search.overdrive.com wasn't working with advanced search - their application expects spaces to be replaced by '+' in author names/book titles, which you can see in my example previously with the library query. Overdrive's own advanced search is broken and escaping the spaces with %20, which causes the lookup to fail.
ldolse is offline   Reply With Quote
Old 02-20-2011, 01:42 PM   #4
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,129
Karma: 5381911
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
calibre already uses openlibrary with ISBNs for covers. You can handle special identifiers by simply having an in memory cache that maps ISBNs to the identifiers. See the amazon plugin for an example which maps isbns to asin and stores the result in a global cache.

Currently cover download is on a first result basis, i.e. as soon as a particular source returns a cover, calibre uses it, it does not wait to get covers from all sources and then pick the best one. Doing so is possible in principle but it means that the download will become much slower.
kovidgoyal is online now   Reply With Quote
Old 02-20-2011, 09:05 PM   #5
ldolse
Wizard
ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.
 
Posts: 1,337
Karma: 123455
Join Date: Apr 2009
Location: Malaysia
Device: PRS-650, iPhone
Does Calibre try OpenLibrary first, then Amazon? Would it be more straightforward to rank the cover download plugins so that higher quality sources are tried before lower quality ones? Or are you saying all potential sources are tried at once?

Unfortunately any ISBN won't map to the Overdrive ID, it has to be the specific ISBN of the Overdrive edition. That said, I think the scraping could still be done and mapped as you described, I'll check out the Amazon code.
ldolse is offline   Reply With Quote
Old 02-20-2011, 09:15 PM   #6
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,228
Karma: 1334002
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
Quote:
Originally Posted by chaley View Post
There seems to be a fair amount in common between what you are proposing and kiwidude's LibraryThing plugin. Both require finding obscure internal IDs, both require scraping, and both require storing some additional information about the book. Perhaps he will come along and enlighten us.
I think you meant my Goodreads plugin rather than LibraryThing

As Kovid has responded there is little I can add. I'm in a holding pattern on any further changes to the Goodreads covers and metadata downloads plugins until Kovid is done with all his metadata API rewrite stuff. No rush since what is there still seems to work in the meantime. Absolutely there are a number of similar issues though.
kiwidude is offline   Reply With Quote
Old 02-20-2011, 09:52 PM   #7
JSWolf
Resident Curmudgeon
JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.
 
JSWolf's Avatar
 
Posts: 37,701
Karma: 18475502
Join Date: Nov 2006
Location: Roslindale, Massachusetts
Device: Sony Reader PRS-650, iPad, nook STR
Most covers on Overdrive are at 510x680. Incorrect aspect ratio. So no, Overdrive is a really poor source for covers. MS Reader specifications say the cover should be 510x60 to fully fill the MS Reader window. But I find this just wrong as a lot of covers to look off like that.

Find a source that uses correct aspect ratio covers.
JSWolf is offline   Reply With Quote
Old 02-21-2011, 12:16 AM   #8
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,129
Karma: 5381911
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Quote:
Originally Posted by ldolse View Post
Does Calibre try OpenLibrary first, then Amazon? Would it be more straightforward to rank the cover download plugins so that higher quality sources are tried before lower quality ones? Or are you saying all potential sources are tried at once?
It's simultaneous. Ranking is a possibility, but I'd be uncomfortable hardcoding the ranking, instead make it use configurable, so the user can decide what source she would prefer.
kovidgoyal is online now   Reply With Quote
Old 02-21-2011, 01:25 AM   #9
ldolse
Wizard
ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.
 
Posts: 1,337
Karma: 123455
Join Date: Apr 2009
Location: Malaysia
Device: PRS-650, iPhone
I don't think aspect ratio is that big a deal, as Calibre's default settings makes that scale dynamically anyway. No offense, JSWolf, but I think you're among a relatively small group of users that pays attention to that, and anyway individual metadata plugins can be disabled.

That said, I agree with Kovid that allowing users to configure their ranking would make more sense. I'm not sure I'm ready to create the GUI required for allowing users to configure that, but I think I feel comfortable getting started on an Overdrive plugin at least.

Are there things coming with the metadata API rewrite that I should wait for? Not entirely sure what the plans are there.
ldolse is offline   Reply With Quote
Old 02-21-2011, 10:23 AM   #10
theducks
Grand Sorcerer
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 14,863
Karma: 5654321
Join Date: Aug 2009
Location: (The original) Silicon Valley, USA
Device: Galaxy Tab 2, Astak Pocket Pro, K4NT
Quote:
Originally Posted by ldolse View Post
I don't think aspect ratio is that big a deal, as Calibre's default settings makes that scale dynamically anyway. No offense, JSWolf, but I think you're among a relatively small group of users that pays attention to that, and anyway individual metadata plugins can be disabled.

That said, I agree with Kovid that allowing users to configure their ranking would make more sense. I'm not sure I'm ready to create the GUI required for allowing users to configure that, but I think I feel comfortable getting started on an Overdrive plugin at least.

Are there things coming with the metadata API rewrite that I should wait for? Not entirely sure what the plans are there.
I will take just about any aspect ratio, as long as it is maintained during re-scale. (I have been watching Letter boxed, Laserdisc movies since 1980. I don't mind black bars )
theducks is offline   Reply With Quote
Old 02-21-2011, 11:16 AM   #11
cybmole
Wizard
cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.cybmole ought to be getting tired of karma fortunes by now.
 
Posts: 2,867
Karma: 1163098
Join Date: Sep 2010
Device: Kobo aura HD, Kobo Arc, Kindle Fire HDX 8.9 , Kindle for PC
there are many sites which host high quality cd, dvd, pc game etc. covers, with uploads from their user community. I guess book covers will evolve similarly, in time.
cybmole is offline   Reply With Quote
Old 02-21-2011, 11:18 AM   #12
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 26,129
Karma: 5381911
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
The new framework is being developed in metadata/sources

Cover download is relatively simple, so it should be easy to port your code once the new framework is ready.
kovidgoyal is online now   Reply With Quote
Old 02-21-2011, 10:46 PM   #13
JSWolf
Resident Curmudgeon
JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.
 
JSWolf's Avatar
 
Posts: 37,701
Karma: 18475502
Join Date: Nov 2006
Location: Roslindale, Massachusetts
Device: Sony Reader PRS-650, iPad, nook STR
Quote:
Originally Posted by theducks View Post
I will take just about any aspect ratio, as long as it is maintained during re-scale. (I have been watching Letter boxed, Laserdisc movies since 1980. I don't mind black bars )
But will you take incorrect aspect ratio. I prefer the correct aspect ratio. In fact, a bug was recently fixed wit regard to aspect ratio that I mentioned.
JSWolf is offline   Reply With Quote
Old 02-21-2011, 10:47 PM   #14
JSWolf
Resident Curmudgeon
JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.
 
JSWolf's Avatar
 
Posts: 37,701
Karma: 18475502
Join Date: Nov 2006
Location: Roslindale, Massachusetts
Device: Sony Reader PRS-650, iPad, nook STR
Quote:
Originally Posted by cybmole View Post
there are many sites which host high quality cd, dvd, pc game etc. covers, with uploads from their user community. I guess book covers will evolve similarly, in time.
There is a website for book covers. But the problem is they are in JPG and over compressed. The artifacts are very easy to se which makes them useless.
JSWolf is offline   Reply With Quote
Old 02-23-2011, 10:23 PM   #15
ldolse
Wizard
ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.ldolse is an accomplished Snipe hunter.
 
Posts: 1,337
Karma: 123455
Join Date: Apr 2009
Location: Malaysia
Device: PRS-650, iPhone
What's the best way to debug metadata download plugins? Is there a simple way to do it from the CLI, or do I need just execute the metadata query from the GUI and watch what happens?
ldolse is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[Covers Plugin] Goodreads Covers **Deprecated** kiwidude Plugins 13 04-17-2011 05:09 PM
Is all Overdrive the same? CWatkinsNash General Discussions 3 12-28-2010 04:01 PM
Covers, covers and damn statistics (wait, I got that wrong). Moejoe Writers' Corner 86 11-29-2010 08:34 PM
Stop Using Overdrive Fat Abe General Discussions 19 09-11-2010 08:30 PM
Overdrive Overseas Honch Which one should I buy? 3 12-08-2009 08:21 AM


All times are GMT -4. The time now is 08:43 AM.


MobileRead.com is a privately owned, operated and funded community.