Using Overdrive for covers?

[kovidgoyal]

This is what I get:

calibre-debug -c "from calibre.ebooks.metadata.sources.base import Source; s = Source(None); print list(s.get_author_tokens(['J. R. R. Tolkien']))"

['Tolkien']

Don't worry about the None it's meaningless (see customize.__init__ for the None)

[ldolse]

Weird - alright - I'll play around with it some more and see where mine is going wrong.

Found the problem - User error - which was what I was starting to suspect when the print statements weren't working. Development environment variable was screwed up in the CLI instance I was debugging from.

[ldolse]

Just got back into developing this plugin. I've got the plugin working pretty well for social metadata/cover download, but I'd like to move it to the discovery phase as discussed previously, as right now the has_cover lookup (pretty sure it's this function) bombards Overdrive with potentially dozens of simultaneous requests for book title variations and ISBNs - since this is done simultaneously it's done too late to cache the results properly and avoid abusing the server.

I've got the plugin using the new set_identifier function to set the id for Overdrive in the identifiers table, but right now that's not something that's available for use in fetch.py or covers.py, both still rely on calling ISBN the old fashioned way, and it wasn't readily apparent to me how to change either of those to check for the existence of some other type of identifier.

I know a lot of this stuff using identifiers isn't quite ready for prime-time use and is tied to the larger metadata enhancements targeted for a later release. Is there something simple that can be done with the current code easily, or is this something I should wait for a while longer?

Ideally I'd like the logic to be something like this:

1. Check Overdrive during the discovery phase with the configured Title/Author (overdrive Identifier is optional if found in the metadata)
2. Populate the basic metadata available from this initial lookup (currently Title/Subtitle, Authors, Series/Series Index, Publisher, Cover URL, social metadata URL, Overdrive Identifier) - some of this would just go into a cache based on Overdrive Identifier (vs. ISBN which is used today).
3. Get the Cover URL from the cache if Overdrive is configured for covers.
4. Get Social Metadata from the url in the cache if configured for Social Metadata (tags, description, pub date, ebook ISBN, language)

[kovidgoyal]

This is exactly how the new plugins work. I've completed the google and amazon plugins with caches for cover url and isbn to identifier mappings, have a look.

There isn't really a way to accomplish this cleanly in the current framework

[ldolse]

I saw that you've been doing a lot of work on those plugins lately, but they're still not used by default, correct? It looked to me like prep work for a later switchover to the new framework. I don't mind waiting longer, just checking.

[kovidgoyal]

No they aren't used yet.

[ldolse]

Cool, I'll mimic the logic in those plugins a little more closely and check it in with the new sources. I've been tweaking the author/title token functions as well.

[ldolse]

Getting around to porting this now that the new framework is usable enough for general testing.

Does the new framework differentiate between 'identify' and social metadata? Just asking because thus far all the plugins list only 'identify' and/or 'cover' in the capabilities.

capabilities = frozenset(['identify', 'cover'])

The way Overdrive works there are three ways to get data:
'identify' - Search overdrive and get a list object back with various metadata.
'cover' - comes from the same search as above, but use a separate function to populate the cover url cache from those results.
'social metadata' - Or whatever it's equivalent might be considered in the new framework - go to the actual book page and scrape for further details not included in the identify results.

I would think that for many metadata providers this might be the way things work, even if it requires scraping - fuzzy search returns a list of results which already contain some metadata, full metadata by going to the book identifier's actual page.

Scraping the book details page is actually the slowest in the case of Overdrive, so I was thinking to make it optional and disabled by default. Not sure if the listed capabilities is the way this will be managed or if I should configure similar to .7.x metadata plugins.


The other way to do it, based on reviewing the new GUI, is to default disable the specific fields that require the book record to be loaded. This looks like the 'touched_fields' configuration in the plugin, but that only adds them to the list - will it be possible to default disable them?

[kovidgoyal]

There's no social metadata anymore. The identify method should return all fields the source knows about. You can default disable fields by overriding the __init__ method in your plugin like this

def __init__(self, *args, **kwargs):
   SourcePlugin.__init__(self, *args, **kwargs)
   self.prefs.defaults['ignore_fields'] =[list of fields from touched_fields]

Though I would suggest using threading to make the detailed queries, see the Amazon plugin for an example.

[ldolse]

Originally Posted by kovidgoyal

There's no social metadata anymore. The identify method should return all fields the source knows about. You can default disable fields by overriding the __init__ method in your plugin like this

def __init__(self, *args, **kwargs):
   SourcePlugin.__init__(self, *args, **kwargs)
   self.prefs.defaults['ignore_fields'] =[list of fields from touched_fields]

global name 'SourcePlugin' is not defined... Not sure where it's supposed to be defined, but I'm pretty sure you meant for me to use that specific string.

Is the list of field's the user is actually requesting available for testing inside of identify?

[kiwidude]

I think that was meant to be "Source" not "SourcePlugin" - as this __init__ method would be on your own plugin inheriting from Source.

[ldolse]

Cool, 'Source' worked. The relevant fields are disabled by default now. Not sure if I have the ability to test whether they're enabled/disabled at the time the plugin is called though, which might make the exercise moot.

[JSWolf]

Overdrive uses the wrong aspect ratio for covers. So using Overdrive for covers is a bad idea. They use 510x680 which is too wide.

[ldolse]

I think we've been over this - Cover download is configurable, you can configure Calibre to not use Overdrive, or to make the Overdrive cover download the lowest priority.

I personally have no problem with the aspect ratio, and I suspect the majority of users probably feel similarly. 510 x 680 is often the highest resolution out there for a lot of covers. If I scale that up linearly it's 600 x 770 pixels on an actual screen 800x600 screen, not sure I'm that fussed about 15 pixels of empty space on the top and bottom for users who force the existing aspect ratio, I'd much rather have the increased resolution.

On top of this Calibre's default settings for epub don't use the cover's actual aspect ratio - instead it sticks it in an SVG which will force it to fill the reader's screen. I realize you don't use this default setting, but the majority of Calibre users do, since the majority of them don't know how to change it.

On a side note, I don't see any issue whatsoever around aspect ratio for the thumbnails Calibre uploads to my Sony, so it may be fixing the aspect ratio using it's image libraries on the fly for many scenarios, someone else may be able to provide more detail on that.

[chaley]

Originally Posted by ldolse

On a side note, I don't see any issue whatsoever around aspect ratio for the thumbnails Calibre uploads to my Sony, so it may be fixing the aspect ratio using it's image libraries on the fly for many scenarios, someone else may be able to provide more detail on that.

The sony driver has an option to either preserve the existing aspect ratio in a container 217 high, or to convert it to the sony 'standard' of 217x168. The option is default True: preserve aspect ratio.