New metadata API in 0.8 questions - Page 2

kiwidude · 04-17-2011, 01:17 PM

Ahh, thx for the pickup on that 2nd regex Kovid - I had it just as a .replace method on my previous plugin and haven't started the testing yet to see I screwed it up

As for the parenthesis - well the logic has been in Goodreads for a while and there was about 700 downloads before I refreshed it recently without anyone beating me up over it

. I agree there will always be edge cases where it is a bad idea, but my hope is that is less than the other way around. I know for a fact that stuff like (2010) in a title (which becomes just "2010" causes a problem for goodreads. As does stuff like (omnibus), both of which are fairly common things for people having imported in their library. So "less is more" as far as that website is concerned.

Obviously if your entire title was in parenthesis then it's a problem, and maybe I should handle that special case.

EDIT: Just saw your additional post - thanks for that, just what I was after...

kovidgoyal · 04-17-2011, 01:21 PM

See my previous post.

kiwidude · 04-17-2011, 01:25 PM

I did - see my previous post

ldolse · 04-17-2011, 01:36 PM

It looks to me like download_cover() and identify() are being called nearly simultaneously, which unfortunately is killing my plugin because of Overdrive's screwed up cookie handling. download_cover() does happen a few seconds later, but this isn't nearly long enough, as there hasn't been enough time for the first call to identify() to populate all the caches so that download_cover() can get the cover from cache successfully.

edit... it looks like download_cover() doesn't get the identifier that's set the first time identify() is called.. that seems to be why the cache lookup is failing.

kovidgoyal · 04-17-2011, 01:36 PM

Email is sooo much better. I've committed the changes to title_tokens

kovidgoyal · 04-17-2011, 02:01 PM

@ldolse: The identifiers that download_cover will get depends on which result was selected as the final result. If that result comes from another plugin, then the identifiers will not be present when download_cover is called.

In such a case, you have to run identify from within download_cover in your plugin. Look at any of the builtin pugins for an example.

There is a slight optimization in that identify results are merged, so if your plugins' result is recognized by the download system as being the same as the final result that is picked, its identifiers will be merged. And so the identifiers will be present when download_cover is called, but this will not always happen.

ldolse · 04-17-2011, 02:21 PM

I was already running identify from the download_cover, the bit I missed was properly populating the isbn to identifier cache correctly with the cache_isbn_to_identifier() function. I just did that and now my cache lookups are working.

Is that function taking care of xisbn somewhere? I saw that you're still importing the xisbn function, but it looks like it's being taken care of outside of the plugins now, haven't followed through all the logic.

Won't there will still be a chance that the identify() [and in turn whatever web service] can get called twice with the same parameters for cases where there are no identifier's whatsoever in the initial query? Once to get the metadata, and a second time from inside download_cover(). Since there isn't any ISBN there's nothing to get a successful cache lookup from, and it will wind up going to the service.

I'm wondering if creating a hash on the title/author combo might make sense as a cache lookup key for this kind of case. With the new framework it's not going to be infinitely variable like it was in the old framework when all the title/author combos in isbndb were queried.

kovidgoyal · 04-17-2011, 02:28 PM

xisbn is used after all the plugins return their results. The results are merged. Th emerge first uses xisbn to get sets of related isbns and merges by those sets.

Then the merging happens for those results that have identical titles and authors.

So only if your result happens to fall through both those checks will there be a double identify.

kiwidude · 04-17-2011, 02:31 PM

Funnily enough my Goodreads Covers plugin had a cache for lookup by title/author for books that did not have an isbn. But after writing all the code for that version of the metadata api I found I couldn't actually do title/author searches, only isbn ones

.

Now that I can store Goodreads id I don't need a cache by anything else for my plugin - once you have hit a result the cache will key it by goodreads id and I can pull it back by that in future.

kovidgoyal · 04-17-2011, 02:34 PM

Quote:

Originally Posted by kiwidude

Funnily enough my Goodreads Covers plugin had a cache for lookup by title/author for books that did not have an isbn. But after writing all the code for that version of the metadata api I found I couldn't actually do title/author searches, only isbn ones

.

Now that I can store Goodreads id I don't need a cache by anything else for my plugin - once you have hit a result the cache will key it by goodreads id and I can pull it back by that in future.

It's a little more subtle than that. The problem is suppose your plugin returns result1 and the amazon plugin returns result2. Further suppose that these results are not merged as described in my previous post. Now suppose the user picks result2. Then the identifiers that are passed to download_cover() will come from result2, not result1. So the download_cover in your plugin will not be able to key off the cached goodreads id.

kiwidude · 04-17-2011, 02:41 PM

Ahhh, got you. I must confess I just saw the mention of caching lookups by title/author and it reminded me of the issue I had of all the books out there that don't have an ISBN.

Which it sounds like there would still be an issue for - unless the user actually picked the Goodreads result out of the set if I understand you correctly (since the books wouldn't have isbns to merge).

kovidgoyal · 04-17-2011, 02:46 PM

Yes, if your results dont have isbns, merging will use exact (case independent) title + author matches. Which will fail most of the time. So then you will end doing double identifies.

So I would highly recommend that you populate your results with isbns if at all possible.

kiwidude · 04-17-2011, 02:52 PM

Yeah I do grab it where I can. I just remember seeing books (and having edge cases thrown at me by users in the early days of testing) where the book only had an ASIN appearing on the Goodreads page. I guess it is more of the indie stuff where they are only publishing electronically and avoiding the ISBN costs which allegedly there is a bit of a trend on, but not being an author/publisher I confess to ignorance.

Of course even if I scraped the ASIN in that scenario (which I may do just for completeness if I can find some examples again) that still won't solve the problem if the only options for merging are ISBN or exactish title/author.

We do what we can

kovidgoyal · 04-17-2011, 03:29 PM

There's no reason the title/author merge cannot be supplemented by an asin (or indeed any other identifier merge). If you think that will help, then I can implement an ASIN based merge as well.

kiwidude · 04-17-2011, 03:50 PM

I guess perhaps I need to really understand more about the merging and the implications. Right now with all my testing etc I only have the Goodreads plugin enabled. And to be honest other than the odd small cover (but I usually manually replace covers anyway) I don't bother turning the others on in normal use anyways. As none of the other sources for my fiction books at least seem to offer much advantage.

So in the case of Amazon I'm not sure I would even have it enabled. As the Goodreads database consumes Amazon and Barnes & Noble and a few others. So provided I don't then there is no merge issue. I might have one of the others enabled but they won't have an ASIN or a goodreads id, so the merge of other identifiers wouldn't achieve anything. I think?

BTW, I am confused now about pubdate. My plugin is retrieving a specific value. But a different value is getting written into the edit metadata dialog. Are you doing some sort of pubdate lookup when you do your XISBN stuff or something?

I ask because one of the other features I liked about Goodreads was its "first published" date that I scrape and use. However I anticipated there are people out there who are very OCD about such things and want the date of their particular edition. So I put an option in my plugin to bring that back instead, and that is the date shown in the log, but a different (earlier) date is getting put into the record.

Is that a subtle secret addition you made to 0.8? In which case I will rip out my changes...

04-17-2011, 01:36 PM	#19
ldolse Wizard Posts: 1,337 Karma: 123455 Join Date: Apr 2009 Location: Malaysia Device: PRS-650, iPhone	It looks to me like download_cover() and identify() are being called nearly simultaneously, which unfortunately is killing my plugin because of Overdrive's screwed up cookie handling. download_cover() does happen a few seconds later, but this isn't nearly long enough, as there hasn't been enough time for the first call to identify() to populate all the caches so that download_cover() can get the cover from cache successfully. edit... it looks like download_cover() doesn't get the identifier that's set the first time identify() is called.. that seems to be why the cache lookup is failing. Last edited by ldolse; 04-17-2011 at 01:55 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Metadata scraper plugin api	kiwidude	Development	5	03-06-2011 11:58 AM
2 metadata questions	bucsie	Calibre	4	12-20-2010 05:47 AM
Downloading Metadata - couple of questions	sadievan	Calibre	6	12-14-2010 09:27 PM
More questions on Metadata	crutledge	Sigil	16	10-23-2010 07:27 PM
calibre now uses the Google Books API to get metadata	kovidgoyal	Calibre	9	03-23-2009 09:36 PM

04-17-2011, 01:17 PM	#16
kiwidude Calibre Plugins Developer Posts: 4,637 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	Ahh, thx for the pickup on that 2nd regex Kovid - I had it just as a .replace method on my previous plugin and haven't started the testing yet to see I screwed it up As for the parenthesis - well the logic has been in Goodreads for a while and there was about 700 downloads before I refreshed it recently without anyone beating me up over it . I agree there will always be edge cases where it is a bad idea, but my hope is that is less than the other way around. I know for a fact that stuff like (2010) in a title (which becomes just "2010" causes a problem for goodreads. As does stuff like (omnibus), both of which are fairly common things for people having imported in their library. So "less is more" as far as that website is concerned. Obviously if your entire title was in parenthesis then it's a problem, and maybe I should handle that special case. EDIT: Just saw your additional post - thanks for that, just what I was after...

04-17-2011, 01:21 PM	#17
kovidgoyal creator of calibre Posts: 43,871 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	See my previous post.

04-17-2011, 01:25 PM	#18
kiwidude Calibre Plugins Developer Posts: 4,637 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	I did - see my previous post

04-17-2011, 02:01 PM	#21
kovidgoyal creator of calibre Posts: 43,871 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	@ldolse: The identifiers that download_cover will get depends on which result was selected as the final result. If that result comes from another plugin, then the identifiers will not be present when download_cover is called. In such a case, you have to run identify from within download_cover in your plugin. Look at any of the builtin pugins for an example. There is a slight optimization in that identify results are merged, so if your plugins' result is recognized by the download system as being the same as the final result that is picked, its identifiers will be merged. And so the identifiers will be present when download_cover is called, but this will not always happen.

04-17-2011, 02:21 PM	#22
ldolse Wizard Posts: 1,337 Karma: 123455 Join Date: Apr 2009 Location: Malaysia Device: PRS-650, iPhone	I was already running identify from the download_cover, the bit I missed was properly populating the isbn to identifier cache correctly with the cache_isbn_to_identifier() function. I just did that and now my cache lookups are working. Is that function taking care of xisbn somewhere? I saw that you're still importing the xisbn function, but it looks like it's being taken care of outside of the plugins now, haven't followed through all the logic. Won't there will still be a chance that the identify() [and in turn whatever web service] can get called twice with the same parameters for cases where there are no identifier's whatsoever in the initial query? Once to get the metadata, and a second time from inside download_cover(). Since there isn't any ISBN there's nothing to get a successful cache lookup from, and it will wind up going to the service. I'm wondering if creating a hash on the title/author combo might make sense as a cache lookup key for this kind of case. With the new framework it's not going to be infinitely variable like it was in the old framework when all the title/author combos in isbndb were queried.

04-17-2011, 02:28 PM	#23
kovidgoyal creator of calibre Posts: 43,871 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	xisbn is used after all the plugins return their results. The results are merged. Th emerge first uses xisbn to get sets of related isbns and merges by those sets. Then the merging happens for those results that have identical titles and authors. So only if your result happens to fall through both those checks will there be a double identify.

04-17-2011, 02:31 PM	#24
kiwidude Calibre Plugins Developer Posts: 4,637 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	Funnily enough my Goodreads Covers plugin had a cache for lookup by title/author for books that did not have an isbn. But after writing all the code for that version of the metadata api I found I couldn't actually do title/author searches, only isbn ones . Now that I can store Goodreads id I don't need a cache by anything else for my plugin - once you have hit a result the cache will key it by goodreads id and I can pull it back by that in future.

04-17-2011, 02:41 PM	#26
kiwidude Calibre Plugins Developer Posts: 4,637 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	Ahhh, got you. I must confess I just saw the mention of caching lookups by title/author and it reminded me of the issue I had of all the books out there that don't have an ISBN. Which it sounds like there would still be an issue for - unless the user actually picked the Goodreads result out of the set if I understand you correctly (since the books wouldn't have isbns to merge).

04-17-2011, 02:46 PM	#27
kovidgoyal creator of calibre Posts: 43,871 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Yes, if your results dont have isbns, merging will use exact (case independent) title + author matches. Which will fail most of the time. So then you will end doing double identifies. So I would highly recommend that you populate your results with isbns if at all possible.

04-17-2011, 02:52 PM	#28
kiwidude Calibre Plugins Developer Posts: 4,637 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	Yeah I do grab it where I can. I just remember seeing books (and having edge cases thrown at me by users in the early days of testing) where the book only had an ASIN appearing on the Goodreads page. I guess it is more of the indie stuff where they are only publishing electronically and avoiding the ISBN costs which allegedly there is a bit of a trend on, but not being an author/publisher I confess to ignorance. Of course even if I scraped the ASIN in that scenario (which I may do just for completeness if I can find some examples again) that still won't solve the problem if the only options for merging are ISBN or exactish title/author. We do what we can

Advert

Advert

04-17-2011, 03:29 PM	#29
kovidgoyal creator of calibre Posts: 43,871 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	There's no reason the title/author merge cannot be supplemented by an asin (or indeed any other identifier merge). If you think that will help, then I can implement an ASIN based merge as well.

04-17-2011, 03:50 PM	#30
kiwidude Calibre Plugins Developer Posts: 4,637 Karma: 2162064 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	I guess perhaps I need to really understand more about the merging and the implications. Right now with all my testing etc I only have the Goodreads plugin enabled. And to be honest other than the odd small cover (but I usually manually replace covers anyway) I don't bother turning the others on in normal use anyways. As none of the other sources for my fiction books at least seem to offer much advantage. So in the case of Amazon I'm not sure I would even have it enabled. As the Goodreads database consumes Amazon and Barnes & Noble and a few others. So provided I don't then there is no merge issue. I might have one of the others enabled but they won't have an ASIN or a goodreads id, so the merge of other identifiers wouldn't achieve anything. I think? BTW, I am confused now about pubdate. My plugin is retrieving a specific value. But a different value is getting written into the edit metadata dialog. Are you doing some sort of pubdate lookup when you do your XISBN stuff or something? I ask because one of the other features I liked about Goodreads was its "first published" date that I scrape and use. However I anticipated there are people out there who are very OCD about such things and want the date of their particular edition. So I put an option in my plugin to bring that back instead, and that is the date shown in the log, but a different (earlier) date is getting put into the record. Is that a subtle secret addition you made to 0.8? In which case I will rip out my changes...