Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Development

Notices

Reply
 
Thread Tools Search this Thread
Old 01-04-2014, 08:56 PM   #1
geekraver
Addict
geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.
 
Posts: 344
Karma: 1010197
Join Date: Jul 2006
Location: Redmond
Device: iPad Pro,Nexus 7,Kindle Paperwhite,Kobo H2O Ed 2
Metadata retrieval using series/series index

Hi all

I am starting to create a new Calibre library for magazines that I have like Hacker Monthly.

My intention with these was to set the title to "Hacker Monthly" for each one, and identify the individual issues using the series index. I wanted to write a metadata download plugin to pull the table of contents off the website and use it to set the comments field (and maybe also tags, which I would generate from the contents with something like https://github.com/apresta/tagger).

The problem I have is it looks like metadata downloader plugins get passed title and author to identify() but not series information. Is my understanding here correct? And what would be the best workaround then? I can't use title and author necessarily to query Calibre itself to get the current metadata if these are the same for all issues. Would it be best then to keep issue numbers in the titles?

Thanks!
Gram
geekraver is offline   Reply With Quote
Old 01-04-2014, 09:05 PM   #2
geekraver
Addict
geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.
 
Posts: 344
Karma: 1010197
Join Date: Jul 2006
Location: Redmond
Device: iPad Pro,Nexus 7,Kindle Paperwhite,Kobo H2O Ed 2
It's a bit of an ugly hack but another way I thought of doing this was to just have a Python script that modifies the OPF files, and then use a consistency check to get the DB to update.

OTOH it looks like the title and author are necessarily unique per "book", or the issues will get folded together as dups. If that is the case, then I guess I have to leave the issue number in the title and this problem goes away.
geekraver is offline   Reply With Quote
Advert
Old 01-04-2014, 09:26 PM   #3
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 34,128
Karma: 10261492
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
The metadata download plugins also get passed the identifiers, so create an identifier to store the issue number, something like this:

issue:101

or just a url directly,

url:http://whatever
kovidgoyal is offline   Reply With Quote
Old 01-05-2014, 01:31 AM   #4
geekraver
Addict
geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.
 
Posts: 344
Karma: 1010197
Join Date: Jul 2006
Location: Redmond
Device: iPad Pro,Nexus 7,Kindle Paperwhite,Kobo H2O Ed 2
Thanks Kovid. I'll look into that approach later. Right now I'm using the title, but I'm not getting very far at all. It doesn't seem my plugin is being called at all. It is loaded successfully, and shows up as a metadata source when downloading metadata, but I see no output from the print statements. My __init__.py is below; can you see what I might be doing wrong?

Code:
from calibre.ebooks.metadata.sources.base import Source

class Magazines(Source):

    name = 'Magazines'
    description = _('Download metadata for various magazines')
    version = (0, 1, 0)
    author = 'Graham Wheeler'
    minimum_calibre_version = (0, 9, 0)

    capabilities = frozenset(['identify'])
    touched_fields = frozenset ([
        'comments'
        ])
    has_html_comments = True


    def identify(self, log, result_queue, abort, title=None, authors=None,
                 identifiers={}, timeout=30):

        print 'In identify: %s ' % title
        from calibre_plugins.magazines.worker import Worker

        # Search for results in different thread, as searching takes time and blocks...
        worker = Worker(result_queue, log, title, authors, self, cfg.getOption(cfg.KEY_MAX_DOWNLOADS))
        worker.start()

        while not abort.is_set():
            worker.join(0.2)

            if not worker.is_alive():
                break

        print 'Out of worker: %s' % result_queue.qsize()

        return None
geekraver is offline   Reply With Quote
Old 01-05-2014, 01:52 AM   #5
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 34,128
Karma: 10261492
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
You did enable it in Preferences->Metadata download didn't you?
kovidgoyal is offline   Reply With Quote
Advert
Old 01-05-2014, 01:54 AM   #6
geekraver
Addict
geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.
 
Posts: 344
Karma: 1010197
Join Date: Jul 2006
Location: Redmond
Device: iPad Pro,Nexus 7,Kindle Paperwhite,Kobo H2O Ed 2
Yes, it is enabled. I tried disabling all other sources too. All I see in the output at the end of the debug run is:

Job: 1 Download metadata for 1 books finished
Starting job: Download metadata for 1 books
Download complete, with 1 failures

The order of those messages is a bit weird.

Last edited by geekraver; 01-05-2014 at 02:11 AM.
geekraver is offline   Reply With Quote
Old 01-05-2014, 07:33 AM   #7
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 34,128
Karma: 10261492
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
You want to look at the metadata download log not the debug log. CLick the view log button in the metadtata download dialog.
kovidgoyal is offline   Reply With Quote
Old 01-05-2014, 12:00 PM   #8
geekraver
Addict
geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.
 
Posts: 344
Karma: 1010197
Join Date: Jul 2006
Location: Redmond
Device: iPad Pro,Nexus 7,Kindle Paperwhite,Kobo H2O Ed 2
Do you mean the 'View Details'? That shows just:

Hacker Monthly #15 (Failed metadata)
geekraver is offline   Reply With Quote
Old 01-05-2014, 12:03 PM   #9
geekraver
Addict
geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.
 
Posts: 344
Karma: 1010197
Join Date: Jul 2006
Location: Redmond
Device: iPad Pro,Nexus 7,Kindle Paperwhite,Kobo H2O Ed 2
BTW I put this at the start of identify:

print 'In identify: %s ' % title
log.info('In identify: %s' % title)

I don't see it either way. I wasn't sure what to use because in the docs on debugging plugins you say to use print, but in the docs for identify you say the log object is for debugging messages.
geekraver is offline   Reply With Quote
Old 01-05-2014, 09:33 PM   #10
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 34,128
Karma: 10261492
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
No, I mean the *View log* button on the *metadata download dialog* the one you get when you click download metadata in the edit metadata dialog.
kovidgoyal is offline   Reply With Quote
Old 01-06-2014, 12:06 AM   #11
geekraver
Addict
geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.
 
Posts: 344
Karma: 1010197
Join Date: Jul 2006
Location: Redmond
Device: iPad Pro,Nexus 7,Kindle Paperwhite,Kobo H2O Ed 2
Ah, I see. I was initiating the metadata download by right clicking and using the context menu. There is no View Log in this case. You have to open the Edit Metadata dialog and do a Download Metadata from within that to get the logs.
geekraver is offline   Reply With Quote
Old 01-06-2014, 12:11 AM   #12
geekraver
Addict
geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.
 
Posts: 344
Karma: 1010197
Join Date: Jul 2006
Location: Redmond
Device: iPad Pro,Nexus 7,Kindle Paperwhite,Kobo H2O Ed 2
Got it working now. Thanks!

So which is better for diagnostics - using log or print?
geekraver is offline   Reply With Quote
Old 01-06-2014, 12:17 AM   #13
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 34,128
Karma: 10261492
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Metadata download is run in a separate process, therefore print will not work for it, you need to use log. And there most definitely is a view log button for bulk metadata downloads. It is on the confirmation dialog that pops up after the download completes.
kovidgoyal is offline   Reply With Quote
Old 01-06-2014, 12:33 AM   #14
geekraver
Addict
geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.geekraver ought to be getting tired of karma fortunes by now.
 
Posts: 344
Karma: 1010197
Join Date: Jul 2006
Location: Redmond
Device: iPad Pro,Nexus 7,Kindle Paperwhite,Kobo H2O Ed 2
I wrote a long response to your earlier post when your first mentioned this, detailing everything I saw, before I tried the metadata download from the edit dialog ; after that I discarded the details. But there was no view log. Maybe if some succeed there would be but if they all fail all you get is an error popup with three buttons: copy to clipboard, view details and OK. And the view details here doesn't show the log, just the one line failure error message.
geekraver is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
series index doesn't get all digits Thorsten90 Library Management 0 07-21-2013 12:52 AM
Variable Series Index Field Gelina Calibre 15 06-04-2013 07:06 AM
Plugboard "Metadata: Show series [series index] - title as title (Kindle)" Deep Cover Library Management 6 11-30-2012 05:17 PM
Setting series index in bulk metadata search&replace bubak Calibre 4 12-19-2010 04:04 PM
Command Line problems: ebook-convert, ebook-meta with tags: --series, --series-index omnivorous Calibre 4 11-07-2010 02:42 PM


All times are GMT -4. The time now is 05:45 AM.


MobileRead.com is a privately owned, operated and funded community.