Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre

Notices

Reply
 
Thread Tools Search this Thread
Old 01-23-2010, 03:20 PM   #1
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
How To: Getting summary metadata and images, not tags, author, title

I add books to my library as soon as I get them. Usually, I take the time to make sure the title and author are right, but I don't always have time to download the metadata and covers. I particularly like getting the summary info.

Later I want to edit metadata, add my personal tags, pick up covers, etc. I open up the edit metadata screen, and tell it to fetch, and I see 3 or four likely descriptions. The problem is always that the title or author may be in all caps. Or the title will have the series name in it, even though the other books in that series don't have the series in the title. If I want to pick up the summary description, it writes over my correct author/title. I've particularly noticed that the title is often incorrectly capitalized. I then have to go back and fix the author and title that was correct when I started!

Another issue occurs when I want to look at several different cover images. To do that you have to choose different results from the fetch table, but each choice writes over the author and title again. I know some people hate the multiple tags - this lets you stop picking them up,while still picking up ratings and images, etc.

The one thing that is really, really nice about open source software is you have control. Now I know that not many of us are programmers, and even fewer are familiar with Python, but it really is easy to make small changes that improve things for special cases. You just need to take a few steps to get the source code, and use Notepad (Windows) to type "# " in front of a few lines to prevent Calibre from overwriting the existing data with the new data. You can pick up only the summary, or even pick up summaries from 3 versions of the book. (Summaries are cumulative - they don't write over your comments or the summaries already in there.)

Here's how I fixed it. First, read this on Setting up a calibre development environment¶

It may seem tricky, but truthfully, all you have to do is download Bazaar, run it once to tell it where you want to put your source code, and set one environment variable in Windows to tell Calibre where you put it. Then you're ready to take control. As long as that environment variable is set, Calibre will use your changes. As soon as you delete or change the environment variable from "CALIBRE_DEVELOP_FROM" to something else (like "CALIBRE_NOT_DEVELOP_FROM") you hide your revised source code and Calibre goes back to its normal behavior.

Now all you need to know is what changes to make. Go to where you told Bazaar to put your source code ( I use a folder called src inside the Calibre folder) and find the file called calibre\gui2\dialogs\metadata_single.py

starting at line 575 you see:

Code:
                        else:
                            book.tags = []
                        # self.title.setText(book.title)
                        # self.authors.setText(authors_to_string(book.authors))
                        # if book.author_sort: self.author_sort.setText(book.author_sort)
                        if book.publisher: self.publisher.setEditText(book.publisher)
                        if book.isbn: self.isbn.setText(book.isbn)
                        if book.pubdate:
                            d = book.pubdate
                            d = d - self.local_timezone_offset
                            self.pubdate.setDate(QDate(d.year, d.month, d.day))
                        summ = book.comments
                        if summ:
                            prefix = unicode(self.comments.toPlainText())
                            if prefix:
                                prefix += '\n'
                            self.comments.setText(prefix + summ)
                        if book.rating is not None:
                            self.rating.setValue(int(book.rating))
                        if book.tags:
                            self.tags.setText(', '.join(book.tags))
Just comment out (that means put "# " in front of) any line that controls something you don't want picked up. In the code above, I've commented out the title, author and author_sort and left everything else. There are a few lines that are inside "if" blocks. For those, you just need to comment out the if line and all indented lines below it.

I know it can be scary to play with the source code, if you've never done it, but all you have to do is delete or change the environment variable, as above, and all your changes stop having any effect on Calibre. It's a lot easier than writing a regex expression. This little change made it possible for me to quickly look at several different images, pick up multiple descriptions, and still get the correct publisher, pubdate on the last selection without having to constantly fix the author and title.

Have fun.

Last edited by Starson17; 01-25-2010 at 09:37 AM.
Starson17 is offline   Reply With Quote
Old 01-23-2010, 11:51 PM   #2
mezme
Connoisseur
mezme began at the beginning.
 
Posts: 58
Karma: 10
Join Date: Dec 2009
Device: PRS700
Perfect - Thanks Starson!!!! This was an awesome and quick solution to the metadownload issue!
mezme is offline   Reply With Quote
Old 01-24-2010, 10:00 AM   #3
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by mezme View Post
This was an awesome and quick solution to the metadownload issue!
Thanks for the kind comment. It was really useful to me, so I thought I'd share.

Remember that it will use your source code as long as the environment variable is set. If you upgrade the executables the usual way, the upgrade won't be seen if you don't also update the source code or turn off the environment variable. Of course, it's a handy way to keep an older version around. You can switch it on only when you're going to be working on the metadata, then switch it off.

I considered trying to write a proposed modification to the main program so you could turn on/off the pickup of each piece of metadata without having to muck around in the source code each time. It's certainly possible to do it, but I'm not sure if it would be accepted or even if it's needed by others. (On top of which, I don't currently have the skill to do it correctly.)

Plus, you'd probably end up adding some checkboxes on the fetch screen that you'd have to keep clicking for each book (slowing the process down), or semi-hidden default checkboxes that might generate lots of support headaches if not used correctly. This way works great and if you are adventurous enough to play with the source, you're probably going to be able to avoid most of the problems.
Starson17 is offline   Reply With Quote
Old 01-25-2010, 02:55 AM   #4
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
Quote:
Originally Posted by Starson17 View Post
I considered trying to write a proposed modification to the main program so you could turn on/off the pickup of each piece of metadata without having to muck around in the source code each time. It's certainly possible to do it, but I'm not sure if it would be accepted or even if it's needed by others. (On top of which, I don't currently have the skill to do it correctly.)
It would be accepted and it is needed but skills I can't help you with.

Quote:
Originally Posted by Starson17 View Post
Plus, you'd probably end up adding some checkboxes on the fetch screen that you'd have to keep clicking for each book (slowing the process down), or semi-hidden default checkboxes that might generate lots of support headaches if not used correctly. This way works great and if you are adventurous enough to play with the source, you're probably going to be able to avoid most of the problems.
Currently options for selecting what metadata to grab are under the plugins. Specifically metadata download plugins. I go to each of the three plugins now and uncheck tags so I don't end up with a million tags. Having boxes for author and title would be a nice touch.

Perhaps you could take your original post and open up a Calibre enhancement ticket clearly stating the goal.

Thanks for the info. If it ever gets implemented I'm sure many folks will use this feature.
DoctorOhh is offline   Reply With Quote
Old 01-25-2010, 10:22 AM   #5
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by dwanthny View Post
It would be accepted and it is needed but skills I can't help you with.
That's OK, I don't mind picking up the skills myself, they just take time, and I'm not there yet.

Quote:
Currently options for selecting what metadata to grab are under the plugins. Specifically metadata download plugins. I go to each of the three plugins now and uncheck tags so I don't end up with a million tags. Having boxes for author and title would be a nice touch.
Thank you for this comment! I've been digging through the code and through the main interface, but it hadn't really dawned on me that the customizations for these plugins is where these options for title/author would be expected to be found. Now that I've looked at them, I see that with the current design, this is exactly where they would be. In fact, I can see that the location of the code change I posted above affects only downloads from the single_metadata screen, not multiple automated metadata updates, whereas if I made the change at the plugins (as Kovid suggested) it would have effect for both single and multiple updates.

Quote:
Perhaps you could take your original post and open up a Calibre enhancement ticket clearly stating the goal.
I thought of that, but I didn't want to ask someone else to do work for me when the code is available for me to do it myself. Plus, I'm still exploring the nooks and crannies of the interface (as in I hadn't even understood that the plugin customization is where this would go) so it's still premature.

Quote:
Thanks for the info. If it ever gets implemented I'm sure many folks will use this feature.
Thank you for the info! One reason I posted here was to gauge whether there was any interest in this feature. Another was to try to elicit comments for how it should work. Your post has helped with both. I will probably keep on hacking around in the code, just to familiarize myself, then maybe I will be in a position to make a very clear improvement request, with some possible code that someone with greater skill could look over and correct or improve (as Kovid did with my last tiny proposed code segment).

I actually spent a lot of time in the plugin module code for the fetchers, but it was heavy going until I read some more on Python and PyQT4.

On your point of "clearly stating the goal" that's always half the battle. I'm not even sure I full understand the way Google, ISBNDB and Amazon plugins currently interact. I had thought that two of them were used to find the ISBN, and the third was used for everything else. However, it's clear from the plugin customizations that all three provide ratings, descriptions and tags.

I think what would work well is to set a default in these plugins that allows the title or author to be overwritten (or not) during a metadata download and then allow the default to be overridden with a checkbox at the single_metadata download screen.

This post is long and rambling enough, however, one last point - if anyone wants to implement this, or make an enhancement request, please do so, and don't wait for me.
Starson17 is offline   Reply With Quote
Old 01-25-2010, 11:07 AM   #6
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,771
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
The way the download system works is as follows:

1) isbndb and google books are queried based on title and author or optionally isbn if it is provided. In single metadata edit the user selects the best match, in auto the best match is guessed based on description length and a couple of other checks that I don't recall right now. neither of these plugins download ratings, but GB does download tags.

2) The isbn number from (1) is fed to the amazon plugin (and any other social metadata plugins) which downloads tags/rating/metadata. The rating is calculated as an average from all social plugins. tags are the union of tags from step 1 and all social plugins. comments are append.
kovidgoyal is offline   Reply With Quote
Old 01-25-2010, 01:26 PM   #7
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by kovidgoyal View Post
The way the download system works is as follows:
Thank you. I've been poking through the code and had figured some of that out, but your explanation helps a lot.

I've also figured out that calibre.ebooks.metadata.__init__.py defines smart_update and there's a convenient spot there at line 252 to block the title and author changes that occur during multiple ebook metadata downloads. That lets me handle the two important cases I need - single and multiple metadata downloads.

I still need to work on understanding the plugin customization code and I'm not totally certain whether blocking parts of smart_update might block other required functionality. (Right now I haven't tested enough, and I'm just blocking the changes when I want to do a global update for multiple books.)

I'd like to get to the point where I could propose some code that would be close to correct. That's hard to do before you understand the basic structure of the code. There's really no better way to learn this stuff than to have a project and try to figure out what you need to know to complete it.
Starson17 is offline   Reply With Quote
Old 01-25-2010, 01:31 PM   #8
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by kovidgoyal View Post
isbndb and google books .... neither of these plugins download ratings, but GB does download tags.
BTW, the plugin for both of these (isbndb and google books) have a checkbox for "download ratings" on the customization screen.
Starson17 is offline   Reply With Quote
Old 01-25-2010, 02:28 PM   #9
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,771
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Quote:
Originally Posted by Starson17 View Post
BTW, the plugin for both of these (isbndb and google books) have a checkbox for "download ratings" on the customization screen.
Yes that because I was lazy and all three share the same customization code.
kovidgoyal is offline   Reply With Quote
Old 01-25-2010, 02:56 PM   #10
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by kovidgoyal View Post
Yes that because I was lazy and all three share the same customization code.
Feet of clay.
Starson17 is offline   Reply With Quote
Old 01-29-2010, 03:47 PM   #11
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by Starson17 View Post
I'd like to get to the point where I could propose some code that would be close to correct.
Ticket # 4728 has code that is working for me. It adds checkboxes on the config (default) and single book metadata fetch screens to allow or prohibit author, title and author_sort overwriting during a global or single metadata update.
Starson17 is offline   Reply With Quote
Old 02-04-2010, 01:08 PM   #12
mezme
Connoisseur
mezme began at the beginning.
 
Posts: 58
Karma: 10
Join Date: Dec 2009
Device: PRS700
I noticed David closed the ticket and added it to the trunk... do you know which version this will be appearing in? *grins* I've added coding changes to the development system but, even after adding the environment variable, I cannot seem to get Calibre to read from the development project versus the calibre directory. I'm sure I'm missing something simple but ahhh well... I'll get it figured out eventually.
mezme is offline   Reply With Quote
Old 02-04-2010, 01:17 PM   #13
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by mezme View Post
I noticed David closed the ticket and added it to the trunk... do you know which version this will be appearing in?
The code is out now - as of 0.6.37. I tweaked mine so I could independently control overwrite of author vs. title, but I didn't want to clutter up the interface with too many options, so I combined them in the code I submitted.
Starson17 is offline   Reply With Quote
Old 02-04-2010, 02:29 PM   #14
mezme
Connoisseur
mezme began at the beginning.
 
Posts: 58
Karma: 10
Join Date: Dec 2009
Device: PRS700
ahh got it! So far it works beautifully. Thanks for the great work on the coding I'm just starting to get into python coding so it would have taken me forever to figure it out on my own. David --- thanks for the quick add on the change!!!
mezme is offline   Reply With Quote
Old 02-04-2010, 03:31 PM   #15
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Quote:
Originally Posted by mezme View Post
ahh got it! So far it works beautifully. Thanks for the great work on the coding I'm just starting to get into python coding so it would have taken me forever to figure it out on my own.
I'm a Python beginner myself. I couldn't have added the checkboxes or stored configuration flag for this option if not for the fact that it was almost identical to Kovid's existing code for optionally downloading the social metadata. All I had to do was find the spots where the flag would change the code behavior, and I'd already identified those.
Starson17 is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Calibre doesnt remember (Title.Author,Series,Metadata) changes?! Rafaelo4 Calibre 9 08-19-2010 07:23 AM
Recognition of author and title from html files/reading metadata from a seperate file Lethe Calibre 5 04-03-2010 08:35 AM
Creating a Library file w/Author, Title, Summary and tag info asktheeightball Calibre 2 01-18-2010 10:28 AM
jetBook reads metadata with FB2 for title & author tselling Ectaco jetBook 0 01-07-2010 09:54 PM
HTML author / title tags ? romi Bookeen 0 05-11-2009 07:47 AM


All times are GMT -4. The time now is 05:17 AM.


MobileRead.com is a privately owned, operated and funded community.