How To: Getting summary metadata and images, not tags, author, title

[Starson17]

I add books to my library as soon as I get them. Usually, I take the time to make sure the title and author are right, but I don't always have time to download the metadata and covers. I particularly like getting the summary info.

Later I want to edit metadata, add my personal tags, pick up covers, etc. I open up the edit metadata screen, and tell it to fetch, and I see 3 or four likely descriptions. The problem is always that the title or author may be in all caps. Or the title will have the series name in it, even though the other books in that series don't have the series in the title. If I want to pick up the summary description, it writes over my correct author/title. I've particularly noticed that the title is often incorrectly capitalized. I then have to go back and fix the author and title that was correct when I started!

Another issue occurs when I want to look at several different cover images. To do that you have to choose different results from the fetch table, but each choice writes over the author and title again. I know some people hate the multiple tags - this lets you stop picking them up,while still picking up ratings and images, etc.

The one thing that is really, really nice about open source software is you have control. Now I know that not many of us are programmers, and even fewer are familiar with Python, but it really is easy to make small changes that improve things for special cases. You just need to take a few steps to get the source code, and use Notepad (Windows) to type "# " in front of a few lines to prevent Calibre from overwriting the existing data with the new data. You can pick up only the summary, or even pick up summaries from 3 versions of the book. (Summaries are cumulative - they don't write over your comments or the summaries already in there.)

Here's how I fixed it. First, read this on Setting up a calibre development environment¶

It may seem tricky, but truthfully, all you have to do is download Bazaar, run it once to tell it where you want to put your source code, and set one environment variable in Windows to tell Calibre where you put it. Then you're ready to take control. As long as that environment variable is set, Calibre will use your changes. As soon as you delete or change the environment variable from "CALIBRE_DEVELOP_FROM" to something else (like "CALIBRE_NOT_DEVELOP_FROM") you hide your revised source code and Calibre goes back to its normal behavior.

Now all you need to know is what changes to make. Go to where you told Bazaar to put your source code ( I use a folder called src inside the Calibre folder) and find the file called calibre\gui2\dialogs\metadata_single.py

starting at line 575 you see:

                        else:
                            book.tags = []
                        # self.title.setText(book.title)
                        # self.authors.setText(authors_to_string(book.authors))
                        # if book.author_sort: self.author_sort.setText(book.author_sort)
                        if book.publisher: self.publisher.setEditText(book.publisher)
                        if book.isbn: self.isbn.setText(book.isbn)
                        if book.pubdate:
                            d = book.pubdate
                            d = d - self.local_timezone_offset
                            self.pubdate.setDate(QDate(d.year, d.month, d.day))
                        summ = book.comments
                        if summ:
                            prefix = unicode(self.comments.toPlainText())
                            if prefix:
                                prefix += '\n'
                            self.comments.setText(prefix + summ)
                        if book.rating is not None:
                            self.rating.setValue(int(book.rating))
                        if book.tags:
                            self.tags.setText(', '.join(book.tags))

Just comment out (that means put "# " in front of) any line that controls something you don't want picked up. In the code above, I've commented out the title, author and author_sort and left everything else. There are a few lines that are inside "if" blocks. For those, you just need to comment out the if line and all indented lines below it.

I know it can be scary to play with the source code, if you've never done it, but all you have to do is delete or change the environment variable, as above, and all your changes stop having any effect on Calibre. It's a lot easier than writing a regex expression. This little change made it possible for me to quickly look at several different images, pick up multiple descriptions, and still get the correct publisher, pubdate on the last selection without having to constantly fix the author and title.

Have fun.

[mezme]

Perfect - Thanks Starson!!!! This was an awesome and quick solution to the metadownload issue!

[Starson17]

Originally Posted by mezme

This was an awesome and quick solution to the metadownload issue!

Thanks for the kind comment. It was really useful to me, so I thought I'd share.

Remember that it will use your source code as long as the environment variable is set. If you upgrade the executables the usual way, the upgrade won't be seen if you don't also update the source code or turn off the environment variable. Of course, it's a handy way to keep an older version around. You can switch it on only when you're going to be working on the metadata, then switch it off.

I considered trying to write a proposed modification to the main program so you could turn on/off the pickup of each piece of metadata without having to muck around in the source code each time. It's certainly possible to do it, but I'm not sure if it would be accepted or even if it's needed by others. (On top of which, I don't currently have the skill to do it correctly.)

Plus, you'd probably end up adding some checkboxes on the fetch screen that you'd have to keep clicking for each book (slowing the process down), or semi-hidden default checkboxes that might generate lots of support headaches if not used correctly. This way works great and if you are adventurous enough to play with the source, you're probably going to be able to avoid most of the problems.

[DoctorOhh]

Originally Posted by Starson17

I considered trying to write a proposed modification to the main program so you could turn on/off the pickup of each piece of metadata without having to muck around in the source code each time. It's certainly possible to do it, but I'm not sure if it would be accepted or even if it's needed by others. (On top of which, I don't currently have the skill to do it correctly.)

It would be accepted and it is needed but skills I can't help you with.

Originally Posted by Starson17

Plus, you'd probably end up adding some checkboxes on the fetch screen that you'd have to keep clicking for each book (slowing the process down), or semi-hidden default checkboxes that might generate lots of support headaches if not used correctly. This way works great and if you are adventurous enough to play with the source, you're probably going to be able to avoid most of the problems.

Currently options for selecting what metadata to grab are under the plugins. Specifically metadata download plugins. I go to each of the three plugins now and uncheck tags so I don't end up with a million tags. Having boxes for author and title would be a nice touch.

Perhaps you could take your original post and open up a Calibre enhancement ticket clearly stating the goal.

Thanks for the info. If it ever gets implemented I'm sure many folks will use this feature.

[Starson17]

Originally Posted by dwanthny

It would be accepted and it is needed but skills I can't help you with.

That's OK, I don't mind picking up the skills myself, they just take time, and I'm not there yet.

Currently options for selecting what metadata to grab are under the plugins. Specifically metadata download plugins. I go to each of the three plugins now and uncheck tags so I don't end up with a million tags. Having boxes for author and title would be a nice touch.

Thank you for this comment! I've been digging through the code and through the main interface, but it hadn't really dawned on me that the customizations for these plugins is where these options for title/author would be expected to be found. Now that I've looked at them, I see that with the current design, this is exactly where they would be. In fact, I can see that the location of the code change I posted above affects only downloads from the single_metadata screen, not multiple automated metadata updates, whereas if I made the change at the plugins (as Kovid suggested) it would have effect for both single and multiple updates.

Perhaps you could take your original post and open up a Calibre enhancement ticket clearly stating the goal.

I thought of that, but I didn't want to ask someone else to do work for me when the code is available for me to do it myself. Plus, I'm still exploring the nooks and crannies of the interface (as in I hadn't even understood that the plugin customization is where this would go) so it's still premature.

Thanks for the info. If it ever gets implemented I'm sure many folks will use this feature.

Thank you for the info! One reason I posted here was to gauge whether there was any interest in this feature. Another was to try to elicit comments for how it should work. Your post has helped with both. I will probably keep on hacking around in the code, just to familiarize myself, then maybe I will be in a position to make a very clear improvement request, with some possible code that someone with greater skill could look over and correct or improve (as Kovid did with my last tiny proposed code segment).

I actually spent a lot of time in the plugin module code for the fetchers, but it was heavy going until I read some more on Python and PyQT4.

On your point of "clearly stating the goal" that's always half the battle. I'm not even sure I full understand the way Google, ISBNDB and Amazon plugins currently interact. I had thought that two of them were used to find the ISBN, and the third was used for everything else. However, it's clear from the plugin customizations that all three provide ratings, descriptions and tags.

I think what would work well is to set a default in these plugins that allows the title or author to be overwritten (or not) during a metadata download and then allow the default to be overridden with a checkbox at the single_metadata download screen.

This post is long and rambling enough, however, one last point - if anyone wants to implement this, or make an enhancement request, please do so, and don't wait for me.

[kovidgoyal]

The way the download system works is as follows:

1) isbndb and google books are queried based on title and author or optionally isbn if it is provided. In single metadata edit the user selects the best match, in auto the best match is guessed based on description length and a couple of other checks that I don't recall right now. neither of these plugins download ratings, but GB does download tags.

2) The isbn number from (1) is fed to the amazon plugin (and any other social metadata plugins) which downloads tags/rating/metadata. The rating is calculated as an average from all social plugins. tags are the union of tags from step 1 and all social plugins. comments are append.

[Starson17]

Originally Posted by kovidgoyal

The way the download system works is as follows:

Thank you. I've been poking through the code and had figured some of that out, but your explanation helps a lot.

I've also figured out that calibre.ebooks.metadata.__init__.py defines smart_update and there's a convenient spot there at line 252 to block the title and author changes that occur during multiple ebook metadata downloads. That lets me handle the two important cases I need - single and multiple metadata downloads.

I still need to work on understanding the plugin customization code and I'm not totally certain whether blocking parts of smart_update might block other required functionality. (Right now I haven't tested enough, and I'm just blocking the changes when I want to do a global update for multiple books.)

I'd like to get to the point where I could propose some code that would be close to correct. That's hard to do before you understand the basic structure of the code. There's really no better way to learn this stuff than to have a project and try to figure out what you need to know to complete it.

[Starson17]

Originally Posted by kovidgoyal

isbndb and google books .... neither of these plugins download ratings, but GB does download tags.

BTW, the plugin for both of these (isbndb and google books) have a checkbox for "download ratings" on the customization screen.

[kovidgoyal]

Originally Posted by Starson17

BTW, the plugin for both of these (isbndb and google books) have a checkbox for "download ratings" on the customization screen.

Yes that because I was lazy and all three share the same customization code.

[Starson17]

Originally Posted by kovidgoyal

Yes that because I was lazy and all three share the same customization code.

Feet of clay.

[Starson17]

Originally Posted by Starson17

I'd like to get to the point where I could propose some code that would be close to correct.

Ticket # 4728 has code that is working for me. It adds checkboxes on the config (default) and single book metadata fetch screens to allow or prohibit author, title and author_sort overwriting during a global or single metadata update.

[mezme]

I noticed David closed the ticket and added it to the trunk... do you know which version this will be appearing in? *grins* I've added coding changes to the development system but, even after adding the environment variable, I cannot seem to get Calibre to read from the development project versus the calibre directory. I'm sure I'm missing something simple but ahhh well... I'll get it figured out eventually.

[Starson17]

Originally Posted by mezme

I noticed David closed the ticket and added it to the trunk... do you know which version this will be appearing in?

The code is out now - as of 0.6.37. I tweaked mine so I could independently control overwrite of author vs. title, but I didn't want to clutter up the interface with too many options, so I combined them in the code I submitted.

[mezme]

ahh got it! So far it works beautifully. Thanks for the great work on the coding I'm just starting to get into python coding so it would have taken me forever to figure it out on my own. David --- thanks for the quick add on the change!!!

[Starson17]

Originally Posted by mezme

ahh got it! So far it works beautifully. Thanks for the great work on the coding I'm just starting to get into python coding so it would have taken me forever to figure it out on my own.

I'm a Python beginner myself. I couldn't have added the checkboxes or stored configuration flag for this option if not for the fact that it was almost identical to Kovid's existing code for optionally downloading the social metadata. All I had to do was find the spots where the flag would change the code behavior, and I'd already identified those.