Cleaning metadata (titles and authors)

[pckopp]

I should read _before_ typing. This works:

[[](?P<series>.+)[]] (?P<title>.+) - (?P<author>[^_]+)

EDIT: Works perfectly in the test box but not in the actual Add Books process. Acts like the original title - author expression. Huh?

Originally Posted by pckopp

I'm one of the users for whom RE is a barely understood concept, so can I just ask a question related to adding books?

My current expression: (?P<title>.+) - (?P<author>[^_]+)

works great. But sometimes I have a filename that includes the series info at the beginning in brackets. Like this:

[Instrumentality Of Mankind] Golden the Ship Was Oh! Oh! Oh! - Cordwainer Smith.epub

I would like the text between the brackets to be in the Series box.

Thanks!

[Starson17]

Originally Posted by chaley

<pontification>
From a development standpoint, it wouldn't be hard ...
</pontification>

I completely agree with this view. However, (no that doesn't mean I'm going to backtrack .... much) Calibre does inhabit the high end when it comes to the technical sophistication of users and the skills needed to use many features. Extra CSS, regex expressions in multiple locations and XPath for TOC require a lot of skill, even with Kovid's excellent wizards.

If I was still struggling through my initial stages of data entry, I would have loved to have had a sticky thread with useful Python code fragments that could be executed with:

calibre-debug -e myscript.py

.
It would have a first post saying there's no bulk undo, use at your own risk, make a backup first, etc. Then some scripts like:
Regex search and replace in title or author.
Populate a field with the contents of other fields or data.
etc.

People could add scripts, as they do recipes, as they're written. I clearly remember my frustration when I couldn't easily directly modify the metadata.db SQL database to do regex searches and fix miscellaneous errors in title/author.

I have to say that in view of the many recent improvements in Calibre, I want such scripts a lot less (plus I'm no longer actively adding my old ebook library).

Still, I have no problem with providing some options (outside the Calibre installed environment) to help the most sophisticated Calibre users who are willing to run at the bleeding edge and take the consequences.

[gwk]

And that chaley would really be perfect.
Throw a regexp in the mix, for the regexp people on this planet, and it would do the job.

By the way, I recognize the regexp issues, not that many people know how to create them.

With respect to python scripts.
They could almost serve as a kind of small apps which the community can add to Calibre.
When apps become popular, simply add them as feature to calibre.

But a bit of basic plumbing and community attention is required, like
- some small examples
- a menu ability in Calibre - which adds for instance the ability to run it against a selection of books (not sure how to do that from the command line)
- a place to post them
- a means to add review comments, etc
- moderation and management

While the first and second items are relatively easy to achieve, the rest requires far more attention and work to pull it off.

Not sure how this worked for the news sources, probably that is the best example.

[chaley]

Originally Posted by gwk

And that chaley would really be perfect.
Throw a regexp in the mix, for the regexp people on this planet, and it would do the job.

The search string is a python regexp pure and simple. The replacement string is a python replacement string (can contain backreferences). The function is one of null, lower case, title case, or upper case, and it is applied after backreference substitution.

It will be in today's release, marked as experimental. Backup your library before hacking.

[Starson17]

Originally Posted by chaley

The search string is a python regexp pure and simple. The replacement string is a python replacement string (can contain backreferences). The function is one of null, lower case, title case, or upper case, and it is applied after backreference substitution.

It will be in today's release, marked as experimental. Backup your library before hacking.

Cool! How about something similar that fills a field from other fields with {title} or {isbn} references, etc.? It would let you build a copy of a field (say copy all {title} into it or all {isbn}), then run the new experimental S&R on the new field with less risk of destruction. Later you could send the modified field back into the real title. I've occasionally wanted to send tags into comments, or custom text fields into tags, but even text only to text only would have some uses for me.

[chaley]

Originally Posted by Starson17

Cool! How about something similar that fills a field from other fields with {title} or {isbn} references, etc.? It would let you build a copy of a field (say copy all {title} into it or all {isbn}), then run the new experimental S&R on the new field with less risk of destruction. Later you could send the modified field back into the real title.

Sounds like you want undo.

In effect, you are asking for some kind of scripting language that supports some level of transform on fields, including copying from one to another. There be dragons here. Such languages never stop growing.

It seems more reasonable to use python plugins for this. Thought is required.

[Manichean]

Originally Posted by chaley

<pontification>
[...]
</pontification>

That's a very good post. I've already noticed a rise in threads about regexp matching for header/footer removal and such, asking just about the same questions all the time. (I actually thought about writing a tutorial, but seeing as I'm just beginning to understand them, I might not be the right guy for that. I haven't fully groked how to cope with linebreaks and somesuch...)

That being said, I really like your mockup. Provided it understands regular expressions as well as "normal" strings (wait, that would be a regular expression in itself... nevermind), it's a gorgeous and, I think, very powerful feature.

[Starson17]

Originally Posted by chaley

Sounds like you want undo.

No, although the ability to work on a copy of a field would be nice. It's merely that I occasionally have wanted to move field contents around.

In effect, you are asking for some kind of scripting language that supports some level of transform on fields, including copying from one to another. There be dragons here. Such languages never stop growing.

It seems more reasonable to use python plugins for this. Thought is required.

I had in mind simple field copying to start. Select the destination field to be acted upon and another field as source. Have the source field appended to the destination field. With that, and your regex field processing, I could do anything to the fields I've ever wanted to do.

[Starson17]

Originally Posted by Starson17

Cool!

Charles. I've tested it, and it works great! The only comment I have is that I'd prefer to leave the highlighting/selection on the previously selected books, so I can hit them several times without reselecting each time.

[chaley]

Originally Posted by Starson17

I had in mind simple field copying to start. Select the destination field to be acted upon and another field as source. Have the source field appended to the destination field. With that, and your regex field processing, I could do anything to the fields I've ever wanted to do.

What would you do about multiplicity? Could you require that the source and destination be the same? If they are not, then you can have all sorts of bizarre behavior, such as authors being munged together because the separating ',' was deleted.

In any event, I think you are suggesting something like:
0. verify that source and dest have the same multiplicity.
1. get value from source
2. process it with regexp, taking into account multiplicity
3. store result in dest.

[chaley]

Originally Posted by Starson17

Charles. I've tested it, and it works great!

Good. Nice to know I don't need to beg Kovid to yank it from the release he is in process of making.

The only comment I have is that I'd prefer to leave the highlighting/selection on the previously selected books, so I can hit them several times without reselecting each time.

The problem is that bulk edit does a resort/re-search when it finishes, often rendering the previous selection invalid. One could argue that it shouldn't do that, but that is an issue to take up with Kovid.

[kovidgoyal]

Originally Posted by chaley

The problem is that bulk edit does a resort/re-search when it finishes, often rendering the previous selection invalid. One could argue that it shouldn't do that, but that is an issue to take up with Kovid.

Well even if it does a resort there's no reason it cant reselect the previously selected entries once its done.

[chaley]

Originally Posted by Manichean

I've already noticed a rise in threads about regexp matching for header/footer removal and such, asking just about the same questions all the time.

I also have noticed this. GRiker gets many of them. However, I think that questions about regexps and pdfs are the winners. And I just made it worse.

(I actually thought about writing a tutorial, but seeing as I'm just beginning to understand them, I might not be the right guy for that. I haven't fully groked how to cope with linebreaks and somesuch...)

Writing such a tutorial would be fantastic. And your 'beginning' state might help. If you have time, then please give it a try. There is a lot of incentive here for experts to help check what you do for technical accuracy.

That being said, I really like your mockup. Provided it understands regular expressions as well as "normal" strings (wait, that would be a regular expression in itself... nevermind), it's a gorgeous and, I think, very powerful feature.

Isn't that a wonder feature of regexps? Any string can match itself, so it is a regexp! When I was teaching programming language compiler construction (regexps have a huge role there), students would light up when they figured that out. Understanding that a regexp is just a string that matches other strings takes much of the magic out of it. Once we have that, all we are really worrying about is what a character is.

[chaley]

Originally Posted by kovidgoyal

Well even if it does a resort there's no reason it cant reselect the previously selected entries once its done.

As long as they are still visible. And as long as we are willing to have the selections scattered all over the map.

If you are OK with this, I will look at doing it.

One rule: I don't think we should select items that are no longer visible.

[kovidgoyal]

Yeah, I'm fine with it. You'll need to add select_row function to BooksView analogous to the set_current_row.