Cleaning metadata (titles and authors)

[Starson17]

Originally Posted by chaley

What would you do about multiplicity? Could you require that the source and destination be the same? If they are not, then you can have all sorts of bizarre behavior, such as authors being munged together because the separating ',' was deleted.

I'm not too worried about bizarre behavior - this is going to wipe your Library if you do it wrong. You're truly brave to make it accessible to all, although I like it.

In any event, I think you are suggesting something like:
0. verify that source and dest have the same multiplicity.
1. get value from source
2. process it with regexp, taking into account multiplicity
3. store result in dest.

Essentially, yes. I envisaged step 0 as testing for a compatible field transfer. I could send a comma separated text string of tags into a comments field, even if not the other way around. I could send a single valued text into tags, etc.
I don't see any critical need to process with regexp in a single shot. Your existing regexp should do it. And, step 3 would be an append so I can append fields.

[phenomshel]

Originally Posted by chaley

I de-pontificated myself and have been playing with bulk edit. Something like this might actually appear ...

Was it painful?

(sorry, couldn't resist...not that I tried very hard...)

[chaley]

Originally Posted by phenomshel

Was it painful?

Oh yes, it was. Hours of existential contemplation. In the end, I realized that I am free (as 'have freedom' as opposed to 'costing nothing', which I also have), and I can walk away! At that point I decided that if it is fun to do, then I will do it. Damn the torpedos!

And while we are metaphilosophical questions, I frequently wonder why so many people have 15,000+ book libraries...

[Manichean]

Originally Posted by chaley

Writing such a tutorial would be fantastic. And your 'beginning' state might help. If you have time, then please give it a try. There is a lot of incentive here for experts to help check what you do for technical accuracy.

Thanks for the encouragement, I think I'll try if I can get something useful together over the weekend. If I do, I'll post it for further refinement. I'll bet that next you guys want me to learn XPath, because that is even worse documented than regular expressions, I think (At least the part in the Calibre manual made very little sense to me. To be honest, I really just had a quick glance, but still...)

Originally Posted by chaley

Isn't that a wonder feature of regexps? Any string can match itself, so it is a regexp! When I was teaching programming language compiler construction (regexps have a huge role there), students would light up when they figured that out. Understanding that a regexp is just a string that matches other strings takes much of the magic out of it. Once we have that, all we are really worrying about is what a character is.

For me, having some experience writing code definitely helped to understanding regexes, so far. It's somewhat more arcane, but not really more complicated than, say, C print format strings. Still, some behaviour of regexes is, lets say somewhat unexpected for me, so I can't imagine how people without any programming experience might feel when confronted with something like this.

[chaley]

Originally Posted by kovidgoyal

Yeah, I'm fine with it. You'll need to add select_row function to BooksView analogous to the set_current_row.

Code pushed.

Added a select_rows_with_id(ids) method that does the right thing.

[lvovsky]

if strange title/author fields (underscores, etc) come form file name, I find it easier to fix file names first with tool like ReNamer and then add them into calibre.

[Manichean]

This is really only a minor matter, but when I click on the "this reference"- link in your new feature, nothing happens. That could be somewhat offputting

[gwk]

Hi Chaley, it works perfect. Thank you very much.
Also the uppercase/lowercae option is really nice.

If you ever want to extend it with the option to use also other fields. I would suggest to define standard regexp groups and use it in a syntax like \g<title>.
That by the way would make it look like the regexp syntax used in the add-book thinky.

[chaley]

Originally Posted by Manichean

This is really only a minor matter, but when I click on the "this reference"- link in your new feature, nothing happens. That could be somewhat offputting

Yea, calibre doesn't attempt to start the browser. What you do is right-click on the link and copy it.

I have wondered whether it would be better to simply have the http:... text and not mark it as a URL. Unfortunately, that would not be copyable, which is probably worse.

[chaley]

Originally Posted by gwk

If you ever want to extend it with the option to use also other fields. I would suggest to define standard regexp groups and use it in a syntax like \g<title>.
That by the way would make it look like the regexp syntax used in the add-book thinky.

Can you give me a use case for cross-field named groups? I can't think of one that isn't extremely contrived, other than Starson's example of wanting to populate one field from another.

[Manichean]

Originally Posted by chaley

Yea, calibre doesn't attempt to start the browser. What you do is right-click on the link and copy it.

I have wondered whether it would be better to simply have the http:... text and not mark it as a URL. Unfortunately, that would not be copyable, which is probably worse.

Huh. It works for the link in the XPath wizard (seen in conversion options).
I'm probably being pedantic here, but as a user, I'd either expect the link to launch or some sort of note explaining that I have to copy the link.

That being said, this feature is great. I used it only once so far and already love it.

[chaley]

Originally Posted by Manichean

Huh. It works for the link in the XPath wizard (seen in conversion options).

Found it. There is a magic property that one must set in the display widget that tells it to open links in a browser. Will be fixed eventually.

[Manichean]

Originally Posted by chaley

Found it. There is a magic property that one must set in the display widget that tells it to open links in a browser. Will be fixed eventually.

Great. Like I said, it was a minor inconvenience (for me, at least) and really more about what I believe is called "user experience"
I would have liked to give you karma, by the way, but currently can't, for whatever reasons.

[gwk]

Okay a few example where a cross field thing can help:

Example 1:
Title contains: Hello world (by Smart Author)
Author contains: World series
Series contains:

Example 2:
Title contains: BookSeries 1 Hello world
Author contains: Smart Author
Series contains:

But if I think about, it can almost be solved by having an option to copy the field accross, and then using the existing mechanism to clean it up.

No real need for extending the regexp, just a selection for a target field would do.

What I found out using the new feature, it just enabled me to do things in seconds, that I did not want to do, just because the amount of work involved.
For instance, I just harmonized the case over all titles to an uppercase on each word, and changed 'S back to 's. This was previously, just too much work.

May be this thing is like wise. The attention from users for the regexp used to add books proves at least proves at least that I am not the only person fighting with the metadata.

[Starson17]

Originally Posted by gwk

No real need for extending the regexp, just a selection for a target field would do.

Exactly.
And to Charles: You know this, but work only on what you enjoy. If I'm going to work on something, I like to know if others will find it useful. I posted only to clue you in on problems I'd seen - things I'd thought about doing myself - things I thought would prove useful - not to convince you to work on anything you might not do otherwise.