Cleaning metadata (titles and authors)

[chaley]

Originally Posted by gwk

Example 1:
Title contains: Hello world (by Smart Author)
Author contains: World series
Series contains:

Example 2:
Title contains: BookSeries 1 Hello world
Author contains: Smart Author
Series contains:

I am trying to work out how one would use cross-field templates here, other than to copy fields.

We have three things:
1) the metadata for a book. Say we are working on 'title'
2) the search expression
3) the target expression

What we do is:
book.meta('title') = re.sub(search_exp, target_exp, book.meta('title'))

We can't put a cross-field reference into search_exp, because then it won't match the metadata. We can put a cross-field reference into the target. Consider:
book.meta('series') = re.sub('', '\g<title>', book.meta('series'))
In your second example, series would now be 'BookSeries 1 Hello world'. You can now hack at it with further search/replaces.

However, I don't think that is what you really want. It seems to be that you really want to be able to say something like:
book.meta('series') = re.sub('(.*\d+) ', '\1', book.meta('title'))

Or perhaps you want both facilities?
book.meta('genre') = re.sub('(Fiction|Fantasy)', '\1 \g<#genre>', book.meta('tags'))

Do I have it right?

(Starson17, could you also answer this?)

[chaley]

Originally Posted by Starson17

And to Charles: You know this, but work only on what you enjoy.

Yea, I had to remind myself of that when some Kindle issues came up. I realized that I was unhappy that I 'had' to do something that I didn't find useful. Then I realized that *I* was the one applying the pressure, and that was stupid. Voila...

Even though I have strong doubts that this search/replace stuff will be in the end a good thing to have done, it is technically fascinating. How best to combine the power of the new metadata management facilities (you don't have that yet -- beta soon) and the just-born search/replace stuff is the sort of challenge I like.

[gwk]

It seams to me that the questions comes down to the following.

Do we need:
- a one to one field copy - and remove some not needed characters
- do we need to combine fields (a many to one copy) - and remove some not needed characters

At this point in time, I cannot seam to find an useful example for the usage of combining fields. Something like this would only occur for something like this:
- series contains: AuthorLastName
- author contains: AuthorFirstName
In this situation you would like to combine both fields to complete the author field.
In my opinion this situation is rare.

[Starson17]

Originally Posted by chaley

Even though I have strong doubts that this search/replace stuff will be in the end a good thing to have done, it is technically fascinating.

If no support issues arise and no one loses their library, it will have been a great thing to have done. If they do .... point to the warnings and sleep well anyway. I've also used it to do a few things I wanted to do, but not enough to do the hard way.

[Starson17]

Originally Posted by gwk

It seams to me that the questions comes down to the following.

Do we need:
- a one to one field copy - and remove some not needed characters
- do we need to combine fields (a many to one copy) - and remove some not needed characters

At this point in time, I cannot seam to find an useful example for the usage of combining fields. Something like this would only occur for something like this:
- series contains: AuthorLastName
- author contains: AuthorFirstName
In this situation you would like to combine both fields to complete the author field.
In my opinion this situation is rare.

You can solve this problem as long as the field copy can be run in append mode to append a source field to a destination field's contents. There's no need to have multiple field -> single field in a single step as long as you can do it in multiple steps for the rare case you might want this.

Charles already has the "remove unwanted characters" built in the regex processing. All we need is two pulldown boxes for existing field references, one labeled "Source Field" and one "Destination Field" and a checkbox for overwrite/append. The code just has to check for compatible field types for the overwrite/append operation.

[kacir]

You guys are incredible!
I have just started to use Calibre to organize some of my chronically disorganized books. I have come across situation when I realized that a bulk edit of metadata using my beloved Regular Expressions could be very useful.
Then I spot this thread.
Before I finish the thread I start Calibre, to try something out, and it tells me there is new version (despite the fact I updated just a few days ago).

After updating Calibre, it turned out there is a sparkling new experimental feature - Search and replace in Bulk metadata edit.

Absolutely incredible!

[kovidgoyal]

@chaley: A feature you might consider adding is a checkbox that control if the search and replace texts are interpreted as regexs or as normal strings. For people that get scared by regexs, normal S&R could still be useful.

[gwk]

Originally Posted by kacir

You guys are incredible!

Yep, it is really a great new feature!!! Well done, and thanks.

[chaley]

Folks:

In the new-metadata releases (will go to alpha today or tomorrow), I am considering implementing the following, chosen via a combo box.

1) Text mode -- case sensitive. Simple text search and replace.
2) Text mode -- case insensitive.
3) Regexp basic -- as today. Source and destination are the same. Search_string is applied to source, matches are replaced with replace_string then processed by function.
4) Regexp advanced. Source is a template, similar to the save-as templates. The source string is built from the template. Destination is explicitly specified, and must be a single field. For the destination, a combo box will offer the choices of 'replace', 'prepend', or 'append'. The search_string will be applied to the template. Matches will be replaced with replace_string and processed by function. When all matches are finished, the result will be saved to the destination according to the 'replace' etc specification.

The advanced form can be used to copy values between fields with some processing on the way, construct composite fields, combine fields, and many other variations. I think this covers all the requests to date.

I know it is hard to make comments without an implementation, but if you have any, speak now.

[kacir]

Originally Posted by chaley

Folks:

In the new-metadata releases (will go to alpha today or tomorrow), I am considering implementing the following, chosen via a combo box.

1) Text mode -- case sensitive. Simple text search and replace.
2) Text mode -- case insensitive.
3) Regexp basic -- as today. Source and destination are the same. Search_string is applied to source, matches are replaced with replace_string then processed by function.
4) Regexp advanced. Source is a template, similar to the save-as templates. The source string is built from the template. Destination is explicitly specified, and must be a single field. For the destination, a combo box will offer the choices of 'replace', 'prepend', or 'append'. The search_string will be applied to the template. Matches will be replaced with replace_string and processed by function. When all matches are finished, the result will be saved to the destination according to the 'replace' etc specification.

The advanced form can be used to copy values between fields with some processing on the way, construct composite fields, combine fields, and many other variations. I think this covers all the requests to date.

I know it is hard to make comments without an implementation, but if you have any, speak now.

Great idea. Power users will love those features.
I can't wait to play with the new features.

[theducks]

Originally Posted by chaley

Folks:


The advanced form can be used to copy values between fields with some processing on the way, construct composite fields, combine fields, and many other variations. I think this covers all the requests to date.

I know it is hard to make comments without an implementation, but if you have any, speak now.

Templates feature requests (yes plural ) for us non- guru types

Provide the template and a "Comment/Description field(to say what it is supposed to be doing)" (balloon tip OK if there is a comment mechanism somewhere.

Provide a Import/Export so we can share solutions

a discard (for the times we really mess up)

[Manichean]

Originally Posted by chaley

Folks:

In the new-metadata releases (will go to alpha today or tomorrow), I am considering implementing the following, chosen via a combo box.

1) Text mode -- case sensitive. Simple text search and replace.
2) Text mode -- case insensitive.
3) Regexp basic -- as today. Source and destination are the same. Search_string is applied to source, matches are replaced with replace_string then processed by function.
4) Regexp advanced. Source is a template, similar to the save-as templates. The source string is built from the template. Destination is explicitly specified, and must be a single field. For the destination, a combo box will offer the choices of 'replace', 'prepend', or 'append'. The search_string will be applied to the template. Matches will be replaced with replace_string and processed by function. When all matches are finished, the result will be saved to the destination according to the 'replace' etc specification.

The advanced form can be used to copy values between fields with some processing on the way, construct composite fields, combine fields, and many other variations. I think this covers all the requests to date.

I know it is hard to make comments without an implementation, but if you have any, speak now.

Sounds like you have all cases covered. I'm really excited to see this.

[DoctorOhh]

Originally Posted by chaley

I know it is hard to make comments without an implementation, but if you have any, speak now.

It sounds interesting, maybe a click through popup box emphasizing the need for a fresh backup of the library and the possibility of total destruction of the library.

For those that know what they're doing this will be an annoyance, for those that aren't sure what they're doing maybe they will think twice.

Keep up the excellent work.

[Manichean]

Originally Posted by dwanthny

It sounds interesting, maybe a click through popup box emphasizing the need for a fresh backup of the library and the possibility of total destruction of the library.

For those that know what they're doing this will be an annoyance, for those that aren't sure what they're doing maybe they will think twice.

Keep up the excellent work.

We need an "expert mode" for Calibre that allows one to destroy the entire library without any pesky popups or anything
Seriously though, in my opinion, keep the warning as it is now, maybe a little more strongly worded. Those who are too stupid, excuse the language, to read what the window they're using says deserve no better. (I'm a fan of YAFIYGI as opposed to WYSIWYG, can you tell? )

[DoctorOhh]

Originally Posted by Manichean

We need an "expert mode" for Calibre that allows one to destroy the entire library without any pesky popups or anything
Seriously though, in my opinion, keep the warning as it is now, maybe a little more strongly worded.

I agree that the pop-up should put the fear of absolute lost of their library as they know it is possible. You already have a "expert mode" calibre uses pop-up warnings when you are about to do something dire the first time, then like many other pop-up warning systems it allows you to check a box so you don't have to see that pop-up again. Seems like a reasonable compromise to me.

For all I know there is a pop-up warning currently in addition to the warning on the tab, since I haven't tried the current experimental feature.