[GUI Plugin] Find Duplicates

[jgforbes]

Thanks for the link. That solved the problem.
However, the question remains as to how it got changed.

[chaley]

Originally Posted by jgforbes

However, the question remains as to how it got changed.

The first line of that post is (emphasis added)

Some plugins, in particular Find Duplicates, can switch calibre's search from showing only matched books to showing all the books with matched books highlighted.

[capink]

To add to what chaley said, the plugin restores the search back to what it was after it finishes. Since in your case calibre crashed, the restore step never happened.

[Wiggo]

Originally Posted by capink

Example No. 4 How to match by alias instead of author.

Spoiler:

This is one of the most interesting examples on how to use templates to find duplicates. It assumes that you are already using user categories in Calibre to set up author aliases as this post by @BetterRed suggests (which is further illustrated here in this attachment).

After creating your aliases as explained above, we will use the template below, which will substitute the author name with the alias if it finds one in the user categories, otherwise it returns the author name unchanged. We can use this template directly as in previous examples, but this time it makes sense to use the template to create a composite column containing aliases that can used for other things, beside duplicate finding, like searching in Calibre by alias.

• Add the code below to Calibre templates (we have already how to do this explained in example No. 3).
  
  

  def evaluate(self, formatter, kwargs, mi, locals, val, col_name, user_cat_prefix):
      new_val = ''
      if hasattr(mi, '_proxy_metadata'):
          all_cats = mi._proxy_metadata.user_categories
          cats = {k:v for k,v in all_cats.items() if k.startswith(user_cat_prefix)}
          SEP = mi.metadata_for_field(col_name)['is_multiple'].get('list_to_ui', '')
          new_val = set()
          if SEP:
              val_ = val.split(SEP)
          else:
              val_ = [val]  
          for user_cat, v in cats.items():
              repl = user_cat.lstrip(user_cat_prefix)
              for user_cat_item, src_cat in v:
                  if src_cat == col_name:
                      for item in val_[:]:
                          if item == user_cat_item:
                              new_val.add(repl)
                              val_.remove(item)
                          else:
                              new_val.add(item)
          if new_val:
              return SEP.join(list(new_val))
      return val

  The rest of the the template setting should look like this:
  

  Function: replace_with_category
  Argument count: 3

  Now, press create button to add the code to Calibre's templates.
  
• Create a new composite column with these exact settings (see attachment 8):
  
  

  Lookup name: alias
  Column heading: alias
  Column type: Column built from other columns, behaves like tags
  Template: {authors:replace_with_category(authors,Authors.Alias.)}

  You will have to restart Calibre for the new column to be effective.
  
  Note: If you don't choose the "column built from other columns, behave like tags" the duplicate search will not work.
  
  Note: if you decide to use different structure for your user category, you have to replace Authors.Alias. with whatever the user category hierarchy you are using
  
  Note: In simple template mode, spaces are significant. Don't add any space after the comma.
  
• Now, to start matching by alias choose the title field and whatever algorithm you want (I will choose similar match for the title).
• Instead of using the authors column we will use the newly built alias column, you will notice as the soon as you choose this column, a new "contains names" checkbox appears (see attachment 9), you have to check it for this duplicate search to work.
  
  This setting tells the plugin to split contents of the alias column using "&" as it does with the authors.
  
  Note that we don't need this option when dealing with the authors column — on any non-composite custom column — as the plugin can know from their metadata that they contain names, and acts accordingly. Unfortunately, Calibre does not give us an option when we create a composite column to indicate that it contains names, so we have to use this option to tell the plugin what to do.
  
• Now choose an algorithm you want for the alias column (you can choose identical here) and click OK to start the find duplicate search.

Note: The above template can be used for purposes other than aliases. Any user categories you apply for series or authors (like nationality) can be added to a composite column appearing in books.

Hi capink,

it is possible to separate the aliases like in the normal authors (black instead of blue ampersand)?

I have a group-search "pen-names" with authors and #alias and do my similar authors search by this pen-names. If I search for "Robert Lynn Asprin" all the books with Robert Asprin & Co-Author will not be found, because the alias is e.g. "Robert Lynn Asprin & Linda Evans" and not "Robert Lynn Asprin" & "Linda Evans".

I hope I expressed it clearly

[capink]

Originally Posted by Wiggo

Hi capink,

it is possible to separate the aliases like in the normal authors (black instead of blue ampersand)?

Unfortunatly, this is not possible. Custom columns in calibre do not support ampersands as a separator. I opened a bug report a while back about this issue, but it was not addressed because of technical difficulties.

Originally Posted by Wiggo

I have a group-search "pen-names" with authors and #alias and do my similar authors search by this pen-names. If I search for "Robert Lynn Asprin" all the books with Robert Asprin & Co-Author will not be found, because the alias is e.g. "Robert Lynn Asprin & Linda Evans" and not "Robert Lynn Asprin" & "Linda Evans".

I hope I expressed it clearly

The solution for your problem is to do away with ampersands as separator and fall back to using commas. You can do this by modifying the template function as below

def evaluate(self, formatter, kwargs, mi, locals, val, col_name, user_cat_prefix):
    new_val = ''
    if hasattr(mi, '_proxy_metadata'):
        all_cats = mi._proxy_metadata.user_categories
        cats = {k:v for k,v in all_cats.items() if k.startswith(user_cat_prefix)}
        SEP = mi.metadata_for_field(col_name)['is_multiple'].get('list_to_ui', '')
        new_val = set()
        if SEP:
            val_ = val.split(SEP)
        else:
            val_ = [val]  
        for user_cat, v in cats.items():
            repl = user_cat.lstrip(user_cat_prefix)
            for user_cat_item, src_cat in v:
                if src_cat == col_name:
                    for item in val_[:]:
                        if item == user_cat_item:
                            new_val.add(repl)
                            val_.remove(item)
                        else:
                            new_val.add(item)
        if new_val:
            return ', '.join(list(new_val))
    return val

Note, however, that this will break author names with commas in them.


Edit: Alternatively, you can keep your original setup, and modify your search from:

#alias:"=Robert Lynn Asprin"

into:

#alias:"Robert Lynn Asprin"

Edit2: If search you are referring to Find Duplicates (Rather than calibre search), the advanced mode of the Find Duplicates has an option called ("composite has names"), all you have to do is to tick that option.

[chaley]

Originally Posted by capink

Unfortunatly, this is not possible. Custom columns in calibre do not support ampersands as a separator. I opened a bug report a while back about this issue, but it was not addressed because of technical difficulties.

Actually, ampersand is the only separator other than comma that is supported. You get that by checking the "Contains names" box in the column definition.

Ahh, but this is supported only in is_multiple text columns, not composite columns.

[capink]

Originally Posted by chaley

Actually, ampersand is the only separator other than comma that is supported. You get that by checking the "Contains names" box in the column definition.

Ahh, but this is supported only in is_multiple text columns, not composite columns.

I don't have it (or can't see it).

Edit: I just read the second part of you reply.

[chaley]

Originally Posted by capink

I don't have it (or can't see it).

I realized after writing my post that you are talking about composite columns. That is why I added the qualifier about "Contains names" appearing only on is_multiple text columns.

Random thought: I haven't taken the time to understand exactly what you are trying to do, but it seems that you are trying to populate a column with the list of alternate names for the authors. Why not do this with action chains, where the chain modifies a "standard" is_multiple composite that has "Contains names" checked? The chain would trigger when a book is modified, updating the target column.

[capink]

Originally Posted by chaley

I realized after writing my post that you are talking about composite columns. That is why I added the qualifier about "Contains names" appearing only on is_multiple text columns.

Random thought: I haven't taken the time to understand exactly what you are trying to do, but it seems that you are trying to populate a column with the list of alternate names for the authors. Why not do this with action chains, where the chain modifies a "standard" is_multiple composite that has "Contains names" checked? The chain would trigger when a book is modified, updating the target column.

The original solution quoted by @wiggo predates Action Chains. The alternative you are suggesting is the one I would go with right now. I am not sure how familiar or proficient @wiggo is with Action Chains, though.

[theducks]

I have the same Grouped Search (penned_as)
It links the normal Authors and my custom column #pseudo4 (Ampersand separated text)

As Chaley suspects, the column type is not appropriate

[Wiggo]

Originally Posted by capink

The solution for your problem is to do away with ampersands as separator and fall back to using commas. You can do this by modifying the template function as below

Edit2: If search you are referring to Find Duplicates (Rather than calibre search), the advanced mode of the Find Duplicates has an option called ("composite has names"), all you have to do is to tick that option.

Thank you so much - this works! As i have no authors with commas, this fits perfectly.
The fuzzy search of Find Duplicates works pretty well for me so far, therefore and because I was overwhelmed by the rest of your tutorial I'll stick with it for now.

Originally Posted by theducks

I have the same Grouped Search (penned_as)
It links the normal Authors and my custom column #pseudo4 (Ampersand separated text)

As Chaley suspects, the column type is not appropriate

I got the idea from you - belated thank you!

[capink]

Originally Posted by Wiggo

Thank you so much - this works! As i have no authors with commas, this fits perfectly.
The fuzzy search of Find Duplicates works pretty well for me so far, therefore and because I was overwhelmed by the rest of your tutorial I'll stick with it for now.

If you are using commas instead of ampersands, do not tick the option "composite has names" in the Find Duplicates.

[Wiggo]

thx!

Regarding the Action Chain, even though I actually have no clue.
If I understood the problem with the names, the ampersands and the multiple composite columns correctly, chaley's solution doesn't work either, because these columns don't have "Contains names", right?

[chaley]

Originally Posted by Wiggo

thx!

Regarding the Action Chain, even though I actually have no clue.
If I understood the problem with the names, the ampersands and the multiple composite columns correctly, chaley's solution doesn't work either, because these columns don't have "Contains names", right?

If you use action chains then you wouldn't use a composite column, instead using a standard "text, shown in tag browser" column with "Contains names" checked. The action chain (actually an action in the chain) would update that column whenever the book changed using a similar technique as the composite-based solution.

However, if what you have is working then it is probably best to stick with it. Action chains can require substantial template-fu.

[Wiggo]

OK, I understand.

Of course I stick with "mine" as I have no idea about templates and stuff like that