[GUI Plugin] Find Duplicates

[drMerry]

Originally Posted by Loeffel

May I ask what else do you have in the exception list? I assmume "Mc" and "von", if I understand this right.

in advance of the reaction of kiwidude, these are some other parts that should in this group as well (dutch):
de
der
des
(van de)
(van der)
(van den)
ter
ten
den

[drMerry]

After the last update, duplicate scan is a lot slower. Does this someting have to do with the highlight fix you did on the quality scan? Or is this fix not added on this plugin?

[chaley]

Adding to the honorific list, you should include "de la", "von", "del", and "della", To be picky, they should be in lower case. In French, the de in "De XXX" is not an indication of nobility, while in "de XXX" it is. I think the same thing is true for von in German and van in Dutch.

[kiwidude]

I added "van" because Kovid had added it to the Calibre code a while ago.

This is the full list of words that are currently ignored by this plugin in an author's name (except for identical author searches):
'von', 'van', 'jr', 'sr', 'i', 'ii' 'iii', 'second', 'third', 'md', 'phd'

@drMerry - no changes were made that would affect performance.

[drMerry]

Originally Posted by chaley

Adding to the honorific list, you should include "de la", "von", "del", and "della", To be picky, they should be in lower case. In French, the de in "De XXX" is not an indication of nobility, while in "de XXX" it is. I think the same thing is true for von in German and van in Dutch.

So, than the question is, how is this function implemented. And, maybe more important. Does this mean that "Peter Pan de XXX" should be ignored as duplicate of "Peter Pan"? Or, should you just check "Peter Pan" against "Peter Pan XXX"

[drMerry]

than what about
dr
dr.
prof
m.d.
ph.d.
sr.
jr.
et al.

[Loeffel]

Oh ohh, there I have started something. ;-)

[drMerry]

Something completely else:
Another problem is the maiden name:
If A. B. marries C. D. and the author uses both names (B. and D.) it could be written as:
A. D.-B. (seen as one name on calibre)
A. D. B.
(Other options could be the author still uses B. or only uses D. but that is not 'check-able' (B against D))
When author swaps B and D, it would be recognized as well.
So I think - should be handled as " " in this plugin

[drMerry]

And of course:
Sir, etc.
Maybe you could implement a way of adding this pre and suffixes manually by the user while it seems everybody needs other pre and suffixes according to the local used ones.

[kiwidude]

@drMerry - the plugin already strips a whole bunch of characters out of the names: "-+.:;" so most of what you have posted above is already taken care of.

[drMerry]

I had a case that it missed yesterday, hope I can find it for you.
What do you mean by stripping. a-b becomes ab or a b?
and a - b?
a b?
a b?
a b?
Does this has negative effects or is more than one space always seen as 1 (also in soundex)?

[kiwidude]

Those particular characters get replace with spaces, so a-b will become "a b". Other characters get removed without space substitutions.

You have the code in the plugin - look in algorithms.py for "get_author_tokens()".

[drMerry]

thanks.
I think the problem was in the ; part.
, is assumed to be ln, fn.
I think there is another way,
if you have "(((\w+)\s(\w+))+[,;$]){2,*}" you could (safely?) assume it is not ln, fn but multiple authors

[kiwidude]

I don't believe this plugin is the correct place to start reinterpreting author names as multiple authors. It is crappy metadata, garbage in, garbage out.

I would suggest it is a Quality Check type thing, except the solution is so simple it is not worth bothering Quality Check with, since you can just search authors:;

[madeinlisboa]

I was looking for this. Thank you