[GUI Plugin] Find Duplicates

[Winnito]

Originally Posted by capink

@kiwidude: He does not want to use fuzzy author matching, because for him, it produces too much false positives.

This question was asked by the same user before, and an answer was provided here.

I was amazed that you remembered, only to find out it was you helping me fix it. Cudos Capink!

[capink]

During recent discussions with kiwidude, the idea of removing the advanced mode from the plugin has come up. The logic of the proposal is that the feature does not seem to be widely used, and it comes with huge codebase.

Removing the feature will make it easier to maintain the plugin, and as bonus, will allow the plugin to support older calibre versions as well. I am not sure I will be able to contribute code for the long term, and I don't think it is fair to burden whoever maintains the plugin with supporting a huge codebase that is seldom used.

I developed this feature for my own use, and I continue to use it. But with the benefit of the hindsight, maybe it was not the best decision to include it in the official version.

The current version which has the advanced mode is stable enough to be used by whoever wish to do so. And given the recent upgrades by calibre to Python 3 and Qt 6, it is reasonable to assume that it will work for for quite some time.

We thought that is better to ask for feedback in this thread before we move ahead with the proposal.

[theducks]

I frequently use Find Library Duplicates prior to merging Intake, but also in case I had moved a title to the Library for younger minds.


Whoops, I use the regular version, not the Advanced versions with the same name

[BetterRed]

If we had two plugins… Find Duplicates, and Find Duplicates (Advanced)… could they co-exist?

BR

[BetterRed]

Originally Posted by BetterRed

If we had two plugins… Find Duplicates, and Find Duplicates (Advanced)… could they co-exist?

BR

And if they can, would FD (Advanced) inherent the features of FD (Basic), or would it replicate them, or would they be removed from FD (Advanced)?

BR

[capink]

Originally Posted by BetterRed

And if they can, would FD (Advanced) inherent the features of FD (Basic), or would it replicate them, or would they be removed from FD (Advanced)?

BR

All of the options above are technically possible. I am not sure forking the plugin would be a good idea, and it would still need someone to maintain it.

In the mean time, the latest version containing the advanced mode can be made available (as a link in a post in this thread) for as long as it works.

[kiwidude]

Originally Posted by BetterRed

And if they can, would FD (Advanced) inherent the features of FD (Basic), or would it replicate them, or would they be removed from FD (Advanced)?

BR

The point that we are raising is that maintaining, documenting and supporting that Advanced features code is an issue - if noone other than capink is actually using it then it should *not* be part of any public Find Duplicates plugin. If anyone comes to the forums to ask about how to use them or has an issue/bug then if capink isn’t around it is problematic. Time is not free, and trying to provide support for 1-2 users in the world if that is all who use it is not practical. Likewise it is more work for maintainers when it comes time for Qt upgrades etc.

Whether capink chooses to just use that particular version that combines both for as long as they can, or pulls that functionality into their own plugin is entirely up to them. Likewise if capink wants to publish it as a completely separate plugin for as long as they are willing to support it that is their choice too.

As I said to capink it can be difficult to know in advance how popular adding a feature to a plugin may be - are we satisfying a requirement for many people or just creating a headache for ourselves of filling some niche requirement. Certainly I suspect many of the options on Modify ePub for instance fall into the latter category. If these Advanced features are something only capink is using then it makes sense to me to remove them.

[BetterRed]

Guys, I was just asking what I thought were pertinent questions, wasn't advocating for any outcome. I have sent a couple of people to the Advanced Metadata Variation feature - but I don't know if they made use of it.

If I had a concern it was this: if a separate Advanced FD retained the features of Standard FD, what happens when Standard FD is functionally enhanced. Would users of Advanced FD be clamouring for the inclusion of the new functionality. So if I was to advocate a position, it would be to remove the functionality of Standard FD from the Advanced FD.

FWIW: re Modify EPUB, if Polish Books could remove the cover and non-dc elements, and put the jacket at the back, I wouldn't need Modify EPUB

BR

[kiwidude]

@BetterRed - we weren't picking on you - we just both happened to post a response at the same time

I understand you were just asking questions, however you are proposing one option which as I explained above is even worse than the current state - of creating a complete copy of Find Duplicates as it is today as "Advanced FD". That makes no sense at all for the reasons I explained above - none of us want to maintain that combination of features in a plugin for the long term. And I certainly dont want to have to maintain the "core" Find Duplicates code in two plugins, nor should anyone else be volunteering to do so.

I think an "Advanced FD" plugin which contained *only* the advanced features menu is indeed a good suggestion to make - but only if there is actual user demand of it. Also capink would have to agree to do all that work to create it, maintain it and support it. Which as they indicated above is not a guarantee in the longer term so it is their choice as to how worthwhile that all is to do for them.

BTW this discussion is in no way any kind of indictment about the code quality or amount of effort put into this Advanced feature by capink. I am extremely grateful to them for all the hours of effort that went into keeping this plugin alive through the python/Qt upgrades with others. But if this particular feature is very minimally used as suspected, then it needs to be removed for the long term benefit of the maintainers to respect their time.

[kiwidude]

And to illustrate my point I have just this moment had a second spelling typo "issue" raised by one of the translators in the advanced features code in as many days. So I have to fix the code/republish the source translations, translators need to verify, then the new translations will be included in the next update. The more code we have the more work for everyone

Shout out to yuchor btw for doing the mighty work on Ukranian translations across all the plugins and spotting the typos - thanks!

[TheMadGuy]

@kiwidude
First of all: Thanks for this AddOn as it helped me a lot in the past.

I am not sure if the problem is now the new Calibre-Version (working on Win10 for the first time with Calibre 6.6) or a reason of your Tool that it does not work as well anymore as it did in the past (with former calibre versions on former windows versions).

I always use the "similar" button for research (length of 99) and am confused:
It finds similarities as "Biggle, Lloyd" and "Biggle jr., Lloyd"
but it niehter finds "Pohl, Frederic" and "Pohl, Frederik"
nor "Hohlbein, Wolfgang" and "Hohlbein,Wolfgang"
nor "Koontz, Dean" and "Koontz, Dean R."

So it is a little bit difficult for me to understand how such close similarities (one letter, one empty space) are not found, but the less similar +/- "jr." always is.

I also tried the fuzzy as well as the soundex once, but the results then are far away from reality.

So could you explain me any possibility to finally find those quite similar one wrong spellings with your tool? Or will there be an update of it to cover them, too?

My only alternative is: If I recognize one of them, I have to search for the wrong spelling manually and then using "search and replace" for the authors section. This might be a way, but I am quite sure that there are much more I never will recognize myself.

[kiwidude]

@TheMadGuy - Thanks for providing those examples. It has been a very long time since I have looked at the code in this plugin for the algorithms myself.

Are you talking about using the "Find Metadata Variations" for these Authors? From my test with it now:

Similar - finds the Biggle (jr.), Lloyd case - no surprise as it strips out common author words like "jr", "sr", "van" etc.
Soundex (length 8) - also finds the Pohl, Frederic/k case.
Fuzzy - all except the Hohlbein comma space case

I agree it should be finding that Hohlbein spacing issue so I can look to tweak things for that. I will also have to remind myself of exactly what all those algorithms are doing under the hood as to whether anything else should be tweaked.

EDIT: I can see why that Hohlbein case is going wrong, that is definitely something I will fix for the next version. It is stripping out the "," before it starts breaking the author name into space delimited parts, an easy fix. Thanks for reporting this!

[kiwidude]

I have also adjusted the author tokens to ignore any single characters (i.e. initials).

So now the results for a Similar search are:


So the only case that isn't being caught by Similar is Frederic/Frederik. Which is what I would expect - that is precisely what a Soundex search is targeting. A "Similar" search is about removing punctuation, author name ordering (and now initials).

[TheMadGuy]

It looks great at your results.... but then this causes the question why I don't see these small differences in my results (like mentioned before). It worked on Win8 with a former calibre (the max. version running on Win8) but that computer passed... so I had no chance than using Win10. Nearly all works fine (it will last a while until quite all does), but I can't understand the search results anymore.
It's the same library as before, the only change is the software version (I got the newest version of your tool). So what could it be?

And with "initials" you mean that single letters are "ignored" (like for Dean "R." Koontz) but "jr." not? I try to understand the logical though behind.... and am not able to

[kiwidude]

@TheMadGuy - rather than dwell on what is in the past since you don't have that machine available any more I suggest you wait until I post a new version with my fixes above and see if the new results make sense to you.

I noticed something else just now that does not seem to be working right. Once I get that corrected I will put a version on this thread that you can test with.