[GUI Plugin] Find Duplicates

[drMerry]

Originally Posted by kiwidude

@drMerry - I think the simplest solution to your issue right now is to do Show all book exemptions, remove all those ones in that group, and instead use a search restriction before you search for duplicates.

Spoiler:

The problem I believe is due to the way exemptions are stored, as every book is being stored as being exempt with every other book. This isn't a scalable approach if (as you have) your group contains a massive number of books.

Right now I will see what others think on the dev thread about how we solve it - either we prevent you marking the group as exempt in the first place by putting in a threshold, or we change the way exemptions are stored. However you have a workaround in the meantime I believe.

In what I would term "normal" usage your exemption groups should not be very big - the 99% scenario I perceive as being 2-3 books/authors in a group. However allowing very fuzzy searches and in your case storing a large number of near duplicate titles as people will have who store magazines etc this situation will arise.

At the moment this is the solution (for first deduplication) but my new post (typing together again )shows some more detail.

[drMerry]

Another one I was thinking of:

Duplicate in tag.
Something different than the earlier post.
When adding books to my library, I give them a default tag (Newly added).
Calibre has some functions to handle added duplicates, but this is not a complete range. If I add 10 books, I would like to say add / skip / merge for every single book based on existing books and formats.
If I add 1000 books, I want to add them all and sort it later.
In both cases Calibre can't help me. I have to add them all and look for duplicates later.

So it would be nice to have a function like:
Find duplicates one or more books containing tag...

In this way, I can skip all books I had in lib before my new collection arrived, but if one of the new books matches my library, I will see it.
(again filtering does not help in this case because I want to test all books but only want to see results based on a filter)

[chaley]

Just a note to say how useful this plugin has turned out to be. I thought I had a clean library. It was clean to 99%, but not totally. The changes to force to ASCII helped me find several cases where an author appeared with and without accented characters. The soundex matches helped find cases of slight author and title misspellings. Having author-only and title-only helped with finding problems with books in both French and English. Working with other combinations found a few other problems. And on top of all that, it is very fast.

Very fine work!

[Xenophon]

I just used the plugin on my library, and promptly turned up a few duplicates -- even though I've made a major effort to keep the library clean.

[latexsalesman]

This is seriously one of the best plugins that I have ever used for our wonderful Calibre!!!

kiwidude I have to say how much that I love this plugin...

Bravo!!!!
Thank You!!!
You Rock!!!

Thanks So Much Great Work...



I want to be your LatexSalesMan!!!!

[kiwidude]

lol... thanks all for the positive comments, much appreciated.

[zespectre]

This plug-in just helped me locate three (rather expensive) books I though had gone missing. Somehow, somewhere the names became corrupted but they turned up in one of the "fuzzy" searches. I fixed the metadata and am now a very happy camper!

[kiwidude]

That sounds a rather unusual side effect benefit of using the plugin! Glad it helped you out, I hate "losing" things too.

[mayoi]

Works beautifully and fast. Karma to you!

[kiwidude]

Changes in this release:

• Add support for binary comparison searches to find book formats with exactly the same content
• Replace how exemptions are stored in the config file to make more scalable
• No longer calculate exemption preview detailed messages for the confirmation dialog for performance
• Compare multiple authors for most author algorithms to increase duplicate coverage.
• Change Manage exemptions dialog to have tab for each author with exemptions and show section only if have exemptions
• Include swapping author name order in all but identical author checks. So A B / B A or A,B / B,A will match.
• Disable the Ignore title, identical author combination as will not a valid one (never duplicates)
• Allow the remove, mark current and mark all group exemption dialogs able to be hidden from showing again.
• Allow various count of result and no result information dialogs able to be hidden from showing again.
• Allow user to reset confirmation dialogs related to find duplicates from the configuration dialog

Important:
I have changed the way your exemptions are stored in the config file, to make this plugin work more efficiently with very large groups of exemptions. You will lose any book or author exemptions when you upgrade to this version. However this is a one-off problem for longer term benefit.

A key change of particular note is the way authors are compared. Previously this plugin only considered the first author of a book if it had multiple. So a book by "Lee Child & David Gemmell" could not match one by "David Gemmell" without using an ignore author search. In addition the plugin will now catch cases of the author names being stored in the wrong order. Previously it would compare "Lee Child" with "Child, Lee". Now it will also match books where the metadata is incorrect like "Lee, Child" and "Child Lee".

With these changes you should now be able to catch more duplicate situations in your library. Special thanks again to chaley for his suggestions and testing with this release.

[butterbescotch]

Thanks for the plug in!

ED: It's now fixed.

ED: I've got 3685 duplicated titles!

[kiwidude]

You are welcome - and I'm not sure what is now "fixed" but I am glad it is...

[theducks]

Originally Posted by butterbescotch

ED: I've got 3685 duplicated titles!

That is more duplicates than I have books in my Library

[Starson17]

Great work kiwidude. I'd written some duplicate finder code and thought my library was in pretty good shape, but I found a variety of book and author duplicates with the new plugin. Kudos to you!

[kiwidude]

Thanks Starson17, that is very cool to hear. After all it was initially your interest in humoring my ramblings before chaley joined in that made me think it was worthwhile attempting it.

The only idea I can recall being suggested for this that is outstanding is the ability to filter the result groups. I don't want to add it unless more than one person requests it though - it isn't something I would use myself but I understand drMerry's point that there are circumstances that applying a search restriction before you search will not work as the alternative.