[GUI Plugin] Find Duplicates

[TheMadGuy]

I'll do my best to patient ;-) Thanks for support

Could you maybe keep in mind that it also doesn#t find "Christian" / "Christina"
In my actual case they are two different writers, but I also had a misspelling once....

[kiwidude]

Originally Posted by TheMadGuy

Could you maybe keep in mind that it also doesn#t find "Christian" / "Christina"
In my actual case they are two different writers, but I also had a misspelling once....

Soundex finds the Christian/Christina example. Similar searches will *never* find misspellings, you have to rely on the other algorithms for that.

I just now realised that this solves this query too from a couple of pages ago:

Originally Posted by Winnito

I'm puzzled how when I do "similar" search in Metadata duplicates it can't find what I guess should be obvous hits like:

Doe, John
Doe, John L.

I have so many authors in my database with and without their middle names, but can't find most of them in duplicate.

Similar gives nothing, Fuzzy gives 20x more false positives...not even close to being useful. Am I missing something? TNx

Back then I didn't have the time to properly read/understand what was being asked for - tonight this change will solve the issue for Winnito too.

[kiwidude]

The release notes for changes and the plugin zip file to download are here:

https://github.com/kiwidude68/calibr...icates-v1.10.0

A change worth re-mentioning is as per recent discussion posts in this thread I have removed the "Advanced" mode capink had previously added to the plugin. Again my thanks to capink for all their other contributions and fixes over the years, much appreciated!

If a few of you could be so kind as to manually download/install the zip file available in the above link and let me know how well this version works for you it would be appreciated. At which point I will release it "officially" for everyone else via the first thread post as usual.

[TheMadGuy]

So my first try with the new version brings much better results. But I had to recognize that there is one problem still present: I got the name list in the left window, and most times the two similar ones one above another. But no matter on which one I click, the shown book remains the same. And the tool blocks action to calibre itself. So whenever I am not sure if the two similar writings are the same writer, I have to take a note and check this name(s) when the tool is closed. I don't know if this could be changed in any way (as it has been from day one), so the updated version delivers much more and much better results from my point and I only can say: Thankx!

[kiwidude]

Originally Posted by TheMadGuy

So my first try with the new version brings much better results. But I had to recognize that there is one problem still present: I got the name list in the left window, and most times the two similar ones one above another. But no matter on which one I click, the shown book remains the same. And the tool blocks action to calibre itself. So whenever I am not sure if the two similar writings are the same writer, I have to take a note and check this name(s) when the tool is closed. I don't know if this could be changed in any way (as it has been from day one), so the updated version delivers much more and much better results from my point and I only can say: Thankx!

If you have "Show matching books" ticked in the Find Metadata Variations dialog, the calibre window in behind should be showing you all the books for the particular (pair) of variations you have selected. In the search bar you can see it is displaying a query for that variation e.g.:
authors:"=Biggle, Lloyd" or authors:"=Biggle jr., Lloyd"

Toggling between the two Biggle entries in the Authors list will not change the net result in the calibre books display, because it is just changing the ordering of that "or" clause. However the point of that dialog is that it should normally allow you to do everything you need to do from it.

So if you decided that "Biggle, Lloyd" was the author name you wanted to use for all the books, then you would select the author variant you want to keep in the list on the left of "Biggle, Lloyd". That way the "Rename to" dropdown defaults to the same name you want. by default all variations in the right are selected - of course these represent the set that will get changed. Then click on Rename. So all books that had an author of "Biggle jr., Lloyd" would now have an author of "Biggle, Lloyd" instead now. At which point the Biggle entries disappear from the list because there are no more variation conflicts in that group.

So there should be no need to write names down - at least not for authors there are not many books for. However if it was an author with a lot of books and you couldn't scroll down to double check all the books for the other author variant to make sure before renaming then indeed some caution could be advisable by taking a note.

Anyways thanks for confirming it is all working ok, I will release it properly tonight.

[kiwidude]

Release notes:
https://github.com/kiwidude68/calibr...icates-v1.10.1

Thanks to @chaley for the help, Find Metadata Variations now respects any virtual library search restriction you have applied. So more consistent with the Find Books behaviour.

I also made it possible to have the calibre search result query reduced when clicking on different results in the Find Metadata Variations dialog. The default behavior remains the same - of the search showing you all books for all variations. But if you deselect a variation in the right side of the dialog then it gets removed from the search query in the calibre window behind.

This should make it slightly nicer for dealing with more results than can fit on screen, given the dialog is modal and blocks you from clicking/scrolling.

I'm also aware that by changing the Similar Authors to ignore initials, you now get more false positives than before. Since "Smith, John" and "Smith, John H." really can be two different authors sometimes. There is no ability to apply exclusions in the Metadata Variations dialog (Ignore is temporary). I know this has been requested by others also. But before I consider adding that I would have to properly resolve the issues some users have with enormous exclusion lists that impact performance before making it worse. That is a rainy day exercise that won't be happening any time soon...

Enjoy!

[Terisa de morgan]

Well, the new version has just given me a shock. I thought my library was cleaner that it is, according to the new version (and the plugin is right....). All because my options have been changed...

[alumoi]

I know my search abilities are not that great, but where can I find the latest working version with the advanced option enabled?

[kiwidude]

Originally Posted by alumoi

I know my search abilities are not that great, but where can I find the latest working version with the advanced option enabled?

You could try a PM to capink - 1.9.7 was the last version. I don’t want that version publicly posted in this thread because I cannot offer support for it, and all my new fixes/additions are only going to apply to this official version. Having multiple versions will confuse everyone and waste time trying to replicate stuff that has since been fixed.

I would ask - what is the actual type of search that you are doing that you needed the advanced feature for? If it is commonplace enough then I could potentially make some tweaks. For instance one use case I saw was for handling author initials which the 1.10.x versions do already.

[alumoi]

Thank you for your replay.
I used advanced search to identify identical authors in 2 different calibre libraries and automatically select the books.
If I use find library duplicates it only tells me I have xxx authors with potential duplicates but if I click OK it doesn't show me the actual books with potential duplicates as advanaced used to.

[alumoi]

To further clarify things, I'm not interested in potential duplicate books, only in potential duplicate authors.
My 'main' library is BEST and current working library is CALIBRE (yeah, I know... ). I want to copy to BEST books from CALIBRE with identical authors (single or multiple authors). With advanced option I just searched for identical authors, clicked OK and a I had a list of all the books I wanted.
Now, as I've said before it only tells me I have xxxx authors and the only option (that I know of) is to click on show details, save the log, manually search for the authors and go from there.

[kiwidude]

Hi alumoi, thanks for the use case. I will take a look as to why I made it behave that way - perhaps I made an assumption as to it not being very useful to show the results and you are just proving me wrong by having a use case for it.

[alumoi]

Much appreciated.

[kiwidude]

Edit: removed while trying for a better performing option

[alumoi]

Hmm, running for almost 3 minutes and still working. Not finding anything and making the main calibre marked as not responding in task manager. Same options as in the image I posted.
I will update when and if it will finish.
As a side note, the old version (1.9.7) was finding and dispalying the results in under 2 minutes.
If it matters, main CALIBRE library has over 280K books and BEST over 85K.

/LE, sorry, can't test under linux right now, only W10.