[GUI Plugin] Find Duplicates

[kiwidude]

Originally Posted by DMee

Trouble is "Similar" authors produces nothing and "Soundex" or "Fuzzy" produce loads with some errors hidden amongst lots of "cruft" I beleive you call it :-)

e.g. Arthur Charles Clark was buried 10 screens down - is it possible to manually adjust the settings to limit A.Bronte and A.Brown clashing but pick up Arthur C. Clark and Arthur Charles Clark ??

Now that I've had a chance to get back to this plugin there was one thing that stood out for me on this. A.Bronte and A.Brown should *not* have been appearing together. I figured out why they are and will fix it. Its to do with the permutations of flipping author names, and means that A.Bronte and A.Brown both end up in a group with a permutation of "B A" caused by flipping the first/surname around and taking the first "initial". This change should reduce a lot of false positives from the fuzzy search, both for the normal Find Book Duplicates and the new Find Metadata Variations screens.

I have also added the soundex spinbox into the Find Metadata Variations screen to give you granularity on that.

Thanks for the feedback.

[DMee]

Originally Posted by kiwidude

I'll PM chaley - it is something we have discussed previously and I thought had been "fixed" in such a way that I did not have to do a "cleanup" call before I do every query but obviously not. At least I can see why it is happening...

So.... whats the interminable wait every time I OK some edits if not Calibre cleaning up?

[DMee]

Originally Posted by kiwidude

Now that I've had a chance to get back to this plugin there was one thing that stood out for me on this. A.Bronte and A.Brown should *not* have been appearing together. I figured out why they are and will fix it. Its to do with the permutations of flipping author names, and means that A.Bronte and A.Brown both end up in a group with a permutation of "B A" caused by flipping the first/surname around and taking the first "initial". This change should reduce a lot of false positives from the fuzzy search, both for the normal Find Book Duplicates and the new Find Metadata Variations screens.

I have also added the soundex spinbox into the Find Metadata Variations screen to give you granularity on that.

Thanks for the feedback.

Not a problem - thanks for the amazing plugins

[kiwidude]

Originally Posted by DMee

So.... whats the interminable wait every time I OK some edits if not Calibre cleaning up?

Any lag you are experiencing there will be calibre copying and renaming directories/books for an author. That is different however to the pruning of the database tables. I've got a workaround of calling db.clean() before I ask for anything with ids (authors, publishers, series or tags) and I've chucked a PM to chaley to see whether he wants to do anything else about it, but at least it doesn't error any more.

[DMee]

So when would Calibre normally clean up if you didn't run plugins??

[kiwidude]

Probably at startup I would imagine - which is why you found that restarting calibre allowed the plugin to start working again. Undoubtedly there are other activity points which may trigger it, I haven't bothered looking since it isn't relevant and I have a workaround.

[chaley]

Originally Posted by kiwidude

I'll PM chaley - it is something we have discussed previously and I thought had been "fixed" in such a way that I did not have to do a "cleanup" call before I do every query but obviously not. At least I can see why it is happening...

The fix is now in trunk.

Originally Posted by kiwidude

Probably at startup I would imagine - which is why you found that restarting calibre allowed the plugin to start working again. Undoubtedly there are other activity points which may trigger it, I haven't bothered looking since it isn't relevant and I have a workaround.

FWIW: the cleanup happens at exit.

[kiwidude]

Changes in this release:

• Now requires calibre 0.8.59
• Add a Find metadata variations option to search for author, series, publisher and tag variations, and allow renaming them from the dialog.
• Fix bug of fuzzy author comparisons which will no longer compute a reverse hash to reduce the false positives it generated

Thanks to those of you who tested and offered feedback on the beta.

@chaley - I appreciate your fix for calibre will be in 0.8.61, I've included a workaround in this plugin if a user is running 0.8.59 or 0.8.60 just so I can release it now.

[kiwidude]

A number of people over the last year or so have asked for the ability to do cross-library duplicate comparisons. I've finally reached the breaking point of deciding a personal need for using it myself and have put something together.

So on the menu you now have a "Find library duplicates..." option with a screen that looks similar to the regular Find book duplicates one. However you can see you choose a library to compare against up the top, and have way less options at the bottom since UI stuff isn't relevant.

As I have stated previously the only practical approach to working with the results is to simply produce a report. You can copy it to your clipboard / text editor, switch to your other library and manually decide what to do about them. It would be an *enormous* amount of work to wrap a cross-library gui for merging/deleting so it isn't going to happen - we discounted having a special plugin screen when originally designing this plugin and the reasons more than equally apply here.

A couple more things to note:

• The report only shows the results of comparing books in the current library with books in the target library. It intentionally will not show you duplicates that exist only in the current library, nor duplicates that exist only in the target library. It is for a direct cross-library comparison only.
• If you do an "Ignore Title" search (i.e. matching only authors) then the report output will differ and show you by author name which authors match and the books associated with that author in the target library
• If you do a binary search, it shows you which format is the binary duplicate
• For title/author/isbn searches, it will show you what formats if any exist for that book
• Unlike "Find book duplicates", it is possible with "Find library duplicates" to do an Ignore Title, Identical Author search. This is a combination which makes no sense when only looking in your own library, but makes perfect sense when comparing across libraries.

I haven't even had a chance to run this on my libraries at home as yet so it may have gremlins. I have run it against some small test libraries and it all seems to do the right thing but any feedback is appreciated.

[kiwidude]

Changes in this release:

• Add a "Find library duplicates" option for cross-library duplicate comparisons into a log report
• If currently running a duplicate book search and execute a metadata variation search, clear search first

See my comments above for a description of how the cross library duplicates search work. Note I did find a bug in the beta for "Author only" searches which is corrected in the 1.5.0 release (so don't be surprised to see more results now than you saw in the beta!).

[BetterRed]

Win7 Pro SP1|Calibre 0.8.61 - for Library Compare. This will save me a lot of time.

Minor Inconsistency

The Compare With Library drop down does not show the list of Calibre Libraries, as I anticipated it would.

Instead I had to navigate to the corresponding folder via the Browse button, using the standard Windows Open Folder dialogue, which opened up assuming my Calibre Libraries were in my Windows Documents Library. I can imagine this might confuse some people. Calibre has a list of Libraries & Folder Paths behind the Library Quick Switch flyout menu. IMO that list should be used here.

Suggested Enhancement

Save results to a text file - I appreciate I can copy it to the clipboard, open a text editor, paste the clipboard and save the file (or shock horror - print it) - but ...

BR

[kiwidude]

There is a reason behind the library path thing. The way calibre chooses to display library names has some limitations which from time to time the odd user has come on to complain about. For users who only have a small number of libraries I completely agree that a dropdown of names would be consistent and simple. Three issues though. First calibre has a limit to the number of libraries it remembers, and there has been at least one user I have seen come on these forums and complain about that. Secondly I need to cater for the user comparing against a library they may not yet have opened in calibre on this machine. And thirdly what about a user who makes a copy of their library folder onto a different drive for backup and is using this feature to compare. Just displaying the calibre library folder is not enough to differentiate which one they are pointing to.

So I took the easy way out and took the same approach as the switch/create library dialog - that is the part of calibre I have chosen to be "consistent" with.

Your second suggestion of a save button is an excellent one, I will add that in.

[kiwidude]

Changes in this release:

• Add a "Save log" button for the "Find library duplicates" result screen.

[Stampercam]

I'm trying to use the find duplicates plugin and am getting the following:

calibre, version 0.8.61
ERROR: Unhandled exception: <b>MemoryError</b>:

Traceback (most recent call last):
File "calibre_plugins.find_duplicates.action", line 187, in toolbar_button_clicked
File "calibre_plugins.find_duplicates.action", line 152, in find_book_duplicates
File "calibre_plugins.find_duplicates.duplicates", line 123, in run_book_duplicates_check
File "calibre_plugins.find_duplicates.book_algorith ms", line 58, in run_duplicate_check
File "calibre_plugins.find_duplicates.book_algorith ms", line 111, in convert_candidates_to_groups
File "calibre_plugins.find_duplicates.book_algorith ms", line 196, in partition_using_exemptions
MemoryError

I'm using win 7
calibre v 0.8.61 and the plugin was just updated to v1.5.1, but I was having the error with the previous ver of the plugin.

Any suggestions?

Cam

[kiwidude]

That part of the plugin hasn't been changed in pretty much forever. Are you running it against some massive library with an obscene amount of duplicates or something?