Duplicate detection plugin

[Starson17]

Originally Posted by kiwidude

Hi Starson17 - ahhh, it must be a very small library (less than 20 books perhaps?). I borrowed some code from my quality check plugin and it would suffer from the same issue. I will stick a new version up shortly.

Yes, less than 20.

I seem to have lost a post, but basically, it said I'd find a "mark all duplicate groups as not duplicates" useful. I also wondered if you were persistently holding the duplicate groups somewhere, or just rerunning the finder each time. Either way, some way to "clear all" persistent entries (dupes and false positives) would be useful, if you don't already have it.

Edit: I suppose the not duplicate groups would be cleared by showing all books in a not duplicate group, selecting all and choosing remove from not duplicates.

[kiwidude]

Originally Posted by Starson17

Yes, less than 20.

Cool - this version should definitely fix that then. It was to do with updating the status bar while it scans, and not updating it too frequently as Calibre stops drawing updates after a while. However this plugin is so blindingly fast searching for dups I will probably just rip out the status bar updates completely anways.

I seem to have lost a post, but basically, it said I'd find a "mark all duplicate groups as not duplicates" useful. I also wondered if you were persistently holding the duplicate groups somewhere, or just rerunning the finder each time. Either way, some way to "clear all" persistent entries (dupes and false positives) would be useful, if you don't already have it.

I didn't delete it, honest

At the moment the "Persistence" is as follows:
(1) The sets of duplicate groups are held in memory only. As it takes less than a second to run the most complex duplicate search on a 40,000 book library requiring a user to re-run when they restart calibre/switch libraries is not a decision to be made based on performance (though it might on other factors).
(2) The sets of "Not duplicates" will be stored in the JSON config file, and obviously manipulated in memory.

So in terms of "clearing" the results of a find, strictly speaking the user doesn't need to do anything as books are not updated with custom columns etc. There is a temporary in-memory special marking of the group under consideration but I don't think that's worthy of note. As soon as they user does some other action reusing that feature like a Quality Check or using Temp Marker that gets reset anyways.

To clear all your "not duplicates" the user would just use the "Show marked not duplicates", select all and choose "Remove from not duplicates".

[Starson17]

Originally Posted by kiwidude

I didn't delete it, honest

How can I possibly trust a moderator with as many posts as you, who runs this like his personal fiefdom?

FYI: It runs on the small test library now.

Selecting a single book and "mark as not duplicate" gives a name error. I know I'm getting ahead of your coding.
Selecting more than one book tells me I clicked the button. I assume the latter is a code stub, but what about the former?

[kiwidude]

Originally Posted by Starson17

How can I possibly trust a moderator with as many posts as you, who runs this like his personal fiefdom?

Haha, yeah I'm such a ego-driven biatch.

Selecting a single book and "mark as not duplicate" gives a name error. I know I'm getting ahead of your coding.
Selecting more than one book tells me I clicked the button. I assume the latter is a code stub, but what about the former?

Both are just code stubs, one obviously better than the other . I'm writing the false positive stuff now so the next version should have those functional.

[Starson17]

Originally Posted by kiwidude

the next version should have those functional.

Congrats and thanks for tackling this. You seem to have it well in hand.

[kacir]

I am very impressed by the 0.1.2 version.
Almost a dream come true. It found lots of duplicates I missed when I was checking for duplicates manually writer after writer. It also works surprisingly fast on a quite large library. I thought that the time would grow exponentialy with the number of books compared against each other, especially when using fuzzy author AND fuzzy book title algorithm.

I am using fuzzy book name and fuzzy author name and it finds duplicates that have a slightly altered title - like missing "The", plus words like ": A novel" written after name of the book and altered author name (Firstname Lastname versus F. Lastname) *at the same time*.

Now, I have to run the plugin, remove all the duplicates and after that report duplicates the plugin failed to identify.

I have mapped Next duplicate to Ctrl+n.
Can I map it to Ctrl+RightArrow? How?

Thank you Grant.
Your other plugins work very well too, and they make Calibre even more powerful tool.

[kiwidude]

Hi Kacir, thanks so much for trying it out and offering feedback.

Yes I think "Similar author, similar title" is going to be the most often used of the choices, it seems to have a nice balance of not too many false positives. Eventually I will add "fuzzier" algorithms but as a starting point it should suit most needs.

In terms of the keyboard shortcuts, unfortunately Ctrl+Right is a bit too used by the core Qt library so it will only work where focus is in certain places (like not in the search box). I will probably remove my current default keyboard shortcuts when it becomes 1.0, too hard to find something that will please everyone. For instance ctrl+n (or arrows) means taking the hands off the mouse, which you will want them on to do selections for merging etc.

I've found a few quirks I'm fixing in 0.2 (and no doubt introducing new ones). At the moment you will find it is actually going "backwards" when you go next for instance if you use a ctrl-based keyboard shortcut, schoolboy error when I copy/paste code.

In terms of false positives I have decided that rather than allowing a user to select just a subset of rows to "Mark as not duplicates", instead the right-click will just be "Mark group as not duplicates". So if the group contains a mixture of results (genuine duplicates and false positives) that means you must merge the genuine ones first and then can use the menu option to say "everything else is not a duplicate" in that group. If (the more likely case) the group only contains non-duplicates then the user doesn't have to faff with selecting rows. This approach should be a little simpler and certainly avoids more complex issues like having to repartition the group the user is currently viewing (maybe only Charles will understand that remark). I will still also have the "Mark all groups as not duplicates" option for users who just want to keep skipping to the end and then just do one click to say everything left is not a duplicate.

Charles you may also be pleased to hear I am going to take advantage of your "dictionary" capability of set_marked_ids(). So rather than doing a search of "marked:true" I can do something like "marked:duplicate_group_1_of_3", and "marked:not_duplicate_books" etc depending on the set being viewed. Gives the user another visual queue as to what is being displayed.

[tomd365]

Kiwidude,
Here is hopefully, some clearer feedback. When the duplicate finder locates 3 books that match and I merge them with "M", the books will merge fine. When I press Ctrl+\ to go to the next set of records, I get the error message below. It is not a big deal - just rerun the duplicates and carry on. It only seems to be a problem when merging 3 records.

Tom D in Michigan

calibre, version 0.7.54
ERROR: Unhandled exception: <b>KeyError</b>:679

Traceback (most recent call last):
File "calibre_plugins.find_duplicates.action", line 82, in show_next_result
File "calibre_plugins.find_duplicates.duplicates", line 238, in show_next_result
File "calibre_plugins.find_duplicates.duplicates", line 277, in _cleanup_deleted_books
KeyError: 679

[kiwidude]

Hi Tom,

Thanks for the further info, but I still have been unable to replicate this. Are you able to do it every time? Does it still happen with the latest zip above on post #17?

If so, can you recreate the scenario and give me an exact sequence of steps in detail of every action you are taking? For instance, I create in my test library three books that will be flagged as duplicates, along with another group to also contain duplicates. I do the find duplicates, it reports the 2 groups, and goes to the first one. If it is the group with 3 books in it, I select a book to be the target, and then select the 2nd and third books and do a merge. Then I do the Ctrl+\.

Does that match your steps?

[chaley]

Originally Posted by kiwidude

Part of my hesitation was thinking about ways users could mark not duplicates. For instance if I let it be a free-for-all, then in it could be a lot of data. Say for instance I allowed a user to mark any books they liked together as not duplicates. So I have a library of 1,000 books as a starting point, and I am confident I have no duplicates. If I selected the whole lot and said mark as not duplicates, then I would ensure that any duplicates that came up in future were not books I had considered. I could skip through the book groups with a very "loose" duplicate matching algorithm, then rather than resolving each group I could just resolve the ones I were dups, then select my whole library and mark it as not containing duplicates again.

Good point. Your idea of permitting non_dup marking only of books that are found to be duplicates is a good one.

At this point you have gone quite far beyond theory. I need to play with the plugin.

[chaley]

Tried it. Good stuff!

I have several intentional duplicates -- books in French and English where the title is not translated. I await the non-dup marking with bated breath.

A suggestion: an option to see the union of all groups, sorted by title. Use case: when I run similar title/ignore author, I get 6 groups where each group contains two books. I could tell at a glance if these are correct by seeing them all, sorted by title. A variant of this would be to show me all the books and have 'change group' use highlighting the books in a particular group.

Behavior oddity: The first time I press next-group after having run the finder, nothing happens. I need to press it again. I suspect that this happens because the first group is already displayed, and the first press displays it again.

Minor suggestion: I would prefer to see the tests listed with title first. Rationale: all of the tests do something with the title, but some ignore the author. In my mind, that makes title 'superior' to author, so I want to see it first.

Minor suggestion: add a 'next group' action so I can add it to my toolbar/menu bar.

[kiwidude]

Originally Posted by chaley

Tried it. Good stuff!

I have several intentional duplicates -- books in French and English where the title is not translated. I await the non-dup marking with bated breath.

Yeah I have gotten a bit sidetracked with fire-fighting on some other plugins - it is "almost" there but just need to implement the "mark group as not duplicate" function once I realised that doing it based on individual pairs of rows would be problematic.

A suggestion: an option to see the union of all groups, sorted by title. Use case: when I run similar title/ignore author, I get 6 groups where each group contains two books. I could tell at a glance if these are correct by seeing them all, sorted by title.

Seeing all the books that were flagged as duplicates and then sorting them by title is certainly easily doable. Of course in this "mode" you wouldn't be able to see what was matched with what from a group perspective, but if the titles start with the same characters (or there are not many) you will work it out. You could always drop in and out of "Next group" mode at any point.

A variant of this would be to show me all the books and have 'change group' use highlighting the books in a particular group.

Is that not the same effect as just turning on highlighting that you can do now? I just tried it, the green highlights select the different rows. The problem is that without the custom column I have no way of sorting the results to appear together...

Behavior oddity: The first time I press next-group after having run the finder, nothing happens. I need to press it again. I suspect that this happens because the first group is already displayed, and the first press displays it again.

Yeah that's one of several quirks that are in that version that are fixed in the next one

Minor suggestion: I would prefer to see the tests listed with title first. Rationale: all of the tests do something with the title, but some ignore the author. In my mind, that makes title 'superior' to author, so I want to see it first.

Good suggestion, will do it.

Minor suggestion: add a 'next group' action so I can add it to my toolbar/menu bar.

Already exists. Add the plugin to your toolbar now. When you click on the button the first time, it detects you have no duplicate groups so pops up with the Find Duplicates dialog. Once you have duplicate groups, clicking it again is the same as next group (and ctrl+click on it for the previous group, though perhaps that may be more useful to have as "Show all duplicate books").

I used to have a qeustion dialog in between, so that when you click the buton the first time it said something like "You have no duplicate groups, do you want to run a search now" or whatever, but that was painful since the obvious answer was "yes". When you resolve the last of your duplicate groups you will already get a dialog telling you there are no more, so the user is well warned that the next time they click on that button it will popup with the find dialog. The only concern I have that is proven by you is the discoverability that clicking on the button is the same as next result once you have clicked find.

[chaley]

Originally Posted by kiwidude

Is that not the same effect as just turning on highlighting that you can do now? I just tried it, the green highlights select the different rows. The problem is that without the custom column I have no way of sorting the results to appear together...

Not quite. The scenario was:
1) show books in all groups
2) have next-group highlight the books in a group.
This lets me see the global view, shows me what is in a group, lets me cycle between groups without the screen repainting, and also lets me find (at some level) books that are in more than one group (it gets highlighted more than once). I do recognize that this mode is not particularly useful if one has more than a screenful of duplicates.

Already exists. Add the plugin to your toolbar now. When you click on the button the first time, it detects you have no duplicate groups so pops up with the Find Duplicates dialog. Once you have duplicate groups, clicking it again is the same as next group (and ctrl+click on it for the previous group, though perhaps that may be more useful to have as "Show all duplicate books").

Ahhh...

I put the plugin on the menu bar instead of the tool bar to save real estate. The menu bar doesn't have default-click/pull-down behavior. When I put it on the toolbar, it behaves as you describe. Not sure I would have found it, though.

[kiwidude]

Originally Posted by chaley

Not quite. The scenario was:
1) show books in all groups
2) have next-group highlight the books in a group.
This lets me see the global view, shows me what is in a group, lets me cycle between groups without the screen repainting, and also lets me find (at some level) books that are in more than one group (it gets highlighted more than once). I do recognize that this mode is not particularly useful if one has more than a screenful of duplicates.

Got you now. Could you do it with this: a search restriction of books that are in duplicate groups, with highlighting turned on to display the current group?

I think I could mark all books in duplicate groups as 'duplicate' in the marked ids dictionary, in addition to just the group under consideration as 'group_x_of_y'. So it could apply a search restriction of 'marked:duplicate', turn on highlighting (so all duplicate books displayed), and then do the normal 'marked:group_x_of_y' walking through groups?

I don't know how easy it is to do without looking into it (for instance can you set a restriction that isn't a saved search) and you have to consider restoring the user's state at the end by removing the restriction etc. Though I guess the user resolving the last group, or marking all groups as not duplicates would be one trigger. Toggling this menu item would be another.

Of course as you say for lots of duplicates that wouldn't be very nice, so this whole mode of behaviour would be something that is just an alternate view they can use. I'll ponder it for now (assuming we are thinking of the same thing)...

[Starson17]

Originally Posted by chaley

When I put it on the toolbar, it behaves as you describe. Not sure I would have found it, though.

Just to add to this usability comment, I doubt I would have found it either. It's very unusual for me to put anything on the toolbar.