Duplicate detection plugin

[kiwidude]

Re the toolbar button - well you have both keyboard shorctus and menu options as additional way to move to the next group, so hopefully that should be enough. Maybe one day Kovid will offer a way of creating plugin widgets to the main GUI (a few people have raised things in the past like wanting a status panel etc). A little panel able to be displayed on the book details pane or something which could have next/previous group buttons, a count of duplicates etc would offer another way to navigate all this stuff. Not likely to happen any time soon if at all though.

[chaley]

Originally Posted by kiwidude

Got you now. Could you do it with this: a search restriction of books that are in duplicate groups, with highlighting turned on to display the current group?

I think I could mark all books in duplicate groups as 'duplicate' in the marked ids dictionary, in addition to just the group under consideration as 'group_x_of_y'. So it could apply a search restriction of 'marked:duplicate', turn on highlighting (so all duplicate books displayed), and then do the normal 'marked:group_x_of_y' walking through groups?

This would work, as long as next_group does a text search instead of true.

To do this the best way, we would:
- need to add the ability for a plugin (or any other code) to apply an arbitrary search as a search restriction. This can be done rather easily by adding a 'current search' option to the restriction box. Selecting 'current search' will copy what is in the search box and apply it as a restriction. I will look at this, as it would be useful independent of duplicate detection.
- have the plugin apply the search 'marked:duplicates' as a restriction.
- make next and previous do as you described.

I don't know how easy it is to do without looking into it (for instance can you set a restriction that isn't a saved search) and you have to consider restoring the user's state at the end by removing the restriction etc. Though I guess the user resolving the last group, or marking all groups as not duplicates would be one trigger. Toggling this menu item would be another.

Manually removing the "Arbitrary search" restriction would also work.

[kiwidude]

Here is another version for you incorporating a number of the changes and additions mentioned above. Add the duplicate exemption support raised the complexity several times but I "think" it is getting there.

If you have downloaded a previous version, please delete the "Find Duplicates.json" config file from your plugins folder. I couldn't be bothered with backwards compatibility for a beta.

Stuff I know differs in this release:

• Fixed various bugs to do with navigating and initial group selection
• Fix the "3 in a group" bug that Tom found.
• Rename the algorithms as per Charles suggestion.
• Implement the "Mark group as exempt", "Mark all groups as exempt", "Show all duplicate exemptions" and "Remove selected duplicate exemptions" features
• Added a "Show all duplicates" menu item. Note this does not as yet attempt to do any of the restriction/highlighting stuff above, will wait for Charles to add stuff to Calibre
• Significantly changed the way books are marked.

The marking dictionary stuff is quite different to my first version. What I do now is store a whole bunch of markings at the same time, and recalculate them all every time you do an action. So for instance you could at any point do "marked:duplicates" to see all the books reported as duplicates. In fact Charles you can do this as a saved search, manually turn on highlighting and then use the next/previous group to do exactly as we discussed above a little more "manually".

At the same time it is also possible to view individual groups using "marked:duplicate_group_000x". Choosing next/previous result will recalculate the marked books, but if you felt so inclined you could just overtype the number and hit Go to see a specific group etc.

The third simultaneous marking is "marked:not_duplicate". This is applied to all books that are part of a duplicate exemption.

In theory we could use the above for sorting when showing all duplicate books, as I store composite values with "duplicate_group_000x" always at the beginning of the "marked" field value. I haven't bothered to apply any sorting at this point but the ability is there.

Removing duplicate exemptions is as I described in posts above - for each book you have seleted, it removes all associated exemption pairings. So if you initially mark a group that has (1,2,3) that will create exemptions of (1,2),(1,3),(2,3). If you were to right click on book 3 and choose remove exemption, then refresh your search it will just keep (1,2).

Clearly I haven't thrashed it and could well have broken new stuff so as always feedback appreciated. If you don't like the menus, names, ordering, want more/less confirmations or whatever please offer suggestions along with anything else that comes to mind. I suspect we are closer to a 1.0 release than 0 which the number indicates but it may just be wishful thinking.

[chaley]

Originally Posted by kiwidude

Stuff I know differs in this release:

• Added a "Show all duplicates" menu item. Note this does not as yet attempt to do any of the restriction/highlighting stuff above, will wait for Charles to add stuff to Calibre

Are talking about arbitrary searches as search restrictions? If so, then the code is now in trunk. The method/changes are in gui.search_restriction_mixin. The API:

apply_text_search_restriction(some_search)

where 'some search' is any search expression. You can also do this from the gui by going to the search restriction box and selecting "*Current search" (the first item).

In theory we could use the above for sorting when showing all duplicate books, as I store composite values with "duplicate_group_000x" always at the beginning of the "marked" field value. I haven't bothered to apply any sorting at this point but the ability is there.

I have never tried to sort by the marked field, but it should work at the db level. However, there is no provision at the GUI level to do it. It would be easy to add, if there is a use case for it.

Will try the new version soon...

[kiwidude]

Originally Posted by chaley

Are talking about arbitrary searches as search restrictions? If so, then the code is now in trunk.

Yes I was talking about that feature, thanks for adding. I will wait for your feedback and thoughts on that latest beta before I make use of it though.

I have never tried to sort by the marked field, but it should work at the db level. However, there is no provision at the GUI level to do it. It would be easy to add, if there is a use case for it.

Yes I did say "in theory" . What I had in mind was that if there is a view like "Show all duplicates", that perhaps the default sort applied should be by the "marked" column. So when they first see the results, all of the groups would be kept together on screen - the user wouldn't be able to see where one group starts and the next ends of course, but if this is being used in conjunction with highlighting then the highlighting would sequentially move down the screen as they move to the next group. Plus in the case of "not duplicates" forcing a group to be partitioned, the group numbers would sequentially be one after another, so at least in this view without the user even trying to sort by title they would "usually" see the next group for the related book next to it.

I say "usually" because it depends on how many other groups that book is in as to whether it will sort next to it. In a happy case, the marked column would be like this:

Say you have Group 5 of (11,12) and Group 6 of (11,13), caused by a duplicate exclusion in place between (12,13).
book 11 is marked 'duplicate_group_0005,duplicates'
book 12 is marked 'duplicate_group_0005,duplicate_group_0006,not_dup licate,duplicates'
book 13 is marked 'duplicate_group_0006,not_duplicate,duplicates'

So sorting by marked text will in this case put them together on screen.

If however book 13 also happens to be part of a group 1, then book 13 is marked: 'duplicate_group_0001,duplicate_group_0006,not_dup licate,duplicates'

So initially it will sort at the top in group 1, not next to the members of its second related group of group 6. However if the user processes their duplicates sequentially and resolves group 1 first, then the marked text for book 13 would be updated to remove that duplicate_group_0001 and it would appear as per my initial example above in the right place. So basically a book that was in multiple groups (assuming it is the "winner" of any earlier merges) will progressively move down the screen to be displayed with its next associated merge group.

It's just a theoretical approach to displaying all duplicate groups at once . And as marked is not a visible column, the user could "break" the sorting at any point by clicking on a column heading, though I would keep re-applying it every time they move next/previous.

Will try the new version soon...

Excellent, as always look forward to your feedback.

[chaley]

I have submitted code to sort on named fields. Assuming it is accepted, the API is:

library_view.sort_by_named_field(field, order, reset=True)

If the field (a lookup name) is a visible column, then that display is sorted by that column and the sort indicators are set. If the field is not visible, then the display is sorted but no sort indicators are set.

Order is a boolean. True == ascending.

If reset is true, the screen is repainted. Note that the screen is always repainted if the field is a visible column.

[kiwidude]

Hi Charles - two questions: (1) what if you want to apply multiple sort columns, and (2) if I sort by one field, is it going to be on top of whatever sorting was already applied (i.e. they become a secondary sort)?

Kind of the same question I know

[chaley]

Originally Posted by kiwidude

Hi Charles - two questions: (1) what if you want to apply multiple sort columns, and (2) if I sort by one field, is it going to be on top of whatever sorting was already applied (i.e. they become a secondary sort)?

The sorts behave like clicking on column headings (or at least should). As such:
- to sort on multiple columns, sort on them one at a time in reverse order.
- the sort is stable, so any existing order becomes secondary.

And you didn't ask, but
- field sorts are entered into sort history. This means that sorts of non-visible columns are preserved across db cache refreshes and db searches.

[chaley]

The sort changes are now in trunk.

[chaley]

OK, finally got time to try it.

I really like using the search restriction and the marking. Setting up the search restriction by hand is easy, although having it done for me would be nice.

One nit: the duplicate groups are random (it seems) in terms of title order. They jump all over the place. Next group bounces from Collected Essays to Xenocide to Collected Poems. It would be nice if the groups were ordered in ascending title sort order, where the lowest title in the group controls the order. Sorting by duplicate group would help because the flow will be 'down', but then the titles will be sorted oddly if a book is in more than one group. I confess that I am not sure what is best here. (You know things are going well when I start mentioning this kind of thing. )

In terms of behavior: clearly switching to 'highlighting mode' should set the search restriction and the highlight_only flag. The issue is the one you mentioned -- when does the highlight_only flag get reset to what it was? I am tempted to put a copy of that checkbox back onto the main gui so that the user can easily see it change and can easily reset it. However, real estate is precious, so I would propose a very terse label, something like "H/O" next to the checkbox with a tooltip that tells the user to see the configuration dialog for more information.

Showing duplicate groups is close to useless if the highlighting flag is set unless the search restriction is set to not_duplicate. Is there a way (will there be a way) to see the books in their non-dup pairs? I would like to select a book and have that books non-dups highlighted. Otherwise I have no idea what the pairs are, at least as I can tell.

I think this plugin is very nice. Once it is stable, I would argue that it be merged into calibre itself.

[kiwidude]

Hey Charles, thanks for the feedback again.

Originally Posted by chaley

I really like using the search restriction and the marking.

I thought you might like the marking. Awesome foresight when you added that feature to Calibre to suggest putting a dictionary in place now wasn't it?

Setting up the search restriction by hand is easy, although having it done for me would be nice.

I've just plumbed in the show all groups to do search restrictions and highlighting. It is awesome! I think there could be a few ways to go with this.

1. Drop the "Show all duplicates" menu option and make this the default behaviour, or...
2. To (just) have a persisted option on the search dialog, so the user can choose whether to display all the results as a group or to review individually, or...
3. To have it both on the dialog, and have the "Show all duplicates" menu option which act as a toggle to turn the behaviour on and off on the fly.

The menu option could come in handy because of the issues with showing duplicate exemptions discussed below, although we could just treat "next result" as a similar trigger to turn the GUI "back into" the show all duplicates mode. Thoughts?

One nit: the duplicate groups are random (it seems) in terms of title order...

That is a tough one if part of the issue is stuff like "A" and "The" prefixes being part of the reason for a title being a duplicate in the first place . There is a step in the process before I allocate the group numbers where I can sort them however I like - so what I have done just now is to sort by their fuzzy key. You would still get some "quirks", but as a general flow there would be some consistency at least.

When displayed, what I am doing (now) is to first sort by title, and then by marked group (which is why I asked about multiple sorts above). So within each group you would have some consistency, plus the overall pattern to the groups.

In terms of behavior: clearly switching to 'highlighting mode' should set the search restriction and the highlight_only flag. The issue is the one you mentioned -- when does the highlight_only flag get reset to what it was? I am tempted to put a copy of that checkbox back onto the main gui so that the user can easily see it change and can easily reset it. However, real estate is precious, so I would propose a very terse label, something like "H/O" next to the checkbox with a tooltip that tells the user to see the configuration dialog for more information.

I think it needs to be on the GUI front screen where it was originally. The other options on that search preferences are all "set once" type things whereas highlighting is something you are more likely to use on/off particularly if we start (ab)using it for functions like this plugin. The behaviour will be that whenever the last duplicate group is resolved/marked not duplicate the gui will get "reset" in terms of restriction, highlighting mode and cleared search. I could also add a "clear duplicate results" menu item which would do the same thing, for if a user aborts resolving duplicates before the end. I would presume that for highlighting we should remember whatever setting the user had before they did a duplicates search - do we do the same for search restriction too?

Showing duplicate groups is close to useless if the highlighting flag is set unless the search restriction is set to not_duplicate. Is there a way (will there be a way) to see the books in their non-dup pairs? I would like to select a book and have that books non-dups highlighted. Otherwise I have no idea what the pairs are, at least as I can tell.

Totally agree its not there yet particularly if you have highlighting turned on and a duplicates restriction, then it really doesn't work . It is a mutually exclusive behaviour with displaying the current duplicate results, because of course you can/will have books appearing in the exemption list that have not been marked as potential duplicates with your current search. So the search restriction definitely has to change. Then you have to cater for getting back into "show results" mode as well.

Displaying the pairs is the tricky one. We are back to the old chestnut of trying to indicate to the user which is the "root". For instance we could just again use highlighting mode, set a restriction of "marked:not_duplicates", and then give you a menu option of "Show exemptions for this book" which then highlights all the books it is paired with. However as soon as they move their selection they will of course lose who the parent was. Buried two layers of menu down isn't exactly the easiest gui access too. And of course there is also the question of whether you would want this ability while looking at your duplicate groups, rather than only when looking at all your exemptions... I don't have any answers at this point, unless we go back to the idea of a separate dialog again for showing the exemptions?

I think this plugin is very nice. Once it is stable, I would argue that it be merged into calibre itself.

Great to hear dude. As for merging it into Calibre, I guess if once we get to the bottom of the duplicate exemptions and highlighting usability stuff that Kovid is happy with it we could do that. Though I am always tempted like with Plugin Updater to let the faithful who happily download plugins to give it a whirl for a week or two to get some wider feedback first but it is entirely up to you guys.

[kovidgoyal]

I'm perfectly fine with having this functionality builtin. I'll leave it up to you and Charles, when you feel it is ready for inclusion, just send me the patch for review.

[kiwidude]

@Charles potentially for this one (and yes I should raise a ticket but I'll ask about it first). Could the library view select_rows logic be tweaked (I know Qt has various options and it is something that bugs me from time to time).

Right now when you have a lot of results, and the screen gets refreshed (like after editing metadata or whatever), all it does when restoring selections is make sure the first row in your selection is visible. And if that row was near the bottom of the screen, the selected row just becomes the last row visible. Which is a pain because you cannot see the next row, and in the case of this duplicates plugin in show all mode trying to highlight the next group means you have to keep on scrolling to see it's members

I would rather that where possible the selection be put into the middle of the screen, so you have visible rows either side in your workflow to look at next or previous rows. Or at least with a decent amount of row padding like 10 rows or something.

Shall I raise a ticket or will you just fix it in your usual efficient fashion?

P.S. I see I just spent my 1000th post on wishing Kovid well. Only appropriate I guess.

[Starson17]

Originally Posted by kiwidude

I've just plumbed in the show all groups to do search restrictions and highlighting. It is awesome!

Agreed. It is really great (at least it is after I realized I had to delete the json file from the earlier beta )

I don't have any answers at this point, unless we go back to the idea of a separate dialog again for showing the exemptions?

I'd provide feedback, but I need to play with it more, with real data (not just my test data).

to get some wider feedback first but it is entirely up to you guys.

It should definitely go into the trunk, but I'd suggest letting more people play with it. Just in case you want to make more major changes. Plus, Kovid's a bit under the weather, so there's no rush.

[kiwidude]

@Starson - yeah I did warn about the config file thing in the post it is attached to, but can't blame you for not reading all of my ramblings

@Charles - I don't suppose there is a convenient way to identify what the current sort column is? I discovered that sorting every time you move to the next duplicate group is quite slow. So what would be better I think (since the topmost sort column is that of "marked") is to check whether that is still the current top sort column, and only apply my sorts again of title then marked if it is not. The user can't sort on "marked" since it is not visible, so this should work as an optimisation.

I know from my ViewManager plugin that I can get down and dirty and roll around in get_state() with sort history etc, but I thought I would ask you first before I did that again to see if you had a better suggestion