[GUI Plugin] Find Duplicates

[chaley]

Originally Posted by capink

It still happens every now and then randomly. The problem is I always notice it after the fact. Whenever I try reproduce that behavior, it acts normally.

I made it happen with the following steps:

1. Choose "Find book duplicates"
2. Choose the option "Show all groups at once with highlighting"
3. Quit calibre without clearing the search bar
4. Restart calibre

The search remains on "Highlight".

You can probably fix this by implementing the "shutting_down" method and restoring the preference.

[capink]

Originally Posted by chaley

I made it happen with the following steps:

1. Choose "Find book duplicates"
2. Choose the option "Show all groups at once with highlighting"
3. Quit calibre without clearing the search bar
4. Restart calibre

The search remains on "Highlight".

You can probably fix this by implementing the "shutting_down" method and restoring the preference.

I thought I had all scenarios of how to exit the duplicates results covered, but I never thought of this simple one

I fixed this as you suggested by implementing shutting_down method

I also tested by switching libraries without clearing duplicates, but that does not lead to the same problem. So, hopefully this solves it for good.

[capink]

Version 1.7.6
Update: Advanced Mode: Add support for saving and restoring match rules.
Update: Advanced Mode: Add option to control sorting of books in duplicate groups. Requested by @tanjamuse
Update: Advanced Mode: Use calibre's template dialog for adding and editing templates.
Update: Advanced Mode: Mark first and last books in each duplicate group: "first_duplicate", "last_duplicate"
Update: Mark records with deleted formats in binary search as "deleted_binary_duplicate"
Update: Option to export duplicate groups to json file. For advanced mode, the sorting of books is retained in the json file.
Fix: Mark exemptions only when showing them and remove the marks afterwards.
Fix: Restore state if the user exits calibre with the duplicates restriction still on. Thanks to @chaley
Fix: Remember last sort in library view and restore it back.


With this new version, instead of repeating steps every time you invoke the advanced mode, you can save the match rules and restore in one click. You can also export these match rules for backup or for importing in other calibre libraries.

The new sort feature (a suggestion by @tanjamuse) allows you to control the sorting of books in duplicate groups. It is also complemented with marking the first book in the group as "first_duplicate" and the last book in the group as "last_duplicate". So if for example, if you want to show all the books in the group(s) except the last book(s) you can type the following in the search bar:

not marked:last_duplicate

N.B whether the above search applies to all groups or to the current group depends on the current search mode (either highlight or restrict results). To make it easier to switch between these two modes, follow the steps in this post.

WARNING: Sometimes you might have one book belonging to two different duplicate groups. Because of the way this plugin marks duplicates, one of these two groups which have the same book might have its sort (and consequently, first_duplicate and last_duplicate marks) messed up. So, it is better to first deal manually with these groups. These groups can be shown by the following search:

marked:entangled_group

And if you want to show the books that belong to more than one group you can use this search:

marked:entangled_book

Another thing you can do with the sort is to use templates instead of columns. Say for example you want to sort the duplicate groups by the number of formats each record has, you can use this template in the sort dialog to achieve this:

{formats:count(,)}

If the above is not enough and you want more control of how to handle duplicate books, there is an option to export duplicate ids to json file, that can be later fed to calibre-debug (or a shell) script to handle them according to the needs of individual users. For the advanced mode, the exported json file retains the sort order specified by the user in the sort dialog.


Finally, for people who use the binary compare with automatic removal of duplicates, they can find the records with automatically deleted formats by typing this search:

marked:deleted_binary_duplicate

[compurandom]

Wow! That's a lot of really useful features added.

Let me request one more... Frequently when I find duplicate books I don't want to delete them, I want to merge them. So, maybe a merge all option, or a mode where a keystroke can merge the duplicates and then move on to the next group?

Also, I'm thinking the entangled groups stuff is important enough that maybe the searches should be stuck in a menu entry, in a way to remind people that they're there.

[capink]

Originally Posted by compurandom

Let me request one more... Frequently when I find duplicate books I don't want to delete them, I want to merge them. So, maybe a merge all option, or a mode where a keystroke can merge the duplicates and then move on to the next group?

I am against adding any auto-delete or auto-merge functionality to this plugin, because it gives new users extra rope to hang themselves with, and any damage done is going to be irreversible. You and others might benefit from such feature, but it can do inadvertent damage to others. And this is why I added the export feature for.

Originally Posted by compurandom

Also, I'm thinking the entangled groups stuff is important enough that maybe the searches should be stuck in a menu entry, in a way to remind people that they're there.

I don't think this is very common situation for most users. I am still weighing how to deal with it and whether it is worth any extra effort, given how uncommon I think it will turn out of most users. Maybe feedback here from others on how frequently they run into this situation can change this.

[compurandom]

Originally Posted by capink

I am against adding any auto-delete or auto-merge functionality to this plugin, because it gives new users extra rope to hang themselves with

I certainly can see that, not sure I'd ever use an automerge.
But like I said, an easy way to merge the current group and move on would be of great value.

(...entangled group searches...)

I don't think this is very common situation for most users.

I suggest burying it in a submenu. That way, it's there for people who wander through the menus to find.

[BetterRed]

Mark the books that need to merged and do it with the metadata edit merge tool when you're done with FD.

BR

[capink]

Here is a version with a couple more fixes for restoring previous gui state.

[BetterRed]

@capink

Can I suggest the updated plugin be released as Version 1.8 or even 2.0.

BR

[capink]

Originally Posted by BetterRed

@capink

Can I suggest the updated plugin be released as Version 1.8 or even 2.0.

BR

Yes. I was thinking about something like that. I don't really know when to bump the minor version, and when to bump the major version? Is there any guidelines I can refer to?

[jhowell]

Originally Posted by capink

Yes. I was thinking about something like that. I don't really know when to bump the minor version, and when to bump the major version? Is there any guidelines I can refer to?

It's arbitrary. Use your own judgement.

[theducks]

Originally Posted by capink

Yes. I was thinking about something like that. I don't really know when to bump the minor version, and when to bump the major version? Is there any guidelines I can refer to?

Bumping Major usually is because:
Not backwards (with the older main app) Compatibility changes

Significant UI changes, where the old guide is just really wrong or would be confusing.

How about,the new guard took over and want to create a definitive Start point

[capink]

Originally Posted by theducks

Bumping Major usually is because:
Not backwards (with the older main app) Compatibility changes

Significant UI changes, where the old guide is just really wrong or would be confusing.

How about,the new guard took over and want to create a definitive Start point

Originally Posted by jhowell

It's arbitrary. Use your own judgement.

Thanks for the input. The changes are incremental and backward compatible. The main code written by kiwidude is almost untouched (except for few fixes), and still drives the functionality of the plugin including the new additions.

If all goes well, I will release as 1.8.

[dunhill]

Originally Posted by capink

Thanks for the input. The changes are incremental and backward compatible. The main code written by kiwidude is almost untouched (except for few fixes), and still drives the functionality of the plugin including the new additions.

If all goes well, I will release as 1.8.

Check your private @capink I send you what you requested

[capink]

Released version 1.8.0. The same as the last beta with one fix for missing import.