[GUI Plugin] Find Duplicates

[Richard1970]

Hello, I´m new to Calibre and this plugin.
I´ve run the plugin to binary compare my library and I´ve got returned a list of 30.000 or more hits. One way or another I´ve marked them without deleting them and now I want to run this compare again and it returns zero matches. So how do I unmark my results?

When this problems gets fixed I want to run the compare again and I only want to delete ofcourse 1 match of the double books. But I get after the compare the full list with all the matches. I want to select with Ctrl-A only one of the doubles. Is this possible and how is it done?

Thank you very much.

[BetterRed]

On Windows, selecting multiple lines in the calibre book list is the same as the file manager - I'd be surprised if that wasn't true of MacOS and Linux.

Ctrl+M will toggle Book List Marks, see Keyboard shortcuts

You can add the Mark Books toolset to toolbars and menus in Preferences.

BR

[Richard1970]

Thanks for your reply.

But it is not exactly what I´m struggling with.

When I now run "Find Book Duplicates Similar Title and Author Matching", I get a long list with entries that are all double (over 10.000 books are double at this moment because I´ve imported some large databases and the names of the books were apparently not exactly the same for Calibre to merge them).

All results from the run are in couples but I want to delete only one of the two double entries out of the database. So I have a list of 20.000 books and I want to select of these only 10.000 books to delete.
Is there a possibility to select only 1 entry of all doubles and delete them all in one click?

[capink]

When using the binary compare, you can check the option "When doing binary compare, automatically delete duplicate formats".

Note that this will only delete the book files (epub, ... etc) but will keep the entries in calibre (because it might contain metadata that some people would want to keep).

So, for entries that have only one book format, when calibre deletes the duplicate from them, they will be empty (i.e have no formats, just metadata). You can filter these using this calibre search:

formats:false

Then, you can delete them, but only if you are sure you don't need the metadata form these empty entries. It is always a good idea to backup your library first just in case, especially when trying something for the first time.

For entries that have more than one format, I'm afraid you will have to delete the entries one by one.

[JSWolf]

Originally Posted by Richard1970

Thanks for your reply.

But it is not exactly what I´m struggling with.

When I now run "Find Book Duplicates Similar Title and Author Matching", I get a long list with entries that are all double (over 10.000 books are double at this moment because I´ve imported some large databases and the names of the books were apparently not exactly the same for Calibre to merge them).

All results from the run are in couples but I want to delete only one of the two double entries out of the database. So I have a list of 20.000 books and I want to select of these only 10.000 books to delete.
Is there a possibility to select only 1 entry of all doubles and delete them all in one click?

But can you be sure that all of those matches are the same eBook? Why not do a binary compare and see how many are the exact same eBook. Then do a smilar compare. But don't just delete willy nilly if you can.

[Richard1970]

Thanks Capin and JSWolf.

That formats:false is something I have to try.

[odinokij]

Originally Posted by odinokij

I have released a new update of the duplicates++ plugin (based on the original Find Duplicates plugin v.1.6.3 by Grant Drake). It includes:

- Finding duplicates by series and authors
- New algorithms fzzEx (extended fuzzy) for titles/series and authors
- Accelerated "Mark current group as exempt" (uncheck option "autosave new exemptions" and use menu "Save new exemptions". The new exemptions are also automatically saved if the end of the duplicated list is reached)
- Several bugs are fixed


The link is the same:
https://1drv.ms/u/s!AobKqg_eHGvlicteuckLgFlxmqE2zw


I hope it may be as useful for you as it's for me

Odinokij.

Hi there,
I've updated the version of the plugin Duplicates++ in order to make it compatible with Python 3 so it works on calibre v5.
The new version has also been updated from its original "Find duplicates" plugin v1.7.2 to its new version 2.0.2
The link to the new version of Duplicates++ plugin is:

https://1drv.ms/u/s!AobKqg_eHGvlicte...jr7lg?e=bTMKRD


Kind regards,
Odinokij

[JSWolf]

Originally Posted by odinokij

Hi there,
I've updated the version of the plugin Duplicates++ in order to make it compatible with Python 3 so it works on calibre v5.
The new version has also been updated from its original "Find duplicates" plugin v1.7.2 to its new version 2.0.2
The link to the new version of Duplicates++ plugin is:

https://1drv.ms/u/s!AobKqg_eHGvlicte...jr7lg?e=bTMKRD


Kind regards,
Odinokij

If you are not going to PM kiwidude to see if it's OK to take over, I suggest you create a new thread with your plugin. If it's as god as you say, then it should be available from Calibre., not some cloud storage.

[capink]

I've been trying to work on the issue of this plugin missing up the user's search preference as described in this thread. This happened to me in the past, but I'm not able to replicate it now. Can anyone give me steps on how to replicate this behavior?

[BetterRed]

Originally Posted by capink

I've been trying to work on the issue of this plugin missing up the user's search preference as described in this thread. This happened to me in the past, but I'm not able to replicate it now. Can anyone give me steps on how to replicate this behavior?

I put the button back on the search bar, its easier to understand now because as well as the Blues Bar (red striped) it it also shows the word "Highlight" or "Filter".

I think it's these options:



IIRC the problem is, if the PI stashes the current value when should it restore the stashed value? The PI has no natural "I'm done with using this" state.

And, there's another 'bug': if books are excluded in Find Duplicates their id's are stashed in the database. If there's anything stashed when calibre starts those books are Marked and a gold pin shows on the left -- which is confusing. The plugin needs to be reworked to avoid use of the same mechanism as the Mark books tool.

The code of the now deprecated Temp Marker PI might assist in 'fixing' this. In essence the built-in Mark books tool replaced the Temp Marker PI.

BR

[capink]

Originally Posted by BetterRed

And, there's another 'bug': if books are excluded in Find Duplicates their id's are stashed in the database. If there's anything stashed when calibre starts those books are Marked and a gold pin shows on the left -- which is confusing. The plugin needs to be reworked to avoid use of the same mechanism as the Mark books tool.
BR

I have already fixed that issue in a version that I will release soon. The plugin used to mark all exemptions at startup or library change. Now, it only marks them once the user chooses the option to display exemptions, and remove the marks afterwards.

Originally Posted by BetterRed

IIRC the problem is, if the PI stashes the current value when should it restore the stashed value? The PI has no natural "I'm done with using this" state.
BR

I still cannot reproduce this behavior using either option. I tried both and the plugin always reverts back to my original search preference. I have a general idea of where to look in the code, but without being able to replicate the problem it is difficult to solve. I need some very specific steps like:

1. What is your original search preference before using the plugin (is it highlight books or restrict the book list to results)
2. When you use the plugin, which option to tick (all group at once vs. one group at a time)
3. How do you exit the duplicate search. There are many ways:
   • By pressing clear duplicate result from the plugin drop down menu.
   • By pressing the restriction immediately to the left of the search bar.
   • By pressing the escape key.
   • By resolving all duplicates through merge, delete or exemption
4. Any other steps necessary to replicate this behavior

ُ
Edit: I tried combinations of all the above with no avail.

[theducks]

BR the PI run's, then Marks the results, then it terminates (at least for one of its modes), to the user.
so there can be no I am done (reviewing and deleting) if that mode is allowed (of coarse, that is the one I use )

[BetterRed]

Originally Posted by theducks

BR the PI run's, then Marks the results, then it terminates (at least for one of its modes), to the user.
so there can be no I am done (reviewing and deleting) if that mode is allowed (of coarse, that is the one I use )

Once I put the button back on the search bar the issue hasn't bothered me.

I think we're saying the same thing

IIRC:-- If Preferences->Searching->Highlight search results… setting is unchecked and you set the FD option to 'highlight', FD sets the Preferences->Searching->Highlight search results… setting to checked and leaves it in that state. And there seems to be no natural place in the PI usage to restore the setting.

If the PI stashed the previous value some place where the regular search could access it, then if the stashed value wasn't empty it could restore the setting and flush the stashed value.

EDIT: See ↓ ↓ ↓

BR

[BetterRed]

Originally Posted by capink

I have already fixed that issue in a version that I will release soon. The plugin used to mark all exemptions at startup or library change. Now, it only marks them once the user chooses the option to display exemptions, and remove the marks afterwards.

Great!

Originally Posted by capink

I still cannot reproduce this behavior using either option. I tried both and the plugin always reverts back to my original search preference. I have a general idea of where to look in the code, but without being able to replicate the problem it is difficult to solve. I need some very specific steps like:

1. What is your original search preference before using the plugin (is it highlight books or restrict the book list to results)
2. When you use the plugin, which option to tick (all group at once vs. one group at a time)
3. How do you exit the duplicate search. There are many ways:
   • By pressing clear duplicate result from the plugin drop down menu.
   • By pressing the restriction immediately to the left of the search bar.
   • By pressing the escape key.
   • By resolving all duplicates through merge, delete or exemption
4. Any other steps necessary to replicate this behavior

ُ
Edit: I tried combinations of all the above with no avail.

Hmmm - I don't use FD much these days - actually at all. My earlier comments were based on memory.

I just did some tests and like you I couldn't make it misbehave. Maybe someone else (like you ) already fixed it, or maybe Kovid has changed the base in some devious some way.

Maybe theducks could do some tests or someone else who uses the PI. It would be good to be able to unstick the stickies.

BR

[capink]

Originally Posted by BetterRed

Great!



Hmmm - I don't use FD much these days - actually at all. My earlier comments were based on memory.

I just did some tests and like you I couldn't make it misbehave. Maybe someone else (like you ) already fixed it, or maybe Kovid has changed the base in some devious some way.

Maybe theducks could do some tests or someone else who uses the PI. It would be good to be able to unstick the stickies.

BR

It still happens every now and then randomly. The problem is I always notice it after the fact. Whenever I try reproduce that behavior, it acts normally.