[GUI Plugin] Find Duplicates

[BetterRed]

Originally Posted by MidwestJen

Every time I open Calibre, my library opens with all of my duplicate exemptions marked. Is there a way for it to remember that without re-marking them each time? Obviously it just takes a few seconds to 'clear all marked books', but it's a tad annoying. It's also a bit baffling that I couldn't really find anyone else mentioning it (then again, I could be using a bad set of keywords to search on).

Is there a setting/preference I'm missing somewhere or is this normal and necessary behavior?

Thank you for any advice or help you can give on this. =)

@MidwestJen - I don't think it will be fixed until kiwidude becomes active again.

Meanwhile here's what I do, I have an Exemptions VL (search term marked:not_book_duplicate). I always start with that VL and do a clear all marked books. If I want to run Find Duplicates I restart calibre.

BR

[dustyp]

Is there any way to delete duplicates?
My library is massive and shows that it contains well over a thousand duplicates.
To show duplicates and then delete the unwanted books will take ages. To be able to highlight or mark all except the first copy would help enormously.

[theducks]

Originally Posted by dustyp

Is there any way to delete duplicates?
My library is massive and shows that it contains well over a thousand duplicates.
To show duplicates and then delete the unwanted books will take ages. To be able to highlight or mark all except the first copy would help enormously.

An 'Auto-incorrect' selection would be a disaster

When I find a dup, one of thos might be a corrected version (Baen bundles usually include a book way previously released. Sometimes, these fix issues that have been reported. To my way of thinking, that is the version to keep. OTOH My edits may be the 'best'/preferred version .

I review all dups. If it is not obvious, I V(iew) each version


Speaking of versions: I have some of those also: Original (first publication) and Re-released (SF classics) that have been edited.
I want to keep both.

[BetterRed]

Originally Posted by dustyp

Is there any way to delete duplicates?
My library is massive and shows that it contains well over a thousand duplicates.
To show duplicates and then delete the unwanted books will take ages. To be able to highlight or mark all except the first copy would help enormously.

@dustyp - you could try copying the books to an empty library, which I'll call XXXX.

First ensure this setting Preferences->Add Books->When using the "Copy to library" action, check for duplicates with the same title and author has a tick in its box.

Then Right Click->Copy to Library XXXX (delete after copy)

That should move the first duplicate to the XXXX library and leave the rest behind - up to you what you do with the latter.

BR

[dustyp]

Originally Posted by BetterRed

@dustyp - you could try copying the books to an empty library, which I'll call XXXX.

First ensure this setting Preferences->Add Books->When using the "Copy to library" action, check for duplicates with the same title and author has a tick in its box.

Then Right Click->Copy to Library XXXX (delete after copy)

That should move the first duplicate to the XXXX library and leave the rest behind - up to you what you do with the latter.

BR

Thank you.
That makes perfect sense and saves an awful amount of work

[lathom]

I'm seeing behavior with the Mark Books plugin that I don't understand. For some reason, everytime I start Calibre, or load my main library, it shows 538 books marked. I don't want any marked, so I clear all the marks. After making sure none are now marked, I restart and the same 538 books are marked again. How do I make it stop doing this?

Thanks
-Andy

[kovidgoyal]

Disable the find duplicates plugin

[BetterRed]

@lathom I don't think the Find Duplicates PI can be disabled, you could remove and add it back on an as required basis - you won't lose the its configuration.

Those 583 books are the books you have excluded from Find Duplicates processing**. If you no longer need the exemptions:

1. select Show book all duplicate exemptions from Find Duplicates drop down menu,
2. select the 583 books (ctrl/a),
3. select Remove selected exemptions from Find Duplicates drop down menu.

If you want to retain the exemptions, then you'll have to clear the marks when calibre starts.

BR

**: FD stores the list of books marked as exempt somewhere (I forget where exactly), when calibre starts FD reads that list and reinstates the marks.

Prior to Mark Books (the existence of FD predates the addition of Mark Books feature by a couple of years) this didn't matter, because there were no push pins to be seen. FD needs to be changed so that the exemptions list is reinstated when Find Duplicates->Show book all duplicate exemptions is selected, rather than when calibre starts.

Added : I just remembered the exemptions (groups of book numbers) are stored in the Preferences table of the library database. That table is used to store all library specific preferences/settings data. The table is backed up to metadata_db_prefs_backup.json, for use by Library Maintenance->Restore database, it can viewed in a text editor

[Rufal]

Originally Posted by dustyp

Is there any way to delete duplicates?
My library is massive and shows that it contains well over a thousand duplicates.
To show duplicates and then delete the unwanted books will take ages. To be able to highlight or mark all except the first copy would help enormously.

I had a few thousand duplicates that where simply author either John Doe or Doe, John.

Sometimes different format of ebook, sometimes not.

I removed them in one go by doing the following:
1) Create a new custom column, called it "fixing"
2) Search for all authors with a comma (author:"," )
3) Virtual library = current search
4) select all and bulk edit
5) Set "fixing" to something (I used "comma")
6) Open "Find Duplicates" and do a search (title = exact and author = similar)
7) sort by column "fixing"
8) Select all that have a value and delete.

Quickest method I could think of.

[odinokij]

Thanks for this plugin, it's great.

I'd like to suggest an improvement for those of us that aren't english speakers, about the duplicate detection of book titles due to lack of articles.

In english books the plugin detects duplicities in titles as "The whatever" and "Whatever" but it doesn't work for books in other languages.

In spanish, the articles and prepositions that should be had in account are "el", "la", "los", "las", "un", "una", "unos", "unas" and "de".

In french the articles and prepositions that should be had in account are "le", "l'", "la", "les", "de" and "du".

It would be great if you could incorporate this kind of duplicate detection for spanish and french (and surely also for other languages).

Thank you very much for all your work, and congratulations for this plugin,

Odinokij.

[BetterRed]

@Odinokij - search this thread for 'articles', I get the impression the PI will detect articles if tweaks and language settings are set appropriately. The tweak - "Set the list of words considered to be "articles" for sort strings" appears to be the 'key'.

BR

[odinokij]

Originally Posted by BetterRed

@Odinokij - search this thread for 'articles', I get the impression the PI will detect articles if tweaks and language settings are set appropriately. The tweak - "Set the list of words considered to be "articles" for sort strings" appears to be the 'key'.

BR

Thanks for your answer BetterRed, but...

In the calibre configuration i've got the default value for "per_language_title_sort_articles" (for spanish: 'spa': ('El\\s+', 'La\\s+', 'Lo\\s+', 'Los\\s+', 'Las\\s+', 'Un\\s+', 'Una\\s+', 'Unos\\s+', 'Unas\\s+') ) that may be considered correct (more or less)

But the Find Duplicate plugin doesn't detect duplicity for "El tercer hombre" vs "Tercer hombre" (both with the same author, and language:Español (spanish)) using "fuzzy-fuzzy" nor soundex(6)-soundex(8).

If I test with the books "The third man" and "Third man" (both with same author and language:Inglés (english) the plugin detects the duplicity in fuzzy-fuzzy mode.

Thank you for your help,
Odinokij

[BetterRed]

Originally Posted by odinokij

Thanks for your answer BetterRed, but...

In the calibre configuration i've got the default value for "per_language_title_sort_articles" (for spanish: 'spa': ('El\\s+', 'La\\s+', 'Lo\\s+', 'Los\\s+', 'Las\\s+', 'Un\\s+', 'Una\\s+', 'Unos\\s+', 'Unas\\s+') ) that may be considered correct (more or less)

But the Find Duplicate plugin doesn't detect duplicity for "El tercer hombre" vs "Tercer hombre" (both with the same author, and language:Español (spanish)) using "fuzzy-fuzzy" nor soundex(6)-soundex(8).

If I test with the books "The third man" and "Third man" (both with same author and language:Inglés (english) the plugin detects the duplicity in fuzzy-fuzzy mode.

Thank you for your help,
Odinokij

Odinokij - I just tried it with "El tercer hombre" and "Tercer hombre" both by Lionel Messi I put Spanish in book language and configured calibre to Spanish - same result as you, did not show up as duplicate at any setting (exact - fuzzy)

If I change the 'El' to 'The' and leave everything else the same - it does show up as a duplicate

My conclusion - either non English articles of speech has never worked in this PI or they've stopped working. I did the same test in calibre 1.48 with same result - so...

I don't think I can help any further on this. Hopefully someone will come by who can help

Good luck

BR

[laly003]

Hi, I've noticed that the binary compare is no longer detecting duplicates. I know I have duplicates and in fact re-imported a book already in my library and the plugin did not detect the duplicate. It just keeps saying no duplicates found. This used to work for me awhile back, sometime last year is when I last used it. I upgraded to the lastest (as of yesterday, 04/26/2015) versions of Windows x64 Calibre and latest plugin version of Find Duplicates. Is anyone else having a problem? Any suggestions? I've tested in 2 different PCs and same results.

[theducks]

It is working, but it is mostly WRONG (binary compare)

of 7 found dup sets.
Only 1 set was a true dup. (visual content check. Title was wrong in DB)

2 sets did not even have the SAME author
3 sets had the same file size, 2 were within a tenth
0 sets had the same Page count (count pages:Adobe method)

I normally use the Title/Author: Similar setting