[GUI Plugin] Find Duplicates

[PPP-Magic]

Doh, not sure how I missed that LOL

Thanks for the help.

[Rellwood]

Might be a dumb or obvious question/answer, but in the case of tags where there are multiple matches for duplicates and only a couple of those matches in the group are actual matches, how do I specify that I only want to use those specific ones out of the group? It seems like it's an all or none.

Example 2stars, 2-stars,2.stars,3-stars,3.stars,5-stars
I match 2-stars for all of them and I want to use 2.stars for the 2 star ones but none of the other ones, which should get their own matches.

I ignore them and end up missing out all together. Am I missing something?

[JSWolf]

Originally Posted by theducks

Read the last line.
3.21 does not meet the PI requirements.

I suspect your distro is typical and WAY BEHIND.

Use the command found on the Calibre Linux Download page and join us with a modern 5.x version.

Your other option, is to uninstall the PI and find someone with an old version of it

Never ever use a Linux repository. It is a waste of time and you could find things break.

[mklauber]

Hey, I'm a big fan of your Find Duplicates Plugin. I was wondering if it would be feasible to have it store the binary hashes in the database, to reduce the amount of work needed to do a comparison. Specifically, the binary compare has gotten slow as my library had increased beyond the 1TB mark. Is this a feature that can be added, or can I fork the code and attempt to add it myself somewhere?

[theducks]

Originally Posted by mklauber

Hey, I'm a big fan of your Find Duplicates Plugin. I was wondering if it would be feasible to have it store the binary hashes in the database, to reduce the amount of work needed to do a comparison. Specifically, the binary compare has gotten slow as my library had increased beyond the 1TB mark. Is this a feature that can be added, or can I fork the code and attempt to add it myself somewhere?

Patch submissions (put in this thread, WELL Labeled) are OK.
MR user should be cautioned about using experimental code.

Now the critique. A hash may only be good at the time.
ANY change to the book (polish, embed, edit) will change that value.
How do you NOW know the value is stale???

Value Storage: A custom column will stay with the Library

[chaley]

Originally Posted by mklauber

Hey, I'm a big fan of your Find Duplicates Plugin. I was wondering if it would be feasible to have it store the binary hashes in the database, to reduce the amount of work needed to do a comparison. Specifically, the binary compare has gotten slow as my library had increased beyond the 1TB mark. Is this a feature that can be added, or can I fork the code and attempt to add it myself somewhere?

Keeping in mind @theduck's comments ...

Plugins are provided as source code. You are free to change them however you want. Getting your changes released requires consent of the plugin's current maintainer. See Writing your own plugins to extend calibre’s functionality for some guidance on writing & changing plugins.

You can store the hashes in calibre's database using db.cache.add_custom_book_data. Personally I would store both the hash and the format's last-modified date so you can have a clue that the hash is still valid.

[capink]

Version 1.9.7

• Update: update to calibre6 icon themes. Code borrowed from @JimmXinu.

[jluaioyj]

I'd like to suggest that Find Duplicates's Binary Compare be enhanced to support storage and comparison by hash/digest to significant speed it up, by avoiding most/all redundant full file compares. A file compare on hash/digest could be an option if hash/digest collision false-matches are suspected, as were discovered occurring sometimes for MD5.

If suggest storage of binary comparison metadata in a custom field, and that this field contains a JSON map with a hash/digest type, and a key for each file format, with the value map containing, a hexadecimal formatted file hash/digest (I'd suggest SHA256), the last file size, and the file last modified timestamp; the later two for validation.

If this custom field was not configured, like the "Last Modified" plugin does, a warning should be displayed, then the current _slow_ full file compare functionality used instead.

During a binary search:
* If any of the field maps are missing, they should be created.
* If the field value is junk or if the hash/digest type is obsolete, the whole field map should be recreated.
* If a format file is missing, it should be removed from the field map.
* If a format file was added, it should be added to the map.
* If the hash/digest value is out-of-date (file size or last modified changed), the type map should be re-built.

It would be a nice if the above rules were applied after a book entry was added, after any formats were added/removed, and after any in-calibre format file changes; obviously, I would not expect this to be able to spot any updates outside of calibre.

I'd suggest that this field, and its value creation, validation, and updating should really be provided by calibre itself.

[capink]

Originally Posted by jluaioyj

I'd like to suggest that Find Duplicates's Binary Compare be enhanced to support storage and comparison by hash/digest to significant speed it up, by avoiding most/all redundant full file compares. A file compare on hash/digest could be an option if hash/digest collision false-matches are suspected, as were discovered occurring sometimes for MD5.

If suggest storage of binary comparison metadata in a custom field, and that this field contains a JSON map with a hash/digest type, and a key for each file format, with the value map containing, a hexadecimal formatted file hash/digest (I'd suggest SHA256), the last file size, and the file last modified timestamp; the later two for validation.

If this custom field was not configured, like the "Last Modified" plugin does, a warning should be displayed, then the current _slow_ full file compare functionality used instead.

During a binary search:
* If any of the field maps are missing, they should be created.
* If the field value is junk or if the hash/digest type is obsolete, the whole field map should be recreated.
* If a format file is missing, it should be removed from the field map.
* If a format file was added, it should be added to the map.
* If the hash/digest value is out-of-date (file size or last modified changed), the type map should be re-built.

It would be a nice if the above rules were applied after a book entry was added, after any formats were added/removed, and after any in-calibre format file changes; obviously, I would not expect this to be able to spot any updates outside of calibre.

I'd suggest that this field, and its value creation, validation, and updating should really be provided by calibre itself.

I never use the binary comparison. I keep my metadata as clean as possible and this helps me more in discovering duplicates even if the files are not identical. However, a quick look at the code reveals that the Find Duplicates already does everything you are asking for; with some caveats:

• The data for size, hash and mtime are stored using a calibre api method called add_multiple_custom_book_data(), instead of storing them in custom columns. This makes more sense and has the advantage of not burdening the user with unnecessary custom columns.
  
  N.B. If you are familiar with the Action Chains Plugin, you can use the chain attached below to see the data stored by the plugin (if any). (Action Chains > Add/Modify Chains > Right click the chain table > import)
  
• When you run the binary check, before using any stored hash, the plugin first verifies it is not stale. If the hash is stale, it is re-calculated.
• The plugin does not calculate and store hashes for all books. For sake of being economical, it only calculates hashes for group of formats that share the same size, which is bound to be a small subset of the formats in the library.

Given the last point, automatic hash calculation on book additions does not make much sense. It can be done but will not be of much use, because only a small subset of these hashes will be needed based on size comparisons. In addition to this, calculating book hashes will slow down adding books, especially if the user is adding a large number of books.

[Rellwood]

I was wondering if you could add a feature that put tags with a specific character as the default rename. I hate having to keep right clicking on the tags with periods.

[Winnito]

Not sure if I asked this before or if this is some other place discussed.

I'm puzzled how when I do "similar" search in Metadata duplicates it can't find what I guess should be obvous hits like:

Doe, John
Doe, John L.

I have so many authors in my database with and without their middle names, but can't find most of them in duplicate.

Similar gives nothing, Fuzzy gives 20x more false positives...not even close to being useful. Am I missing something? TNx

[kiwidude]

Originally Posted by Winnito

Not sure if I asked this before or if this is some other place discussed.

I'm puzzled how when I do "similar" search in Metadata duplicates it can't find what I guess should be obvous hits like:

Doe, John
Doe, John L.

I have so many authors in my database with and without their middle names, but can't find most of them in duplicate.

Similar gives nothing, Fuzzy gives 20x more false positives...not even close to being useful. Am I missing something? TNx

Works fine for me - both as a normal duplicate check and as a metadata check

[capink]

@kiwidude: He does not want to use fuzzy author matching, because for him, it produces too much false positives.

This question was asked by the same user before, and an answer was provided here.

[kiwidude]

@capink - nice one - my bad for not properly reading the post while I was focused on other things...

[theducks]

Then I would suggest the old Eyeball Mk II
In the Tag Browser: Right click (authors): Manage

Be sure to check (just do a Search and validate what you see) before you rename. Some Authors use their MI to differentiate from another Author with the same Fn Ln

And some Publishers leave the Initial off the cover . Gotta love consistancy in the same house

BTW I have to do this with Series. Sometimes I add a 'tie breaker': Series (Authors Initials), to the name. The Paren does not mess the sort.