Identifying duplicate books with same content, but different tags

[kirk8677]

It seems that the duplicate book finder plug in finds books with duplicate tags.

For me, the reverse functionality would be more useful: identifying duplicates that are the same book from their binary content, but MAY have different tags (or could be the same).

The duplicate file finder utilities I have seen DO NOT recognize a duplicate file when the tag changes, otherwise I would run files through a de-dupe software BEFORE adding.

Any suggestions?

Thanks!

[DNSB]

Originally Posted by kirk8677

It seems that the duplicate book finder plug in finds books with duplicate tags.

For me, the reverse functionality would be more useful: identifying duplicates that are the same book from their binary content, but MAY have different tags (or could be the same).

The duplicate file finder utilities I have seen DO NOT recognize a duplicate file when the tag changes, otherwise I would run files through a de-dupe software BEFORE adding.

Any suggestions?

Thanks!

I've been using the Find Duplicates plugin for calibre with the binary option. Please note that any changes to the file such as using Polish, Modify Epub, etc. will make this option useless.

[kirk8677]

Thanks, I didn't see that option - do you know if there's any way to run this automatically when adding a book to the library, so that it's not added at all if it binary matches?

[DNSB]

Originally Posted by kirk8677

Thanks, I didn't see that option - do you know if there's any way to run this automatically when adding a book to the library, so that it's not added at all if it binary matches?

Not as far as I know. Though my workflow would make checking on add rather useless. I add to an Intake library and then move to my Main library when I've finished metadata cleanup, covers, editing if needed, etc.

OTOH, I can't think of a ebook supplier that supplies identical files with different metadata that is not also embedded in the file which would make the binary compare pretty useless.

Which brings up the question as to which source you are using that supplies binary identical ebook files with different title/authors/etc. information.

[kirk8677]

It actually also doesn't really seem to work for me. The experiment I did was to export a book, change the metadata, then add the changed one back. I did a binary compare and it didn't see the duplicate, for some reason.

[kirk8677]

Originally Posted by DNSB

Not as far as I know. Though my workflow would make checking on add rather useless. I add to an Intake library and then move to my Main library when I've finished metadata cleanup, covers, editing if needed, etc.

OTOH, I can't think of a ebook supplier that supplies identical files with different metadata that is not also embedded in the file which would make the binary compare pretty useless.

Which brings up the question as to which source you are using that supplies binary identical ebook files with different title/authors/etc. information.

There are numerous sources of free, poorly organized books on the internet. Shall I point you to a few of them, in case you haven't seen them?

[DNSB]

Originally Posted by kirk8677

It actually also doesn't really seem to work for me. The experiment I did was to export a book, change the metadata, then add the changed one back. I did a binary compare and it didn't see the duplicate, for some reason.

Which matches what I said in the second message in this thread:

Please note that any changes to the file such as using Polish, Modify Epub, etc. will make this option useless.

When you modify the metadata in the file, it is no longer binary identical. The binary compare first looks at the file sizes, if the file sizes are identical, it generates SHA hashes. If those hashes match, the two files are considered as being duplicates.

[DNSB]

Originally Posted by kirk8677

There are numerous sources of free, poorly organized books on the internet. Shall I point you to a few of them, in case you haven't seen them?

I've seen enough of them over the years. For the most part, the type of site that should include the Jolly Roger in their logos since most of those "free, poorly organized books" are also still in copyright.

'Nuff said. Good luck.

[kirk8677]

Originally Posted by DNSB

I've seen enough of them over the years. For the most part, the type of site that should include the Jolly Roger in their logos since most of those "free, poorly organized books" are also still in copyright.

'Nuff said. Good luck.

False. Good luck.