.... and again duplicates ....

[jekkii]

I systemize and sort my library till now on CDs with Calibre. I always find new duplicates.
I don't know how Python works as for finding duplicates (comparing titles and authors?).
Earlier i used a software programmed in Visual Basic 6 and based on MS Access (i already mentioned, this : http://depositfiles.com/de/files/xmgx3g3nr). It is fairly rudimentary but one good thing was that the duplicates were really well filtered. There were calculated MD5 hashes of the book files during the scan process and the files with the same hash identified as duplicates. I have only a vague imagination what hash is and how complicated it would be to integrate this process in Calibre, but the result was very good.
So if i have e.g. a book with the title "Nice world" and the same book with the title "World nice" (because the scanner haven't made somehow the right job), Calibre finds them two different books although they are same. On the other way (per hash) there would have been identified as duplicates.
Best regards

[kacir]

Originally Posted by jekkii

I systemize and sort my library till now on CDs with Calibre. I always find new duplicates.
I don't know how Python works as for finding duplicates (comparing titles and authors?).
Earlier i used a software programmed in Visual Basic 6 and based on MS Access (i already mentioned, this : http://depositfiles.com/de/files/xmgx3g3nr). It is fairly rudimentary but one good thing was that the duplicates were really well filtered. There were calculated MD5 hashes of the book files during the scan process and the files with the same hash identified as duplicates. I have only a vague imagination what hash is and how complicated it would be to integrate this process in Calibre, but the result was very good.
So if i have e.g. a book with the title "Nice world" and the same book with the title "World nice" (because the scanner haven't made somehow the right job), Calibre finds them two different books although they are same. On the other way (per hash) there would have been identified as duplicates.
Best regards

See this thread for better discussion.
https://www.mobileread.com/forums/sho...d.php?t=118013
A system for Duplicate detection is in the making.

There is a problem with your hash theory.
You can have a duplicate book but version in Calibre is in txt format and version what you are adding is in epub. So you want the program to add the epub next to the txt.
I have many, many texts in Calibre that are in several formats.

[itimpi]

There is also the fact that a book can be a duplicte even though it is not byte identical to an existing file. for instance it might just have different metadata stored inside it.

The key point is that Calibre is working at the 'book' level and not the 'file' level when considering duplicates.

[DoctorOhh]

Originally Posted by itimpi

There is also the fact that a book can be a duplicte even though it is not byte identical to an existing file. for instance it might just have different metadata stored inside it.

The key point is that Calibre is working at the 'book' level and not the 'file' level when considering duplicates.

Excellent point, anyone wishing to use MD5 hashes to find dupes should do so prior to adding the books. Just viewing a epub in calibre changes the book file and thus the hash due to adding or changing the bookmark.

[kiwidude]

Originally Posted by dwanthny

Excellent point, anyone wishing to use MD5 hashes to find dupes should do so prior to adding the books.

Couldn't agree more - there are reasons why we haven't bothered to mention any kind of hash duplicate comparisons in the proposals for duplicate functionality in Calibre and you have covered most of them.

Personally I am very much in the "before you add to Calibre" camp. Why waste your time cleaning up filenames of files (or fixing up metadata inside Calibre)? Just run a hash comparison using any one of a number of free utilities out there on the internet first on your source folder and Calibre, then delete from the source folder. Don't directly delete from Calibre's folders though - or if you do you will need to run one of the repair database options to get Calibre's internal database matching the fact that a book format is no longer present.

Just viewing a epub in calibre changes the book file and thus the hash due to adding or changing the bookmark.

Ok, so this used to "always" be the case. Then a number of us campaigned to have an option available such that the EPUB would *not* be touched. It screws with incremental backups and obviously hash comparisons for those of use who have no interest in "touching" an EPUB just by opening it in the viewer. I wrote a hacky patch then chaley from memory did the job "properly" in a Calibre release a few months ago.

In the ebook viewer preferences if you disable "Remember the current page when quitting" and don't add bookmarks then your EPUB should remain untouched - or at least that was the hope .