[GUI Plugin] Find Duplicates

[BetterRed]

Originally Posted by JSWolf

When I use "Show one group at a time", how do I switch from one group to the next?

Find Duplicates->Next/Previous result. I have them shortcutted to ctrl/alt/right & left respectively.

Originally Posted by LDMartin1959

BetterRed, I am searching for dups in a single library (and if it makes a difference, I only have the one), binary compare, show all groups at once with highlighting, sort groups by number of duplicates.

I believe binary compares can be slow if there are lots of them - suggest you peruse this thread for discussion. I don't normally use Find Duplicates binary compare, but I just did two tests. The PI found a duplicate PDF of a 19th century novel, that I added into a Yale Law 'book' on UN R2P provisions, took a few seconds - this was a small test library. A second test in a library of ~8000 'books' found three binary duplicates that I need to look into - this took a couple of minutes.

BR

[JSWolf]

Originally Posted by BetterRed

Find Duplicates->Next/Previous result. I have them shortcutted to ctrl/alt/right & left respectively.

Thanks. Found it.

[LDMartin1959]

Originally Posted by BetterRed

Find Duplicates->Next/Previous result. I have them shortcutted to ctrl/alt/right & left respectively.


I believe binary compares can be slow if there are lots of them - suggest you peruse this thread for discussion. I don't normally use Find Duplicates binary compare, but I just did two tests. The PI found a duplicate PDF of a 19th century novel, that I added into a Yale Law 'book' on UN R2P provisions, took a few seconds - this was a small test library. A second test in a library of ~8000 'books' found three binary duplicates that I need to look into - this took a couple of minutes.

BR

BetterRed, I have done binary compares in the past and understand they can take awhile. But max has been maybe an hour or so, never going into the realm of days.

[BetterRed]

Originally Posted by LDMartin1959

BetterRed, I have done binary compares in the past and understand they can take awhile. But max has been maybe an hour or so, never going into the realm of days.

@LDMartin1959 - Hmmm, so we can blame the Time Lords - ask the good Dr Who

Seriously though, the only thing I can imagine, and it's a long stretch, is that via junctions and/or symlinks you have introduced self referential loops into the library folder structure. You could use Nir Sofer's NTFSLinksView utility to have a look for such.

BR

[LDMartin1959]

Originally Posted by BetterRed

@LDMartin1959 - Hmmm, so we can blame the Time Lords - ask the good Dr Who

Seriously though, the only thing I can imagine, and it's a long stretch, is that via junctions and/or symlinks you have introduced self referential loops into the library folder structure. You could use Nir Sofer's NTFSLinksView utility to have a look for such.

BR

I'm somewhat familiar with symbolic links, not so with junctions. Would those be something that might have been created by Calibre itself? As far as I am aware I have not done anything manually which would have created such things.

With additional playing around I may have discovered that the "sort groups by number of duplicates" may be partly to blame (not 100% sure). When I remove that option, the search will complete after a few hours. But why that option should cause minutes to become hours and hours to become days, I don't know.

Thanks for the utility reference. Sadly, it is Windows only.

[BetterRed]

Originally Posted by LDMartin1959

I'm somewhat familiar with symbolic links, not so with junctions. Would those be something that might have been created by Calibre itself? As far as I am aware I have not done anything manually which would have created such things.

AFAIK calibre makes no use of symlinks or junctions, so if they did exist they would have been created by you via the 'ln' command or some other program, as I said 'it's a long stretch'. Junctions are similar to and 'sort of' deprecated by folder symlinks. Interestingly they played a role in Stuxnet.

With additional playing around I may have discovered that the "sort groups by number of duplicates" may be partly to blame (not 100% sure). When I remove that option, the search will complete after a few hours. But why that option should cause minutes to become hours and hours to become days, I don't know.

@LDMartin1959 - I can only imagine that the sort is being done whenever a duplicate is found (ie 2,000 duplicates = 2,000 sorts), rather than once at the end of search - hard to believe but.

If binary duplicate searching of this ilk is an ongoing need then you might want to consider using a separate tool to find them.

BR

[LDMartin1959]

Originally Posted by BetterRed

@LDMartin1959 - I can only imagine that the sort is being done whenever a duplicate is found (ie 2,000 duplicates = 2,000 sorts), rather than once at the end of search - hard to believe but.

If binary duplicate searching of this ilk is an ongoing need then you might want to consider using a separate tool to find them.

BR

Thanks for the input.

[davidfor]

Looking at the code, the sort is done once after all the duplicates have been found. I can see the sort would be longer than the name sort, but, I wouldn't have expected it to be that much longer.

[BetterRed]

Originally Posted by davidfor

Looking at the code, the sort is done once after all the duplicates have been found. I can see the sort would be longer than the name sort, but, I wouldn't have expected it to be that much longer.

Good to know the sorting is done at the end, otherwise my faith in kiwidudes good sense would have been called into question.

Curious: Why would you you expect sorting a collection of ints to be slower than sorting a collection of char arrays

BR

[davidfor]

Originally Posted by BetterRed

Good to know the sorting is done at the end, otherwise my faith in kiwidudes good sense would have been called into question.

Curious: Why would you you expect sorting a collection of ints to be slower than sorting a collection of char arrays

It isn't sorting just by an integer. The sort code is:

if by_title:
    skeys = sorted(candidates_map.keys())
else:
    skeys = sorted(candidates_map.keys(),
               key=lambda ckey: '%04d%s' % (len(candidates_map[ckey]), ckey),
               reverse=True)

So, get the count of duplicates books for a title, convert to string and combine with the title. That will be slower for a lot of duplicates, but I'm not sure how noticeable it would be.

[BetterRed]

@davidfor - I still think in C and asm The conversion could chew cycles, IIRC conversions in .Net CIL and JRE were/are known to be slow, dunno about Python IL.

@LDMartin1959 - I import books into a 'receiving dock' library and do an inter library Find Duplicates. Any duplicates get moved into a 'duplicates' library, and the non-duplicates get moved to the 'main' library. For me this works better functionally and it's faster than doing everything in one library - and it keeps my 'main' library pristine.

BR

[LDMartin1959]

Originally Posted by BetterRed

@davidfor - I still think in C and asm The conversion could chew cycles, IIRC conversions in .Net CIL and JRE were/are known to be slow, dunno about Python IL.

@LDMartin1959 - I import books into a 'receiving dock' library and do an inter library Find Duplicates. Any duplicates get moved into a 'duplicates' library, and the non-duplicates get moved to the 'main' library. For me this works better functionally and it's faster than doing everything in one library - and it keeps my 'main' library pristine.

BR

BR, not a bad idea. Unfortunately, it's too late for me! I've already dug myself into a hole and thrown the dirt back over the top!!

[BetterRed]

Originally Posted by LDMartin1959

BR, not a bad idea. Unfortunately, it's too late for me! I've already dug myself into a hole and thrown the dirt back over the top!!

- 'fraid, I'm out of ladders

BR

[AnitaBlake]

Dear plugin developer,

first of all thank you very much for great plugin you developed for us.
I am using it very often and it saved a lot of time during my library management.

To extend its perfection, I have one idea . Maybe some other plugin user will find it useful as well.

Would it be please possible to make Title part of "Title/Author" search as variable to have possibility to do the search against custom defined column eg. Original_Title/Author?

If not, anyway thanx again for such a great piece of software.

[rebl]

Later Edit:
I found it! The setting was there in preferences/searching: Highlight search results....
It was checked and I am 99,99% sure I haven't done this.
Problem was solved by unchecking it.
I'm not sure if I should delete whole post, or leave it in case others have this problem.
I am also not sure if indeed Duplicate Finder modified this setting, in case it absolutely did not the this is wholly off-topping and I'm sorry for that.


L.E.2:
I see my problem was already mentioned buy I did not know the corect wording of that so my searches did not get me to that page:
https://www.mobileread.com/forums/sho...d.php?t=237884
So it seems that would confirm that Duplicate Finder does modify that setting.
In my humble opinion, a warning about that on it's download page would be nice (if it's already there then it must have escaped me somehow).