Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Plugins

Notices

Reply
 
Thread Tools Search this Thread
Old 08-03-2014, 02:04 AM   #481
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 3,996
Karma: 2925589
Join Date: Mar 2012
Location: NSW Australia
Device: none
Quote:
Originally Posted by JSWolf View Post
When I use "Show one group at a time", how do I switch from one group to the next?
Find Duplicates->Next/Previous result. I have them shortcutted to ctrl/alt/right & left respectively.

Quote:
Originally Posted by LDMartin1959 View Post
BetterRed, I am searching for dups in a single library (and if it makes a difference, I only have the one), binary compare, show all groups at once with highlighting, sort groups by number of duplicates.
I believe binary compares can be slow if there are lots of them - suggest you peruse this thread for discussion. I don't normally use Find Duplicates binary compare, but I just did two tests. The PI found a duplicate PDF of a 19th century novel, that I added into a Yale Law 'book' on UN R2P provisions, took a few seconds - this was a small test library. A second test in a library of ~8000 'books' found three binary duplicates that I need to look into - this took a couple of minutes.

BR
BetterRed is offline   Reply With Quote
Old 08-03-2014, 12:13 PM   #482
JSWolf
Resident Curmudgeon
JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.
 
JSWolf's Avatar
 
Posts: 38,553
Karma: 19637653
Join Date: Nov 2006
Location: Roslindale, Massachusetts
Device: Kobo Aura H2O, Sony PRS-650, Sony PRS-T1, nook STR, iPad 1, iPhone 5
Quote:
Originally Posted by BetterRed View Post
Find Duplicates->Next/Previous result. I have them shortcutted to ctrl/alt/right & left respectively.
Thanks. Found it.
JSWolf is offline   Reply With Quote
Old 08-05-2014, 05:38 PM   #483
LDMartin1959
Member
LDMartin1959 began at the beginning.
 
Posts: 10
Karma: 10
Join Date: Oct 2012
Device: iPhone/iPad/Calibre
Quote:
Originally Posted by BetterRed View Post
Find Duplicates->Next/Previous result. I have them shortcutted to ctrl/alt/right & left respectively.


I believe binary compares can be slow if there are lots of them - suggest you peruse this thread for discussion. I don't normally use Find Duplicates binary compare, but I just did two tests. The PI found a duplicate PDF of a 19th century novel, that I added into a Yale Law 'book' on UN R2P provisions, took a few seconds - this was a small test library. A second test in a library of ~8000 'books' found three binary duplicates that I need to look into - this took a couple of minutes.

BR
BetterRed, I have done binary compares in the past and understand they can take awhile. But max has been maybe an hour or so, never going into the realm of days.
LDMartin1959 is offline   Reply With Quote
Old 08-05-2014, 07:40 PM   #484
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 3,996
Karma: 2925589
Join Date: Mar 2012
Location: NSW Australia
Device: none
Quote:
Originally Posted by LDMartin1959 View Post
BetterRed, I have done binary compares in the past and understand they can take awhile. But max has been maybe an hour or so, never going into the realm of days.
@LDMartin1959 - Hmmm, so we can blame the Time Lords - ask the good Dr Who

Seriously though, the only thing I can imagine, and it's a long stretch, is that via junctions and/or symlinks you have introduced self referential loops into the library folder structure. You could use Nir Sofer's NTFSLinksView utility to have a look for such.

BR
BetterRed is offline   Reply With Quote
Old 08-06-2014, 12:17 PM   #485
LDMartin1959
Member
LDMartin1959 began at the beginning.
 
Posts: 10
Karma: 10
Join Date: Oct 2012
Device: iPhone/iPad/Calibre
Quote:
Originally Posted by BetterRed View Post
@LDMartin1959 - Hmmm, so we can blame the Time Lords - ask the good Dr Who

Seriously though, the only thing I can imagine, and it's a long stretch, is that via junctions and/or symlinks you have introduced self referential loops into the library folder structure. You could use Nir Sofer's NTFSLinksView utility to have a look for such.

BR
I'm somewhat familiar with symbolic links, not so with junctions. Would those be something that might have been created by Calibre itself? As far as I am aware I have not done anything manually which would have created such things.

With additional playing around I may have discovered that the "sort groups by number of duplicates" may be partly to blame (not 100% sure). When I remove that option, the search will complete after a few hours. But why that option should cause minutes to become hours and hours to become days, I don't know.

Thanks for the utility reference. Sadly, it is Windows only.
LDMartin1959 is offline   Reply With Quote
Old 08-06-2014, 08:19 PM   #486
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 3,996
Karma: 2925589
Join Date: Mar 2012
Location: NSW Australia
Device: none
Quote:
Originally Posted by LDMartin1959 View Post
I'm somewhat familiar with symbolic links, not so with junctions. Would those be something that might have been created by Calibre itself? As far as I am aware I have not done anything manually which would have created such things.

AFAIK calibre makes no use of symlinks or junctions, so if they did exist they would have been created by you via the 'ln' command or some other program, as I said 'it's a long stretch'. Junctions are similar to and 'sort of' deprecated by folder symlinks. Interestingly they played a role in Stuxnet.

With additional playing around I may have discovered that the "sort groups by number of duplicates" may be partly to blame (not 100% sure). When I remove that option, the search will complete after a few hours. But why that option should cause minutes to become hours and hours to become days, I don't know.
@LDMartin1959 - I can only imagine that the sort is being done whenever a duplicate is found (ie 2,000 duplicates = 2,000 sorts), rather than once at the end of search - hard to believe but.

If binary duplicate searching of this ilk is an ongoing need then you might want to consider using a separate tool to find them.

BR
BetterRed is offline   Reply With Quote
Old 08-06-2014, 11:33 PM   #487
LDMartin1959
Member
LDMartin1959 began at the beginning.
 
Posts: 10
Karma: 10
Join Date: Oct 2012
Device: iPhone/iPad/Calibre
Quote:
Originally Posted by BetterRed View Post
@LDMartin1959 - I can only imagine that the sort is being done whenever a duplicate is found (ie 2,000 duplicates = 2,000 sorts), rather than once at the end of search - hard to believe but.

If binary duplicate searching of this ilk is an ongoing need then you might want to consider using a separate tool to find them.

BR
Thanks for the input.
LDMartin1959 is offline   Reply With Quote
Old 08-07-2014, 12:42 AM   #488
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 6,090
Karma: 6238033
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo Touch, Kobo Glo
Looking at the code, the sort is done once after all the duplicates have been found. I can see the sort would be longer than the name sort, but, I wouldn't have expected it to be that much longer.
davidfor is offline   Reply With Quote
Old 08-07-2014, 01:58 AM   #489
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 3,996
Karma: 2925589
Join Date: Mar 2012
Location: NSW Australia
Device: none
Quote:
Originally Posted by davidfor View Post
Looking at the code, the sort is done once after all the duplicates have been found. I can see the sort would be longer than the name sort, but, I wouldn't have expected it to be that much longer.
Good to know the sorting is done at the end, otherwise my faith in kiwidudes good sense would have been called into question.

Curious: Why would you you expect sorting a collection of ints to be slower than sorting a collection of char arrays

BR
BetterRed is offline   Reply With Quote
Old 08-07-2014, 02:13 AM   #490
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 6,090
Karma: 6238033
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo Touch, Kobo Glo
Quote:
Originally Posted by BetterRed View Post
Good to know the sorting is done at the end, otherwise my faith in kiwidudes good sense would have been called into question.

Curious: Why would you you expect sorting a collection of ints to be slower than sorting a collection of char arrays
It isn't sorting just by an integer. The sort code is:

Code:
if by_title:
    skeys = sorted(candidates_map.keys())
else:
    skeys = sorted(candidates_map.keys(),
               key=lambda ckey: '%04d%s' % (len(candidates_map[ckey]), ckey),
               reverse=True)
So, get the count of duplicates books for a title, convert to string and combine with the title. That will be slower for a lot of duplicates, but I'm not sure how noticeable it would be.
davidfor is offline   Reply With Quote
Old 08-07-2014, 04:41 AM   #491
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 3,996
Karma: 2925589
Join Date: Mar 2012
Location: NSW Australia
Device: none
@davidfor - I still think in C and asm The conversion could chew cycles, IIRC conversions in .Net CIL and JRE were/are known to be slow, dunno about Python IL.

@LDMartin1959 - I import books into a 'receiving dock' library and do an inter library Find Duplicates. Any duplicates get moved into a 'duplicates' library, and the non-duplicates get moved to the 'main' library. For me this works better functionally and it's faster than doing everything in one library - and it keeps my 'main' library pristine.

BR
BetterRed is offline   Reply With Quote
Old 08-07-2014, 03:23 PM   #492
LDMartin1959
Member
LDMartin1959 began at the beginning.
 
Posts: 10
Karma: 10
Join Date: Oct 2012
Device: iPhone/iPad/Calibre
Quote:
Originally Posted by BetterRed View Post
@davidfor - I still think in C and asm The conversion could chew cycles, IIRC conversions in .Net CIL and JRE were/are known to be slow, dunno about Python IL.

@LDMartin1959 - I import books into a 'receiving dock' library and do an inter library Find Duplicates. Any duplicates get moved into a 'duplicates' library, and the non-duplicates get moved to the 'main' library. For me this works better functionally and it's faster than doing everything in one library - and it keeps my 'main' library pristine.

BR
BR, not a bad idea. Unfortunately, it's too late for me! I've already dug myself into a hole and thrown the dirt back over the top!!
LDMartin1959 is offline   Reply With Quote
Old 08-07-2014, 07:22 PM   #493
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 3,996
Karma: 2925589
Join Date: Mar 2012
Location: NSW Australia
Device: none
Quote:
Originally Posted by LDMartin1959 View Post
BR, not a bad idea. Unfortunately, it's too late for me! I've already dug myself into a hole and thrown the dirt back over the top!!
- 'fraid, I'm out of ladders

BR
BetterRed is offline   Reply With Quote
Old 08-09-2014, 06:28 AM   #494
AnitaBlake
Junior Member
AnitaBlake began at the beginning.
 
Posts: 1
Karma: 10
Join Date: Aug 2014
Device: Kindle
Post Duplicate find extension

Dear plugin developer,

first of all thank you very much for great plugin you developed for us.
I am using it very often and it saved a lot of time during my library management.

To extend its perfection, I have one idea . Maybe some other plugin user will find it useful as well.

Would it be please possible to make Title part of "Title/Author" search as variable to have possibility to do the search against custom defined column eg. Original_Title/Author?

If not, anyway thanx again for such a great piece of software.

AnitaBlake is offline   Reply With Quote
Old 08-19-2014, 10:25 AM   #495
rebl
Enthusiast
rebl began at the beginning.
 
Posts: 44
Karma: 10
Join Date: Mar 2010
Location: Bukarest
Device: Kindle 5 NT
Later Edit:
I found it! The setting was there in preferences/searching: Highlight search results....
It was checked and I am 99,99% sure I haven't done this.
Problem was solved by unchecking it.
I'm not sure if I should delete whole post, or leave it in case others have this problem.
I am also not sure if indeed Duplicate Finder modified this setting, in case it absolutely did not the this is wholly off-topping and I'm sorry for that.


L.E.2:
I see my problem was already mentioned buy I did not know the corect wording of that so my searches did not get me to that page:
http://www.mobileread.com/forums/sho...d.php?t=237884
So it seems that would confirm that Duplicate Finder does modify that setting.
In my humble opinion, a warning about that on it's download page would be nice (if it's already there then it must have escaped me somehow).

Last edited by rebl; 08-22-2014 at 03:49 AM.
rebl is offline   Reply With Quote
Reply

Tags
cross library duplicates, in library duplicates

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[GUI Plugin] Generate Cover kiwidude Plugins 504 Today 09:12 PM
[GUI Plugin] Quality Check kiwidude Plugins 797 12-16-2014 01:01 PM
[GUI Plugin] Open With kiwidude Plugins 251 12-15-2014 01:07 PM
[GUI Plugin] View Manager kiwidude Plugins 91 11-17-2014 01:40 AM
[GUI Plugin] Plugin Updater **Deprecated** kiwidude Plugins 159 06-19-2011 01:27 PM


All times are GMT -4. The time now is 10:40 PM.


MobileRead.com is a privately owned, operated and funded community.