Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Plugins

Notices

Reply
 
Thread Tools Search this Thread
Old 12-02-2012, 11:17 PM   #346
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
Quote:
Originally Posted by luciaisacat View Post
I installed v 1.6 and now the plugin fails to find duplicates. Anyone else is experiencing this problem?
1.6 is working fine for me. What options are you selecting when finding duplicates?

The plugin states:

Quote:
Special Notes:
  • Requires Calibre 0.8.59 or later.
What version of calibre are you running?

Quote:
Originally Posted by luciaisacat View Post
Is there an archive for old plugin?
No.
DoctorOhh is offline   Reply With Quote
Old 12-02-2012, 11:40 PM   #347
luciaisacat
Enthusiast
luciaisacat began at the beginning.
 
luciaisacat's Avatar
 
Posts: 32
Karma: 10
Join Date: May 2012
Device: android
Quote:
Originally Posted by DoctorOhh View Post
1.6 is working fine for me. What options are you selecting when finding duplicates?
Title / Author = identical / identical


Quote:
Originally Posted by DoctorOhh View Post
What version of calibre are you running?
0.9.8

Do you know where could I retrieve v.1.5.x of the Find duplicate files plugin?

Many thanks!

lucia
luciaisacat is offline   Reply With Quote
Old 12-02-2012, 11:50 PM   #348
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
Quote:
Originally Posted by luciaisacat View Post

Title / Author = identical / identical
What makes you think it is not working? Have you tried

Title / Author = similar / similar

or

Title / Author = fuzzy / identical

or

Title / Author = soundex / identical

Or any other number of combinations?


Quote:
Originally Posted by luciaisacat View Post
Do you know where could I retrieve v.1.5.x of the Find duplicate files plugin?
No.
DoctorOhh is offline   Reply With Quote
Old 12-02-2012, 11:55 PM   #349
luciaisacat
Enthusiast
luciaisacat began at the beginning.
 
luciaisacat's Avatar
 
Posts: 32
Karma: 10
Join Date: May 2012
Device: android
Quote:
Originally Posted by DoctorOhh View Post
What makes you think it is not working? Have you tried

Title / Author = similar / similar

or

Title / Author = fuzzy / identical

or

Title / Author = soundex / identical

Or any other number of combinations?




No.
Yes, tried all possible combinations and at the end I just created a duplicate myself with same author and same title. It does not work!

for your
luciaisacat is offline   Reply With Quote
Old 12-03-2012, 12:02 AM   #350
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
Quote:
Originally Posted by luciaisacat View Post
Yes, tried all possible combinations and at the end I just created a duplicate myself with same author and same title. It does not work!
That seems thorough enough. You might want to restart calibre and have another go at it. Other than a complete reboot I have no other ideas.
DoctorOhh is offline   Reply With Quote
Old 12-03-2012, 12:06 AM   #351
luciaisacat
Enthusiast
luciaisacat began at the beginning.
 
luciaisacat's Avatar
 
Posts: 32
Karma: 10
Join Date: May 2012
Device: android
Quote:
Originally Posted by DoctorOhh View Post
That seems thorough enough. You might want to restart calibre and have another go at it. Other than a complete reboot I have no other ideas.
(Unfortunately) done that too!
luciaisacat is offline   Reply With Quote
Old 12-03-2012, 12:09 AM   #352
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 9,864
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
Quote:
Originally Posted by luciaisacat View Post
(Unfortunately) done that too!
I'm also using calibre 0.9.8 and v1.6 of the plugin, just to be thorough I added an exact dupe and the plugin had no problems finding it. I guess we'll have to wait for Kiwidude's input.
DoctorOhh is offline   Reply With Quote
Old 12-03-2012, 04:14 AM   #353
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,601
Karma: 2092290
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
The v1.6 of this plugin *does* work - it works for me, Doctor Ohh, and about 25,000 other users. So it looks like something specific to your machine. However your complete lack of detail in your posts makes it impossible to suggest anything useful. The obvious questions are:
- whether you even have any duplicates for it to find.
- do you have a search restriction in place you have forgotten about which means they don't show up.
- exactly what type of duplicates search you are trying to do (attach screenshots, one showing your options, one showing your dups that exist you think you should be matching).

If you still have no joy (and you are doing anything other than a binary search) then zip up your metadata.db file from the root of your library folder, upload it somewhere and PM me a link to it along with your screenshot of your search options so I can try to replicate it.

There are no downloads available for older versions, I don't even keep them myself. I have neither the time, motivation (or sufficient donations) to support them. If there is a bug in the latest version then I would rather fix and push that out.

Last edited by kiwidude; 12-03-2012 at 05:27 AM. Reason: Fix typo in version number to prevent confusion
kiwidude is offline   Reply With Quote
Old 12-04-2012, 08:00 PM   #354
luciaisacat
Enthusiast
luciaisacat began at the beginning.
 
luciaisacat's Avatar
 
Posts: 32
Karma: 10
Join Date: May 2012
Device: android
Quote:
Originally Posted by kiwidude View Post
The v1.6 of this plugin *does* work - it works for me, Doctor Ohh, and about 25,000 other users. So it looks like something specific to your machine. However your complete lack of detail in your posts makes it impossible to suggest anything useful. The obvious questions are:
- whether you even have any duplicates for it to find.
- do you have a search restriction in place you have forgotten about which means they don't show up.
- exactly what type of duplicates search you are trying to do (attach screenshots, one showing your options, one showing your dups that exist you think you should be matching).

If you still have no joy (and you are doing anything other than a binary search) then zip up your metadata.db file from the root of your library folder, upload it somewhere and PM me a link to it along with your screenshot of your search options so I can try to replicate it.

There are no downloads available for older versions, I don't even keep them myself. I have neither the time, motivation (or sufficient donations) to support them. If there is a bug in the latest version then I would rather fix and push that out.
Many thanks for your help. I will go once again through all the steps and will try again.
luciaisacat is offline   Reply With Quote
Old 12-18-2012, 10:31 AM   #355
Weekendmedic
Junior Member
Weekendmedic doesn't litterWeekendmedic doesn't litter
 
Posts: 8
Karma: 110
Join Date: Dec 2012
Location: Upstate NY
Device: Kindle, Android/Moon+
Kiwidude and others, thanks so much for all of your efforts to make Calibre so amazing.

I manage a very large library (hovering just over 62k titles currently), and use the Find Duplicates plugin often. I generally weed my library by using Find Dups->Find Book Dups->Title/Author->Fuzzy->Fuzzy->Show All Groups. Because I generally auto-import large collections, I tend to get about 5000 dup sets per search, with about 10k titles.

My server (dedicated to this application) is a Windows 7 64bit install, 1.5gHz, 2GB ram, HP box with 4TB in 2 logical drives. With little else running on the box other than Calibre, I'm finding it very slow to work through duplicates, and wonder if I can use the plugin better, or if I'm pushing it too hard.

When I get my duplicate search results, I skip to the bottom of the list to find the sets with many duplicates, and select the title I want to merge into, then ctrl-click the others , hit "m" to merge, and then I wait.

I know what the app is doing in the background, it's doing a fair amount of work to merge the titles. My question is - is there a way for the application to make that work happen in the background, so that I can go on an work on the next group? Perhaps queue the merg up to happen a little later (or as the server is available)?

If I can make this go faster, I can work my giant duplicate list down toward zero, and make it manageable for the future. With the long pause after each merge command is sent, it's tough to keep up. Is there a better way?

Thanks!

Last edited by Weekendmedic; 12-18-2012 at 10:38 AM. Reason: More specifics on my install (64bit)
Weekendmedic is offline   Reply With Quote
Old 12-19-2012, 03:55 AM   #356
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,601
Karma: 2092290
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
@WeekendMedic - performance of merging is nothing to do with this plugin. Calibre does not scale well when it comes to performance once you get above 10,000 titles.
kiwidude is offline   Reply With Quote
Old 12-21-2012, 02:30 PM   #357
Weekendmedic
Junior Member
Weekendmedic doesn't litterWeekendmedic doesn't litter
 
Posts: 8
Karma: 110
Join Date: Dec 2012
Location: Upstate NY
Device: Kindle, Android/Moon+
Quote:
Originally Posted by kiwidude View Post
@WeekendMedic - performance of merging is nothing to do with this plugin. Calibre does not scale well when it comes to performance once you get above 10,000 titles.
Kiwidude - thanks for the reply, good to hear that your side of the world is still there (12/21/12 and all).

I understand that the merging performance isn't part of the plugin - is there any way to allow the plugin to move on to the next group while merge toils in the background? I copied 100 books off to a second library with duplicates present, found similar speeds in merging titles in that (little) library as I do in my big one. I realize the application is doing relatively heavy background work when I ask it to merge, just wondering if I have to wait to select the next merging pair (or triplet, etc) until after the first pair have completed.

Thanks!
Weekendmedic is offline   Reply With Quote
Old 12-21-2012, 07:22 PM   #358
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,601
Karma: 2092290
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
As I said, merging is not part of this plugin. It isn't initiated by it, nor does this plugin control in any way whether merging runs in the foreground or background. So there is *nothing* this plugin can do about it.
kiwidude is offline   Reply With Quote
Old 01-01-2013, 09:29 AM   #359
sethcohn
Junior Member
sethcohn began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Jun 2005
Redirected here from https://www.mobileread.com/forums/sho...d.php?t=201256

Reading this thread, it seems folks have asked for 'binary' close before, and that request has been rejected. To be clear, I'm asking for a binary identical EXCEPT for certain files in the book (ie metadata related, like UUID, calibre related, etc)

Looking over the code in find duplicates, seems nontrivial to me but Kovid thinks otherwise. You can't use the entire file to hash, you have to consider the file minus the parts like the metadata and other excluded items, but I'm not a python or Calibre wiz, so not sure how much work this would take.

An example might be good here: 2 files, both converted from the same source material, but done at different times, using identical settings for conversion, but perhaps with different versions of Calibre, will generate files that are _close_ to identical, but fail binary dupe, because of the UUID, the timestamps, the Calibre version.... maybe a Calibre bookmark file, and so on. A function to identify _these_ as duplicate _would_ be useful. If the files were converted using different settings, if one file has split html inside and the other not, that's not identical and should be looked at manually (I agree with past discussions), but in this case (and I've got a lot of these), these files are identical in every way that matters, yet fail the binary test, due to factors I can't control for. Even rebuilding these into new books will continue to fail because the UUIDs and timestamps will continue to remain different. (Even a (re)build of the same book twice in a row as two different books, these should be flaggable as identical, but aren't, due to timestamps in the metadata and thus the hashes are different, even if UUID is the same.)
sethcohn is offline   Reply With Quote
Old 01-01-2013, 03:23 PM   #360
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,601
Karma: 2092290
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
As you have gone to the trouble of reading previous discussions in this thread I won't rehash them in detail here. My opinion on it hasn't changed - it isn't trivial, it isn't generic across formats and it just isn't all that useful in my opinion to justify the effort and how much the plugin would have to be hacked to support it.

There is another plugin called Similar Stories which you could try instead.
kiwidude is offline   Reply With Quote
Reply

Tags
cross library duplicates, in library duplicates

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[GUI Plugin] Quality Check kiwidude Plugins 1171 03-23-2024 05:18 AM
[GUI Plugin] View Manager kiwidude Plugins 413 03-17-2024 12:01 AM
[GUI Plugin] Open With kiwidude Plugins 402 03-16-2024 11:44 PM
[GUI Plugin] Generate Cover kiwidude Plugins 811 03-16-2024 11:31 PM
[GUI Plugin] Plugin Updater **Deprecated** kiwidude Plugins 159 06-19-2011 12:27 PM


All times are GMT -4. The time now is 02:16 PM.


MobileRead.com is a privately owned, operated and funded community.