Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Plugins

Notices

Reply
 
Thread Tools Search this Thread
Old 10-20-2012, 05:55 AM   #331
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,617
Karma: 2124234
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
@rotanal - changing a binary comparison to not actually be a binary comparison will not *ever* happen with this plugin.

I really don't see a use case for some kind of fuzzy comparison. You already have all the fuzzy comparisons required based on *metadata* to identify whether two books are duplicates. The only reason for doing a binary comparison is to most quickly be able to delete the corresponding duplicate with 100% assurance that you are not accidentally losing something.

Scanning say two epubs and then deciding that based on their content they are "mostly similar" tells you nothing useful (and bear in mind that is just an easier to compare format, let alone all the others). The one exception being if you have screwed up your library metadata and given the book a title/author which it isn't. But that is such a niche case it isn't remotely worth the enormous effort to cater for.

You can find duplicates using the existing metadata based functions. Having found those duplicates, deciding which ones to merge is another matter. Again if you think that "a few bytes" difference means one can safely and automatically deleted as being "almost binary" you are mistaken. As ilovejedd points out those "minor" differences could be the difference between a corrupt and a non corrupt book. Or one that is formatted to your liking versus one that is not. Or one that has been proofed for errors versus one that has not. Or a later edition. Or a different cover. Or one which has encoding a set correctly to make readable, etc, etc. You can't reliably automate those evaluations to determine what is the best version to keep. You have to open them up side by side and decide based on your own personal criteria.

As I have had said repeatedly from the time I created this plugin ages ago, there is a space for someone to write a separate smart merge plugin. To allow a user who having been given the duplicate results from this plugin to make better informed merge decisions to help with deciding which to keep. For instance if two epubs are being merged, it could examine the zip files and compare contents files to tell you which differ etc. But such a plugin does not exist and I have no personal interest in writing it as I have no need for it.
kiwidude is offline   Reply With Quote
Old 10-20-2012, 12:34 PM   #332
ilovejedd
hopeless n00b
ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.
 
ilovejedd's Avatar
 
Posts: 5,111
Karma: 19597086
Join Date: Jan 2009
Location: in the middle of nowhere
Device: PW4, PW3, Libra H2O, iPad 10.5, iPad 11, iPad 12.9
Feature request:
Would it be possible to add searching for duplicates based on other identifiers aside from the ISBN? In particular, uri/url? I've got a lot of fanfics in my library and I've encountered cases where both the title and author's pen name for a fanfic have changed while the fanfic url (ergo identifier) remains the same.

Thanks!
ilovejedd is offline   Reply With Quote
Old 10-26-2012, 06:37 AM   #333
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,617
Karma: 2124234
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
@ilovejdd - apologies for the delay but Sigil development has vamped nearly every waking hour for months. To answer your question - that sounds a good suggestion to me. Change "ISBN Compare" to "Identifier" with a dropdown next to it for the user to choose which type from.
kiwidude is offline   Reply With Quote
Old 10-26-2012, 07:45 AM   #334
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,617
Karma: 2124234
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
Beta for v1.6.0

This version lets you find duplicates using any identifiers rather than only ISBN. The list of identifiers comes from whatever identifiers exist in the books you have in your library.

As per usual let me know if any issues found before I officially release it...

Last edited by kiwidude; 10-29-2012 at 04:38 PM. Reason: Removed attachment as officially released
kiwidude is offline   Reply With Quote
Old 10-29-2012, 04:39 PM   #335
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,617
Karma: 2124234
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
v1.6.0 Released

Changes in this release:
  • Change "ISBN Compare" to "Identifier" with a dropdown allowing comparison of any identifier field.
  • Add a context menu to the metadata variations list to allow choosing the selected name on the right side.

To the few of you that tried the beta above, please just force a refresh to pickup the latest addition as the version nubmer is the same but I did make some additions.
kiwidude is offline   Reply With Quote
Old 10-30-2012, 03:25 AM   #336
alexd
Junior Member
alexd began at the beginning.
 
Posts: 2
Karma: 10
Join Date: Oct 2012
Device: kindle touch
Thanks for this wonderful plugin!

And I have a question:
Would it be possible to implement a duplicate search for 2 libraries?
With a source and a destination library and the duplicates will be marked in the source library. Cause I would like to compare 2 libraries without merging them.
alexd is offline   Reply With Quote
Old 10-30-2012, 04:14 AM   #337
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,617
Karma: 2124234
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
@alexd - welcome to MobileRead.

You can already to that with this plugin, choose the second option down of "Find Library Duplicates..."
kiwidude is offline   Reply With Quote
Old 10-30-2012, 07:24 AM   #338
alexd
Junior Member
alexd began at the beginning.
 
Posts: 2
Karma: 10
Join Date: Oct 2012
Device: kindle touch
THANK YOU for the hint!
I use the plugin for ages, but always with the icon only!
*shame on me*
alexd is offline   Reply With Quote
Old 11-10-2012, 12:21 PM   #339
JohnnyBook
Groupie
JohnnyBook holds these truths to be self-evident.JohnnyBook holds these truths to be self-evident.JohnnyBook holds these truths to be self-evident.JohnnyBook holds these truths to be self-evident.JohnnyBook holds these truths to be self-evident.JohnnyBook holds these truths to be self-evident.JohnnyBook holds these truths to be self-evident.JohnnyBook holds these truths to be self-evident.JohnnyBook holds these truths to be self-evident.JohnnyBook holds these truths to be self-evident.JohnnyBook holds these truths to be self-evident.
 
Posts: 174
Karma: 126824
Join Date: Dec 2008
Location: Out There
Device: K3 W/3G (Fixed screen!) & Paperwhite Wifi
Has something changed reticently with the plugin?

I use find duplicates (Binary compare) all the time. (every time I add more books, most times I do not find any. But with the Baen Books Monthly Bundles, sometimes they repeat books, so I find a few)

In any case, previously when I did a search (2-3 months ago?) it would take a few seconds to do a search (5-10? seconds maybe a little more? But it was not an excessive length of time)

But lately it will take 2-3 minutes!! to do the search, and I still have roughly the same number of books in my library.

AFAIK

I have only done a couple things reticently

I added a bunch of PDFs to my library awhile ago (maybe 200-300 books?) Would PDFs cause a slow down?

The program used to take a couple of minutes to start up, until I saw a post about speeding up program starting times, that suggested not including formats in the tag browser. (Now it takes about 10 seconds to start up!!) Could the plugin now need to do whatever the tag browser used to do before it can complete the search?


Thanks,
JohnnyBook is offline   Reply With Quote
Old 11-10-2012, 12:38 PM   #340
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 29,703
Karma: 54369092
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
Any format should not affect startup time

UNLESS
you have a weird custom column that requires FILE level access.
How many Column Coloring rules do you have? (That has a BIG impact on my startup time)

BTW did you change/version update your Anti-virus program?
theducks is offline   Reply With Quote
Old 11-11-2012, 12:31 AM   #341
JohnnyBook
Groupie
JohnnyBook holds these truths to be self-evident.JohnnyBook holds these truths to be self-evident.JohnnyBook holds these truths to be self-evident.JohnnyBook holds these truths to be self-evident.JohnnyBook holds these truths to be self-evident.JohnnyBook holds these truths to be self-evident.JohnnyBook holds these truths to be self-evident.JohnnyBook holds these truths to be self-evident.JohnnyBook holds these truths to be self-evident.JohnnyBook holds these truths to be self-evident.JohnnyBook holds these truths to be self-evident.
 
Posts: 174
Karma: 126824
Join Date: Dec 2008
Location: Out There
Device: K3 W/3G (Fixed screen!) & Paperwhite Wifi
Hmm I think you miss read my post.

Calibre used to take a couple of minutes to start up, until I saw a post about speeding up program starting times, that suggested NOT including formats in the tag browser. (Now it takes about 10 seconds to start up!!)


(I don't use any Column Coloring rules)

The problem I have is the Duplicate's plugin, 2-3 months ago it would take a few seconds to do a search (5-10? seconds maybe a little more? But it was not an excessive length of time)

Bit now it will take 2-3 minutes!! to do the search, and I still have roughly the same number of books in my library.


My theory was maybe the plugin now needs to do (whatever the tag browser was doing with formats) before it can complete the search?


My Antivirus is Norton. I have no clue when it updates.
JohnnyBook is offline   Reply With Quote
Old 11-11-2012, 08:58 AM   #342
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 29,703
Karma: 54369092
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
Quote:
Originally Posted by JohnnyBook View Post
Hmm I think you miss read my post.

Calibre used to take a couple of minutes to start up, until I saw a post about speeding up program starting times, that suggested NOT including formats in the tag browser. (Now it takes about 10 seconds to start up!!)


(I don't use any Column Coloring rules)

The problem I have is the Duplicate's plugin, 2-3 months ago it would take a few seconds to do a search (5-10? seconds maybe a little more? But it was not an excessive length of time)

Bit now it will take 2-3 minutes!! to do the search, and I still have roughly the same number of books in my library.


My theory was maybe the plugin now needs to do (whatever the tag browser was doing with formats) before it can complete the search?


My Antivirus is Norton. I have no clue when it updates.
The reason having the tag browser (or covers,or details) open affects the startup time, is those require additional Database and/or cover file access cycles before the GUI can be populated. (Calibre does not have a 'Always start with the X, Y, Z, shown ' option, but opens with the 'last known' display (initial Install is 'All)'.

IIRC User plugins just get initialized (they don't process the DB/files until invoked). Note: many plugins are evolving. There are basic speed-ups , then more/better checks tend to use up that performance gain


'Yellow box' products used to default to Hourly update checks
(Actual updates only happened when found)

Out of the chute, their default settings were always too aggressive (cpu/file system abusive) for my tastes
My systems are dialed way back, as I can usually smell a scam link (I watch the actual URL that is displayed in the status line in Firefox). My wife tends to click first , so I have hers set closer to defaults. YMMV use settings that best reflect your awareness.


BTW the duplicates plugin has settings that will affect times.
When I use Similar/Similar, I see the checks complete in seconds (4K+ books).
If you do a Binary search, that possibly requires that each FILE be opened (Speaking of reading Files: Defrag , when was it last done?) to calculate the values.
theducks is offline   Reply With Quote
Old 11-12-2012, 01:31 PM   #343
country0129
Country0129
country0129 ought to be getting tired of karma fortunes by now.country0129 ought to be getting tired of karma fortunes by now.country0129 ought to be getting tired of karma fortunes by now.country0129 ought to be getting tired of karma fortunes by now.country0129 ought to be getting tired of karma fortunes by now.country0129 ought to be getting tired of karma fortunes by now.country0129 ought to be getting tired of karma fortunes by now.country0129 ought to be getting tired of karma fortunes by now.country0129 ought to be getting tired of karma fortunes by now.country0129 ought to be getting tired of karma fortunes by now.country0129 ought to be getting tired of karma fortunes by now.
 
Posts: 55
Karma: 506306
Join Date: Apr 2012
Location: Louisiana
Device: Kindle, Kindle Fire, PC
Icon Missing On Toolbar

Something weird! I've been using "Find Duplicates" plugin for two years. Since last updating Calibre versions and any plugin updates (always regularly do this,) my "Find Duplicate" icon is missing from the toolbar.

Checking for installed plugins, it's still an active plugin, but I have no way to access its properties of which I know.

Any ideas?
country0129 is offline   Reply With Quote
Old 11-12-2012, 02:22 PM   #344
PeterT
Grand Sorcerer
PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.
 
PeterT's Avatar
 
Posts: 12,124
Karma: 73448616
Join Date: Nov 2007
Location: Toronto
Device: Nexus 7, Clara, Touch, Tolino EPOS
Maybe just head over to Preferences | Toolbar and select either Main Toolbar or Main Toolbar when a device is connecterd (or both) and add Find Duplicates to the icons displayed,
PeterT is offline   Reply With Quote
Old 12-02-2012, 06:06 PM   #345
luciaisacat
Enthusiast
luciaisacat began at the beginning.
 
luciaisacat's Avatar
 
Posts: 32
Karma: 10
Join Date: May 2012
Device: android
I installed v 1.6 and now the plugin fails to find duplicates. Anyone else is experiencing this problem? Is there an archive for old plugin? Thanks

Lucia
luciaisacat is offline   Reply With Quote
Reply

Tags
cross library duplicates, in library duplicates

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[GUI Plugin] Open With kiwidude Plugins 403 Yesterday 08:39 AM
[GUI Plugin] Quality Check kiwidude Plugins 1171 03-23-2024 05:18 AM
[GUI Plugin] View Manager kiwidude Plugins 413 03-17-2024 12:01 AM
[GUI Plugin] Generate Cover kiwidude Plugins 811 03-16-2024 11:31 PM
[GUI Plugin] Plugin Updater **Deprecated** kiwidude Plugins 159 06-19-2011 12:27 PM


All times are GMT -4. The time now is 08:05 AM.


MobileRead.com is a privately owned, operated and funded community.