Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Plugins

Notices

Reply
 
Thread Tools Search this Thread
Old 04-27-2011, 03:05 PM   #16
drMerry
Addict
drMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmos
 
drMerry's Avatar
 
Posts: 293
Karma: 21022
Join Date: Mar 2011
Location: NL
Device: Sony PRS-650
Quote:
Originally Posted by kiwidude View Post
@drMerry - I think the simplest solution to your issue right now is to do Show all book exemptions, remove all those ones in that group, and instead use a search restriction before you search for duplicates.
Spoiler:


The problem I believe is due to the way exemptions are stored, as every book is being stored as being exempt with every other book. This isn't a scalable approach if (as you have) your group contains a massive number of books.

Right now I will see what others think on the dev thread about how we solve it - either we prevent you marking the group as exempt in the first place by putting in a threshold, or we change the way exemptions are stored. However you have a workaround in the meantime I believe.

In what I would term "normal" usage your exemption groups should not be very big - the 99% scenario I perceive as being 2-3 books/authors in a group. However allowing very fuzzy searches and in your case storing a large number of near duplicate titles as people will have who store magazines etc this situation will arise.
At the moment this is the solution (for first deduplication) but my new post (typing together again )shows some more detail.
drMerry is offline   Reply With Quote
Old 04-28-2011, 03:03 PM   #17
drMerry
Addict
drMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmosdrMerry has become one with the cosmos
 
drMerry's Avatar
 
Posts: 293
Karma: 21022
Join Date: Mar 2011
Location: NL
Device: Sony PRS-650
Another one I was thinking of:

Duplicate in tag.
Something different than the earlier post.
When adding books to my library, I give them a default tag (Newly added).
Calibre has some functions to handle added duplicates, but this is not a complete range. If I add 10 books, I would like to say add / skip / merge for every single book based on existing books and formats.
If I add 1000 books, I want to add them all and sort it later.
In both cases Calibre can't help me. I have to add them all and look for duplicates later.

So it would be nice to have a function like:
Find duplicates one or more books containing tag...

In this way, I can skip all books I had in lib before my new collection arrived, but if one of the new books matches my library, I will see it.
(again filtering does not help in this case because I want to test all books but only want to see results based on a filter)
drMerry is offline   Reply With Quote
 
Enthusiast
Old 04-28-2011, 03:43 PM   #18
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 5,281
Karma: 821512
Join Date: Jan 2010
Location: France
Device: Many android devices
Just a note to say how useful this plugin has turned out to be. I thought I had a clean library. It was clean to 99%, but not totally. The changes to force to ASCII helped me find several cases where an author appeared with and without accented characters. The soundex matches helped find cases of slight author and title misspellings. Having author-only and title-only helped with finding problems with books in both French and English. Working with other combinations found a few other problems. And on top of all that, it is very fast.

Very fine work!
chaley is offline   Reply With Quote
Old 04-28-2011, 05:06 PM   #19
Xenophon
curmudgeon
Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.Xenophon ought to be getting tired of karma fortunes by now.
 
Xenophon's Avatar
 
Posts: 1,453
Karma: 5657812
Join Date: Jun 2006
Location: Pittsburgh, PA USA
Device: Kobo Aura (soon), (soon-to-be-ex)nook, (ex)PRS-700, (ex)PRS-500
I just used the plugin on my library, and promptly turned up a few duplicates -- even though I've made a major effort to keep the library clean.
Xenophon is offline   Reply With Quote
Old 04-29-2011, 05:53 PM   #20
latexsalesman
Enthusiast
latexsalesman doesn't litterlatexsalesman doesn't litter
 
Posts: 36
Karma: 124
Join Date: Nov 2008
Location: Lockport, NY
Device: Pw2 @ Retina Mini
This is seriously one of the best plugins that I have ever used for our wonderful Calibre!!!

kiwidude I have to say how much that I love this plugin...

Bravo!!!!
Thank You!!!
You Rock!!!

Thanks So Much Great Work...



I want to be your LatexSalesMan!!!!
latexsalesman is offline   Reply With Quote
Old 04-29-2011, 07:50 PM   #21
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,224
Karma: 1334002
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
lol... thanks all for the positive comments, much appreciated.
kiwidude is offline   Reply With Quote
Old 04-29-2011, 10:20 PM   #22
zespectre
BookMonkey
zespectre might easily be mistaken for a Texanzespectre might easily be mistaken for a Texanzespectre might easily be mistaken for a Texanzespectre might easily be mistaken for a Texanzespectre might easily be mistaken for a Texanzespectre might easily be mistaken for a Texanzespectre might easily be mistaken for a Texanzespectre might easily be mistaken for a Texanzespectre might easily be mistaken for a Texanzespectre might easily be mistaken for a Texanzespectre might easily be mistaken for a Texan
 
zespectre's Avatar
 
Posts: 90
Karma: 18008
Join Date: Apr 2011
Location: Shenandoah Valley
Device: NOOK, iPod Touch, Asus Netbook
This plug-in just helped me locate three (rather expensive) books I though had gone missing. Somehow, somewhere the names became corrupted but they turned up in one of the "fuzzy" searches. I fixed the metadata and am now a very happy camper!
zespectre is offline   Reply With Quote
Old 04-30-2011, 05:00 AM   #23
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,224
Karma: 1334002
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
That sounds a rather unusual side effect benefit of using the plugin! Glad it helped you out, I hate "losing" things too.
kiwidude is offline   Reply With Quote
Old 04-30-2011, 09:52 AM   #24
mayoi
Member
mayoi writes the songs that make the whole world sing.mayoi writes the songs that make the whole world sing.mayoi writes the songs that make the whole world sing.mayoi writes the songs that make the whole world sing.mayoi writes the songs that make the whole world sing.mayoi writes the songs that make the whole world sing.mayoi writes the songs that make the whole world sing.mayoi writes the songs that make the whole world sing.mayoi writes the songs that make the whole world sing.mayoi writes the songs that make the whole world sing.mayoi writes the songs that make the whole world sing.
 
mayoi's Avatar
 
Posts: 127
Karma: 40426
Join Date: Jan 2011
Location: USA
Device: PW2 and K3 Kindles, TF101
Works beautifully and fast. Karma to you!
mayoi is offline   Reply With Quote
Old 05-03-2011, 06:41 AM   #25
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,224
Karma: 1334002
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
v1.1 Released

Changes in this release:
  • Add support for binary comparison searches to find book formats with exactly the same content
  • Replace how exemptions are stored in the config file to make more scalable
  • No longer calculate exemption preview detailed messages for the confirmation dialog for performance
  • Compare multiple authors for most author algorithms to increase duplicate coverage.
  • Change Manage exemptions dialog to have tab for each author with exemptions and show section only if have exemptions
  • Include swapping author name order in all but identical author checks. So A B / B A or A,B / B,A will match.
  • Disable the Ignore title, identical author combination as will not a valid one (never duplicates)
  • Allow the remove, mark current and mark all group exemption dialogs able to be hidden from showing again.
  • Allow various count of result and no result information dialogs able to be hidden from showing again.
  • Allow user to reset confirmation dialogs related to find duplicates from the configuration dialog

Important:
I have changed the way your exemptions are stored in the config file, to make this plugin work more efficiently with very large groups of exemptions. You will lose any book or author exemptions when you upgrade to this version. However this is a one-off problem for longer term benefit.

A key change of particular note is the way authors are compared. Previously this plugin only considered the first author of a book if it had multiple. So a book by "Lee Child & David Gemmell" could not match one by "David Gemmell" without using an ignore author search. In addition the plugin will now catch cases of the author names being stored in the wrong order. Previously it would compare "Lee Child" with "Child, Lee". Now it will also match books where the metadata is incorrect like "Lee, Child" and "Child Lee".

With these changes you should now be able to catch more duplicate situations in your library. Special thanks again to chaley for his suggestions and testing with this release.
kiwidude is offline   Reply With Quote
Old 05-05-2011, 07:46 AM   #26
butterbescotch
Enthusiast
butterbescotch began at the beginning.
 
Posts: 36
Karma: 10
Join Date: Mar 2011
Device: none
Thanks for the plug in!

ED: It's now fixed.

ED: I've got 3685 duplicated titles!

Last edited by butterbescotch; 05-05-2011 at 07:52 AM.
butterbescotch is offline   Reply With Quote
Old 05-05-2011, 07:51 AM   #27
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,224
Karma: 1334002
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
You are welcome - and I'm not sure what is now "fixed" but I am glad it is...
kiwidude is offline   Reply With Quote
Old 05-05-2011, 09:37 AM   #28
theducks
Grand Sorcerer
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 14,518
Karma: 5567081
Join Date: Aug 2009
Location: The (original) Silicon Valley, USA
Device: Galaxy Tab 2, Astak Pocket Pro, K4NT
Quote:
Originally Posted by butterbescotch View Post

ED: I've got 3685 duplicated titles!
That is more duplicates than I have books in my Library
theducks is offline   Reply With Quote
Old 05-06-2011, 10:01 AM   #29
Starson17
Wizard
Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.Starson17 can program the VCR without an owner's manual.
 
Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
Great work kiwidude. I'd written some duplicate finder code and thought my library was in pretty good shape, but I found a variety of book and author duplicates with the new plugin. Kudos to you!
Starson17 is offline   Reply With Quote
Old 05-06-2011, 10:40 AM   #30
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,224
Karma: 1334002
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
Thanks Starson17, that is very cool to hear. After all it was initially your interest in humoring my ramblings before chaley joined in that made me think it was worthwhile attempting it.

The only idea I can recall being suggested for this that is outstanding is the ability to filter the result groups. I don't want to add it unless more than one person requests it though - it isn't something I would use myself but I understand drMerry's point that there are circumstances that applying a search restriction before you search will not work as the alternative.
kiwidude is offline   Reply With Quote
Reply

Tags
cross library duplicates, in library duplicates

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[GUI Plugin] Generate Cover kiwidude Plugins 489 08-15-2014 09:39 AM
[GUI Plugin] Quality Check kiwidude Plugins 738 08-02-2014 10:06 PM
[GUI Plugin] View Manager kiwidude Plugins 82 08-01-2014 12:37 PM
[GUI Plugin] Open With kiwidude Plugins 228 07-31-2014 01:06 AM
[GUI Plugin] Plugin Updater **Deprecated** kiwidude Plugins 159 06-19-2011 12:27 PM


All times are GMT -4. The time now is 07:27 AM.


MobileRead.com is a privately owned, operated and funded community.