[GUI Plugin] Find Duplicates - Page 2

drMerry · 04-27-2011, 04:05 PM

Quote:

Originally Posted by kiwidude

@drMerry - I think the simplest solution to your issue right now is to do Show all book exemptions, remove all those ones in that group, and instead use a search restriction before you search for duplicates.

Spoiler:

At the moment this is the solution (for first deduplication) but my new post (typing together again

)shows some more detail.

drMerry · 04-28-2011, 04:03 PM

Another one I was thinking of:

Duplicate in tag.
Something different than the earlier post.
When adding books to my library, I give them a default tag (Newly added).
Calibre has some functions to handle added duplicates, but this is not a complete range. If I add 10 books, I would like to say add / skip / merge for every single book based on existing books and formats.
If I add 1000 books, I want to add them all and sort it later.
In both cases Calibre can't help me. I have to add them all and look for duplicates later.

So it would be nice to have a function like:
Find duplicates one or more books containing tag...

In this way, I can skip all books I had in lib before my new collection arrived, but if one of the new books matches my library, I will see it.
(again filtering does not help in this case because I want to test all books but only want to see results based on a filter)

chaley · 04-28-2011, 04:43 PM

Just a note to say how useful this plugin has turned out to be. I thought I had a clean library. It was clean to 99%, but not totally. The changes to force to ASCII helped me find several cases where an author appeared with and without accented characters. The soundex matches helped find cases of slight author and title misspellings. Having author-only and title-only helped with finding problems with books in both French and English. Working with other combinations found a few other problems. And on top of all that, it is very fast.

Very fine work!

Xenophon · 04-28-2011, 06:06 PM

I just used the plugin on my library, and promptly turned up a few duplicates -- even though I've made a major effort to keep the library clean.

latexsalesman · 04-29-2011, 06:53 PM

This is seriously one of the best plugins that I have ever used for our wonderful Calibre!!!

kiwidude I have to say how much that I love this plugin...

Bravo!!!!
Thank You!!!
You Rock!!!

Thanks So Much Great Work...

I want to be your LatexSalesMan!!!!

kiwidude · 04-29-2011, 08:50 PM

lol... thanks all for the positive comments, much appreciated.

zespectre · 04-29-2011, 11:20 PM

This plug-in just helped me locate three (rather expensive) books I though had gone missing. Somehow, somewhere the names became corrupted but they turned up in one of the "fuzzy" searches. I fixed the metadata and am now a very happy camper!

kiwidude · 04-30-2011, 06:00 AM

That sounds a rather unusual side effect benefit of using the plugin! Glad it helped you out, I hate "losing" things too.

mayoi · 04-30-2011, 10:52 AM

Works beautifully and fast. Karma to you!

kiwidude · 05-03-2011, 07:41 AM

Changes in this release:

Add support for binary comparison searches to find book formats with exactly the same content
Replace how exemptions are stored in the config file to make more scalable
No longer calculate exemption preview detailed messages for the confirmation dialog for performance
Compare multiple authors for most author algorithms to increase duplicate coverage.
Change Manage exemptions dialog to have tab for each author with exemptions and show section only if have exemptions
Include swapping author name order in all but identical author checks. So A B / B A or A,B / B,A will match.
Disable the Ignore title, identical author combination as will not a valid one (never duplicates)
Allow the remove, mark current and mark all group exemption dialogs able to be hidden from showing again.
Allow various count of result and no result information dialogs able to be hidden from showing again.
Allow user to reset confirmation dialogs related to find duplicates from the configuration dialog

Important:
I have changed the way your exemptions are stored in the config file, to make this plugin work more efficiently with very large groups of exemptions. You will lose any book or author exemptions when you upgrade to this version. However this is a one-off problem for longer term benefit.

A key change of particular note is the way authors are compared. Previously this plugin only considered the first author of a book if it had multiple. So a book by "Lee Child & David Gemmell" could not match one by "David Gemmell" without using an ignore author search. In addition the plugin will now catch cases of the author names being stored in the wrong order. Previously it would compare "Lee Child" with "Child, Lee". Now it will also match books where the metadata is incorrect like "Lee, Child" and "Child Lee".

With these changes you should now be able to catch more duplicate situations in your library. Special thanks again to chaley for his suggestions and testing with this release.

butterbescotch · 05-05-2011, 08:46 AM

Thanks for the plug in!

ED: It's now fixed.

ED: I've got 3685 duplicated titles!

kiwidude · 05-05-2011, 08:51 AM

You are welcome - and I'm not sure what is now "fixed" but I am glad it is...

theducks · 05-05-2011, 10:37 AM

Quote:

Originally Posted by butterbescotch

ED: I've got 3685 duplicated titles!

That is more duplicates than I have books in my Library

Starson17 · 05-06-2011, 11:01 AM

Great work kiwidude. I'd written some duplicate finder code and thought my library was in pretty good shape, but I found a variety of book and author duplicates with the new plugin. Kudos to you!

kiwidude · 05-06-2011, 11:40 AM

Thanks Starson17, that is very cool to hear. After all it was initially your interest in humoring my ramblings before chaley joined in that made me think it was worthwhile attempting it.

The only idea I can recall being suggested for this that is outstanding is the ability to filter the result groups. I don't want to add it unless more than one person requests it though - it isn't something I would use myself but I understand drMerry's point that there are circumstances that applying a search restriction before you search will not work as the alternative.

05-03-2011, 07:41 AM	#25
kiwidude Calibre Plugins Developer Posts: 4,684 Karma: 2162246 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	v1.1 Released Changes in this release: Add support for binary comparison searches to find book formats with exactly the same content Replace how exemptions are stored in the config file to make more scalable No longer calculate exemption preview detailed messages for the confirmation dialog for performance Compare multiple authors for most author algorithms to increase duplicate coverage. Change Manage exemptions dialog to have tab for each author with exemptions and show section only if have exemptions Include swapping author name order in all but identical author checks. So A B / B A or A,B / B,A will match. Disable the Ignore title, identical author combination as will not a valid one (never duplicates) Allow the remove, mark current and mark all group exemption dialogs able to be hidden from showing again. Allow various count of result and no result information dialogs able to be hidden from showing again. Allow user to reset confirmation dialogs related to find duplicates from the configuration dialog Important: I have changed the way your exemptions are stored in the config file, to make this plugin work more efficiently with very large groups of exemptions. You will lose any book or author exemptions when you upgrade to this version. However this is a one-off problem for longer term benefit. A key change of particular note is the way authors are compared. Previously this plugin only considered the first author of a book if it had multiple. So a book by "Lee Child & David Gemmell" could not match one by "David Gemmell" without using an ignore author search. In addition the plugin will now catch cases of the author names being stored in the wrong order. Previously it would compare "Lee Child" with "Child, Lee". Now it will also match books where the metadata is incorrect like "Lee, Child" and "Child Lee". With these changes you should now be able to catch more duplicate situations in your library. Special thanks again to chaley for his suggestions and testing with this release.

05-05-2011, 08:46 AM	#26
butterbescotch Enthusiast Posts: 36 Karma: 10 Join Date: Mar 2011 Device: none	Thanks for the plug in! ED: It's now fixed. ED: I've got 3685 duplicated titles! Last edited by butterbescotch; 05-05-2011 at 08:52 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[GUI Plugin] Quality Check	kiwidude	Plugins	1206	Today 08:38 PM
[GUI Plugin] Generate Cover	kiwidude	Plugins	833	09-13-2024 12:42 PM
[GUI Plugin] View Manager	kiwidude	Plugins	415	05-11-2024 04:28 AM
[GUI Plugin] Open With	kiwidude	Plugins	403	04-01-2024 09:39 AM
[GUI Plugin] Plugin Updater Deprecated	kiwidude	Plugins	159	06-19-2011 01:27 PM

04-28-2011, 04:03 PM	#17
drMerry Addict Posts: 293 Karma: 21022 Join Date: Mar 2011 Location: NL Device: Sony PRS-650	Another one I was thinking of: Duplicate in tag. Something different than the earlier post. When adding books to my library, I give them a default tag (Newly added). Calibre has some functions to handle added duplicates, but this is not a complete range. If I add 10 books, I would like to say add / skip / merge for every single book based on existing books and formats. If I add 1000 books, I want to add them all and sort it later. In both cases Calibre can't help me. I have to add them all and look for duplicates later. So it would be nice to have a function like: Find duplicates one or more books containing tag... In this way, I can skip all books I had in lib before my new collection arrived, but if one of the new books matches my library, I will see it. (again filtering does not help in this case because I want to test all books but only want to see results based on a filter)

04-28-2011, 04:43 PM	#18
chaley Grand Sorcerer Posts: 12,029 Karma: 7257323 Join Date: Jan 2010 Location: Notts, England Device: Kobo Libra 2	Just a note to say how useful this plugin has turned out to be. I thought I had a clean library. It was clean to 99%, but not totally. The changes to force to ASCII helped me find several cases where an author appeared with and without accented characters. The soundex matches helped find cases of slight author and title misspellings. Having author-only and title-only helped with finding problems with books in both French and English. Working with other combinations found a few other problems. And on top of all that, it is very fast. Very fine work!

04-28-2011, 06:06 PM	#19
Xenophon curmudgeon Posts: 1,487 Karma: 5748190 Join Date: Jun 2006 Location: Redwood City, CA USA Device: Kobo Aura HD, (ex)nook, (ex)PRS-700, (ex)PRS-500	I just used the plugin on my library, and promptly turned up a few duplicates -- even though I've made a major effort to keep the library clean.

04-29-2011, 06:53 PM	#20
latexsalesman Enthusiast Posts: 39 Karma: 124 Join Date: Nov 2008 Location: Lockport, NY Device: Pw2 @ Retina Mini	This is seriously one of the best plugins that I have ever used for our wonderful Calibre!!! kiwidude I have to say how much that I love this plugin... Bravo!!!! Thank You!!! You Rock!!! Thanks So Much Great Work... I want to be your LatexSalesMan!!!!

04-29-2011, 08:50 PM	#21
kiwidude Calibre Plugins Developer Posts: 4,684 Karma: 2162246 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	lol... thanks all for the positive comments, much appreciated.

04-29-2011, 11:20 PM	#22
zespectre BookMonkey Posts: 90 Karma: 18008 Join Date: Apr 2011 Location: Shenandoah Valley Device: NOOK, iPod Touch, Asus Netbook	This plug-in just helped me locate three (rather expensive) books I though had gone missing. Somehow, somewhere the names became corrupted but they turned up in one of the "fuzzy" searches. I fixed the metadata and am now a very happy camper!

04-30-2011, 06:00 AM	#23
kiwidude Calibre Plugins Developer Posts: 4,684 Karma: 2162246 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	That sounds a rather unusual side effect benefit of using the plugin! Glad it helped you out, I hate "losing" things too.

04-30-2011, 10:52 AM	#24
mayoi Member Posts: 127 Karma: 40426 Join Date: Jan 2011 Location: USA Device: PW2 and K3 Kindles, TF101	Works beautifully and fast. Karma to you!

05-05-2011, 08:51 AM	#27
kiwidude Calibre Plugins Developer Posts: 4,684 Karma: 2162246 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	You are welcome - and I'm not sure what is now "fixed" but I am glad it is...

05-06-2011, 11:01 AM	#29
Starson17 Wizard Posts: 4,004 Karma: 177841 Join Date: Dec 2009 Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T	Great work kiwidude. I'd written some duplicate finder code and thought my library was in pretty good shape, but I found a variety of book and author duplicates with the new plugin. Kudos to you!

Advert

Advert

05-06-2011, 11:40 AM	#30
kiwidude Calibre Plugins Developer Posts: 4,684 Karma: 2162246 Join Date: Oct 2010 Location: Australia Device: Kindle Oasis	Thanks Starson17, that is very cool to hear. After all it was initially your interest in humoring my ramblings before chaley joined in that made me think it was worthwhile attempting it. The only idea I can recall being suggested for this that is outstanding is the ability to filter the result groups. I don't want to add it unless more than one person requests it though - it isn't something I would use myself but I understand drMerry's point that there are circumstances that applying a search restriction before you search will not work as the alternative.