Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Plugins

Notices

Reply
 
Thread Tools Search this Thread
Old 07-16-2012, 12:33 PM   #271
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,230
Karma: 1345754
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
Quote:
Originally Posted by DMee View Post
Trouble is "Similar" authors produces nothing and "Soundex" or "Fuzzy" produce loads with some errors hidden amongst lots of "cruft" I beleive you call it :-)

e.g. Arthur Charles Clark was buried 10 screens down - is it possible to manually adjust the settings to limit A.Bronte and A.Brown clashing but pick up Arthur C. Clark and Arthur Charles Clark ??
Now that I've had a chance to get back to this plugin there was one thing that stood out for me on this. A.Bronte and A.Brown should *not* have been appearing together. I figured out why they are and will fix it. Its to do with the permutations of flipping author names, and means that A.Bronte and A.Brown both end up in a group with a permutation of "B A" caused by flipping the first/surname around and taking the first "initial". This change should reduce a lot of false positives from the fuzzy search, both for the normal Find Book Duplicates and the new Find Metadata Variations screens.

I have also added the soundex spinbox into the Find Metadata Variations screen to give you granularity on that.

Thanks for the feedback.
kiwidude is offline   Reply With Quote
Old 07-16-2012, 12:55 PM   #272
DMee
Zealot
DMee began at the beginning.
 
Posts: 113
Karma: 10
Join Date: Jun 2011
Location: Stockport, UK
Device: ipad, iphone, kindle
Quote:
Originally Posted by kiwidude View Post
I'll PM chaley - it is something we have discussed previously and I thought had been "fixed" in such a way that I did not have to do a "cleanup" call before I do every query but obviously not. At least I can see why it is happening...
So.... whats the interminable wait every time I OK some edits if not Calibre cleaning up?
DMee is offline   Reply With Quote
Old 07-16-2012, 12:57 PM   #273
DMee
Zealot
DMee began at the beginning.
 
Posts: 113
Karma: 10
Join Date: Jun 2011
Location: Stockport, UK
Device: ipad, iphone, kindle
Thumbs up

Quote:
Originally Posted by kiwidude View Post
Now that I've had a chance to get back to this plugin there was one thing that stood out for me on this. A.Bronte and A.Brown should *not* have been appearing together. I figured out why they are and will fix it. Its to do with the permutations of flipping author names, and means that A.Bronte and A.Brown both end up in a group with a permutation of "B A" caused by flipping the first/surname around and taking the first "initial". This change should reduce a lot of false positives from the fuzzy search, both for the normal Find Book Duplicates and the new Find Metadata Variations screens.

I have also added the soundex spinbox into the Find Metadata Variations screen to give you granularity on that.

Thanks for the feedback.
Not a problem - thanks for the amazing plugins
DMee is offline   Reply With Quote
Old 07-16-2012, 01:17 PM   #274
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,230
Karma: 1345754
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
Quote:
Originally Posted by DMee View Post
So.... whats the interminable wait every time I OK some edits if not Calibre cleaning up?
Any lag you are experiencing there will be calibre copying and renaming directories/books for an author. That is different however to the pruning of the database tables. I've got a workaround of calling db.clean() before I ask for anything with ids (authors, publishers, series or tags) and I've chucked a PM to chaley to see whether he wants to do anything else about it, but at least it doesn't error any more.
kiwidude is offline   Reply With Quote
Old 07-16-2012, 01:33 PM   #275
DMee
Zealot
DMee began at the beginning.
 
Posts: 113
Karma: 10
Join Date: Jun 2011
Location: Stockport, UK
Device: ipad, iphone, kindle
So when would Calibre normally clean up if you didn't run plugins??
DMee is offline   Reply With Quote
Old 07-16-2012, 01:49 PM   #276
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,230
Karma: 1345754
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
Probably at startup I would imagine - which is why you found that restarting calibre allowed the plugin to start working again. Undoubtedly there are other activity points which may trigger it, I haven't bothered looking since it isn't relevant and I have a workaround.
kiwidude is offline   Reply With Quote
Old 07-17-2012, 05:00 AM   #277
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 5,907
Karma: 1217216
Join Date: Jan 2010
Location: France
Device: Many android devices
Quote:
Originally Posted by kiwidude View Post
I'll PM chaley - it is something we have discussed previously and I thought had been "fixed" in such a way that I did not have to do a "cleanup" call before I do every query but obviously not. At least I can see why it is happening...
The fix is now in trunk.
Quote:
Originally Posted by kiwidude View Post
Probably at startup I would imagine - which is why you found that restarting calibre allowed the plugin to start working again. Undoubtedly there are other activity points which may trigger it, I haven't bothered looking since it isn't relevant and I have a workaround.
FWIW: the cleanup happens at exit.
chaley is offline   Reply With Quote
Old 07-17-2012, 03:43 PM   #278
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,230
Karma: 1345754
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
v1.4.0 Released

Changes in this release:
  • Now requires calibre 0.8.59
  • Add a Find metadata variations option to search for author, series, publisher and tag variations, and allow renaming them from the dialog.
  • Fix bug of fuzzy author comparisons which will no longer compute a reverse hash to reduce the false positives it generated

Thanks to those of you who tested and offered feedback on the beta.

@chaley - I appreciate your fix for calibre will be in 0.8.61, I've included a workaround in this plugin if a user is running 0.8.59 or 0.8.60 just so I can release it now.
kiwidude is offline   Reply With Quote
Old 07-20-2012, 09:02 AM   #279
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,230
Karma: 1345754
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
Beta for next version

A number of people over the last year or so have asked for the ability to do cross-library duplicate comparisons. I've finally reached the breaking point of deciding a personal need for using it myself and have put something together.

So on the menu you now have a "Find library duplicates..." option with a screen that looks similar to the regular Find book duplicates one. However you can see you choose a library to compare against up the top, and have way less options at the bottom since UI stuff isn't relevant.

As I have stated previously the only practical approach to working with the results is to simply produce a report. You can copy it to your clipboard / text editor, switch to your other library and manually decide what to do about them. It would be an *enormous* amount of work to wrap a cross-library gui for merging/deleting so it isn't going to happen - we discounted having a special plugin screen when originally designing this plugin and the reasons more than equally apply here.

A couple more things to note:
  • The report only shows the results of comparing books in the current library with books in the target library. It intentionally will not show you duplicates that exist only in the current library, nor duplicates that exist only in the target library. It is for a direct cross-library comparison only.
  • If you do an "Ignore Title" search (i.e. matching only authors) then the report output will differ and show you by author name which authors match and the books associated with that author in the target library
  • If you do a binary search, it shows you which format is the binary duplicate
  • For title/author/isbn searches, it will show you what formats if any exist for that book
  • Unlike "Find book duplicates", it is possible with "Find library duplicates" to do an Ignore Title, Identical Author search. This is a combination which makes no sense when only looking in your own library, but makes perfect sense when comparing across libraries.
I haven't even had a chance to run this on my libraries at home as yet so it may have gremlins. I have run it against some small test libraries and it all seems to do the right thing but any feedback is appreciated.
Attached Thumbnails
Click image for larger version

Name:	CrossLibrary.png
Views:	46
Size:	30.3 KB
ID:	89443  

Last edited by kiwidude; 07-20-2012 at 05:34 PM. Reason: Removed attachment as officially released
kiwidude is offline   Reply With Quote
Old 07-20-2012, 05:35 PM   #280
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,230
Karma: 1345754
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
v1.5.0 Released

Changes in this release:
  • Add a "Find library duplicates" option for cross-library duplicate comparisons into a log report
  • If currently running a duplicate book search and execute a metadata variation search, clear search first

See my comments above for a description of how the cross library duplicates search work. Note I did find a bug in the beta for "Author only" searches which is corrected in the 1.5.0 release (so don't be surprised to see more results now than you saw in the beta!).
kiwidude is offline   Reply With Quote
Old 07-20-2012, 09:55 PM   #281
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 3,981
Karma: 2925589
Join Date: Mar 2012
Location: NSW Australia
Device: none
Win7 Pro SP1|Calibre 0.8.61 - for Library Compare. This will save me a lot of time.

Minor Inconsistency

The Compare With Library drop down does not show the list of Calibre Libraries, as I anticipated it would.

Instead I had to navigate to the corresponding folder via the Browse button, using the standard Windows Open Folder dialogue, which opened up assuming my Calibre Libraries were in my Windows Documents Library. I can imagine this might confuse some people. Calibre has a list of Libraries & Folder Paths behind the Library Quick Switch flyout menu. IMO that list should be used here.

Suggested Enhancement

Save results to a text file - I appreciate I can copy it to the clipboard, open a text editor, paste the clipboard and save the file (or shock horror - print it) - but ...

BR
BetterRed is offline   Reply With Quote
Old 07-21-2012, 04:59 AM   #282
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,230
Karma: 1345754
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
There is a reason behind the library path thing. The way calibre chooses to display library names has some limitations which from time to time the odd user has come on to complain about. For users who only have a small number of libraries I completely agree that a dropdown of names would be consistent and simple. Three issues though. First calibre has a limit to the number of libraries it remembers, and there has been at least one user I have seen come on these forums and complain about that. Secondly I need to cater for the user comparing against a library they may not yet have opened in calibre on this machine. And thirdly what about a user who makes a copy of their library folder onto a different drive for backup and is using this feature to compare. Just displaying the calibre library folder is not enough to differentiate which one they are pointing to.

So I took the easy way out and took the same approach as the switch/create library dialog - that is the part of calibre I have chosen to be "consistent" with.

Your second suggestion of a save button is an excellent one, I will add that in.
kiwidude is offline   Reply With Quote
Old 07-21-2012, 08:58 AM   #283
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,230
Karma: 1345754
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
v1.5.1 Released

Changes in this release:
  • Add a "Save log" button for the "Find library duplicates" result screen.
kiwidude is offline   Reply With Quote
Old 07-21-2012, 09:07 AM   #284
Stampercam
Connoisseur
Stampercam is a splendid one to beholdStampercam is a splendid one to beholdStampercam is a splendid one to beholdStampercam is a splendid one to beholdStampercam is a splendid one to beholdStampercam is a splendid one to beholdStampercam is a splendid one to beholdStampercam is a splendid one to beholdStampercam is a splendid one to beholdStampercam is a splendid one to beholdStampercam is a splendid one to behold
 
Stampercam's Avatar
 
Posts: 82
Karma: 19674
Join Date: Jan 2011
Device: ipad, MiGear
I'm trying to use the find duplicates plugin and am getting the following:

calibre, version 0.8.61
ERROR: Unhandled exception: <b>MemoryError</b>:

Traceback (most recent call last):
File "calibre_plugins.find_duplicates.action", line 187, in toolbar_button_clicked
File "calibre_plugins.find_duplicates.action", line 152, in find_book_duplicates
File "calibre_plugins.find_duplicates.duplicates", line 123, in run_book_duplicates_check
File "calibre_plugins.find_duplicates.book_algorith ms", line 58, in run_duplicate_check
File "calibre_plugins.find_duplicates.book_algorith ms", line 111, in convert_candidates_to_groups
File "calibre_plugins.find_duplicates.book_algorith ms", line 196, in partition_using_exemptions
MemoryError

I'm using win 7
calibre v 0.8.61 and the plugin was just updated to v1.5.1, but I was having the error with the previous ver of the plugin.

Any suggestions?

Cam
Stampercam is offline   Reply With Quote
Old 07-21-2012, 09:17 AM   #285
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,230
Karma: 1345754
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
That part of the plugin hasn't been changed in pretty much forever. Are you running it against some massive library with an obscene amount of duplicates or something?
kiwidude is offline   Reply With Quote
Reply

Tags
cross library duplicates, in library duplicates

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[GUI Plugin] Quality Check kiwidude Plugins 797 12-16-2014 01:01 PM
[GUI Plugin] Open With kiwidude Plugins 251 12-15-2014 01:07 PM
[GUI Plugin] View Manager kiwidude Plugins 91 11-17-2014 01:40 AM
[GUI Plugin] Generate Cover kiwidude Plugins 502 10-10-2014 07:47 AM
[GUI Plugin] Plugin Updater **Deprecated** kiwidude Plugins 159 06-19-2011 01:27 PM


All times are GMT -4. The time now is 12:57 AM.


MobileRead.com is a privately owned, operated and funded community.