[GUI Plugin] Find Duplicates - Page 36

laly003 · 04-27-2015, 11:15 AM

Well, not working at all for me, like I mentioned, I re-imported an existing book..therefore same title/author/metadata/file, etc and it did not recognized it as a dup using binary compare...using title/author I am finding over 70 dups. I like using binary compare because I don't need to clean up the titles/authors first. This was working great last time I used it but that was last year, maybe summer time... Wish I had not upgraded versions

laly003 · 04-28-2015, 11:29 AM

Update: Disregard my post about binary compare not working. I have multiple Calibre libraries setup and I tried the search in different library and it worked so must be something wrong or odd about the library where it's not working...

mbovenka · 05-05-2015, 07:05 AM

I think I am hitting some corner case where the Soundex algorithm starts eating memory. When I use Soundex match on my Titles, the Calibre process starts using more and more RAM (from a base value of about 900 MB) until it hits the 32-bit process limit and goes kaboom (with the portable version) or starts swapping like mad after filling my 8 GB RAM and completely locking my PC (with the 64-bit version). With the Fuzzy algorithm it uses no more than 1 GB (32-bit)or 1.5 GB (64-bit).

The debug log of the 32-bit version is here:

Spoiler:

If any more info is needed, let me know.

JSWolf · 05-07-2015, 06:43 AM

Quote:

Originally Posted by mbovenka

I think I am hitting some corner case where the Soundex algorithm starts eating memory. When I use Soundex match on my Titles, the Calibre process starts using more and more RAM (from a base value of about 900 MB) until it hits the 32-bit process limit and goes kaboom (with the portable version) or starts swapping like mad after filling my 8 GB RAM and completely locking my PC (with the 64-bit version). With the Fuzzy algorithm it uses no more than 1 GB (32-bit)or 1.5 GB (64-bit).

The debug log of the 32-bit version is here:

Spoiler:

If any more info is needed, let me know.

What you need to do is stop running the 32-bit portable version and install the 64-bit version. The 32-bit version cannot access all of your 8GB of ram. You need the 64-bit version so it can access more ram.

mbovenka · 05-07-2015, 02:14 PM

Quote:

Originally Posted by JSWolf

What you need to do is stop running the 32-bit portable version and install the 64-bit version. The 32-bit version cannot access all of your 8Gto swAB of ram. You need the 64-bit version so it can access more ram.

Jon, I am running the 64-bit version. All that does is eat my RAM and start to swap itself (and my PC) to death, as I mentioned in my earlier post. I tested with the 32-bit version to avoid that.

truth1ness · 05-10-2015, 08:58 PM

Great plugin.

Is there any way to tell it to ignore a certain Tag? Fuzzy search is always catching my News downloads because the magazines are almost all identically named and it really clutters up the duplicates page. It'd be great if I could ignore everything tagged "news".

BetterRed · 05-10-2015, 11:48 PM

Quote:

Originally Posted by truth1ness

It'd be great if I could ignore everything tagged "news".

Try putting the non-news into a separate VL -- not tags:"=news" -- and run Find Dups from that VL

BR

LadyKate · 05-11-2015, 05:48 PM

I'm wondering if there is a way to have find duplicates run as if newly installed again.

The reason for this is I have several titles (about 85 or so) which were flagged as duplicates using title/author search but have had the title corrected. They are still marked as duplicate in the duplicate exemption list.

While some are valid duplicate titles (two editions of a book or a different translator) others were in error (such as the name of the series being entered originally instead of the name of the book).

Just curious about this.

BetterRed · 05-11-2015, 07:53 PM

@LadyKate - I would do this

Click Show all book duplicate exemptions from the Find Duplicates drop-down menu
Select all books in the Book List(ctrl/a)
Click Remove selected exemptions from the Find Duplicates drop-down menu
Run Find Duplicates again and redo the exemptions - hopefully far fewer

Watch out for this ==>> Some plugins, can change calibre's search highlighting

BR

LadyKate · 07-24-2015, 06:40 PM

A binary search of the library takes a long time. Would a CRC check be faster?

I didn't see the option for a CRC vs the binary check. I'm not sure if it would be significantly faster or not.

LadyKate · 07-24-2015, 06:48 PM

Quote:

Originally Posted by BetterRed

@LadyKate - I would do this

Click Show all book duplicate exemptions from the Find Duplicates drop-down menu
Select all books in the Book List(ctrl/a)
Click Remove selected exemptions from the Find Duplicates drop-down menu
Run Find Duplicates again and redo the exemptions - hopefully far fewer

Watch out for this ==>> Some plugins, can change calibre's search highlighting

BR

Thank you. I should have thought of that

I will do that because I prefer my duplicates exceptions to be those that are really the same title but with differences.

I tried this but could only find a way to remove the duplicate marking on a case by case basis.

One case where two books(Legend of the Heart Eaters and Menace of the Night Creatures both books by M. E. Holley in the Jonah and the Last Great Dragon Series) resulted in the message calibre, version 2.28.0
ERROR: Unhandled exception: <b>TooBigError</b>:TooBigError: string or blob too big

calibre 2.28 [64bit] isfrozen: True is64bit: True
Windows-8-6.2.9200 Windows ('64bit', 'WindowsPE')
('Windows', '8', '6.2.9200')
Python 2.7.9
Windows: ('8', '6.2.9200', '', 'Multiprocessor Free')
Successfully initialized third party plugins: DeDRM && EpubMerge && User Category && Modify ePub && Quality Check && View Manager && FictionDB && Find Duplicates && FanFictionDownLoader && EpubSplit && Similar Stories && Quick Preferences && Barnes & Noble && Beam Ebooks && Generate Cover && Goodreads && Count Pages && KindleUnpack - The Plugin && Clipboard Search && Save To Format && Webscription && Fantastic Fiction && Goodreads Sync && Resize Cover && Manage Series && Fantastic Fiction Adults
Traceback (most recent call last):
File "calibre_plugins.find_duplicates.action", line 255, in manage_exemptions_for_book
File "calibre_plugins.find_duplicates.duplicates", line 393, in remove_from_book_exemptions
File "calibre_plugins.find_duplicates.config", line 127, in set_exemption_list
File "calibre_plugins.find_duplicates.config", line 96, in set_library_config
File "site-packages\calibre\db\backend.py", line 141, in set_namespaced
File "site-packages\calibre\db\backend.py", line 122, in __setitem__
File "site-packages\calibre\db\backend.py", line 804, in execute
TooBigError: TooBigError: string or blob too big

Not sure how to manage this.

eschwartz · 08-06-2015, 12:20 AM

Quote:

Originally Posted by LadyKate

A binary search of the library takes a long time. Would a CRC check be faster?

I didn't see the option for a CRC vs the binary check. I'm not sure if it would be significantly faster or not.

I assume the plugin already uses the most efficient way, without offering options.

On second thought, that probably is hashes already -- you are comparing a whole bunch of files to every other file in the group, comparing a checksum is easier than reading the same files again and again.

LadyKate · 08-11-2015, 08:09 PM

I am trying to check for duplicates between two libraries but don't know how an option works.

I've googled and tried to find info on this but failed.

With my main library closed, I have a temp library that I use for adding new items open. In Dluplicate Search Options I have chosen Library Compare, Binary Compare.

I want to choose when doing a Binary compare, automatically remove duplicate formats but don't know what will happen.

Which library will the duplicate format be removed from?
Which format will be removed?

Is it only duplicate formats that are in separate records be removed?

Anyone know? or can anyone point me at where to find this info?

Sorry. I just made two test libraries to see how this works and found the option to remove the duplicate formats does not exist.

BetterRed · 08-11-2015, 08:39 PM

@LadyKate - if this related to the optional Find Duplicates plug-in then you should have posted this in its thread (there's an index at the top of the sub forum).

If you can confirm it is then I'll move it.

And assuming you are referring to the Find Duplicates PI, then I don't think it moves anything. You must do that via the built-in Copy to library feature. The Add Books settings control how the Copy to library feature handles 'duplicates'.

BR

LadyKate · 08-14-2015, 04:55 AM

Yes, Sorry, wrong place.

05-05-2015, 07:05 AM	#528
mbovenka Wizard Posts: 2,018 Karma: 13471689 Join Date: Oct 2007 Location: Almere, The Netherlands Device: Kobo Sage	Find Duplicates runs out of memory I think I am hitting some corner case where the Soundex algorithm starts eating memory. When I use Soundex match on my Titles, the Calibre process starts using more and more RAM (from a base value of about 900 MB) until it hits the 32-bit process limit and goes kaboom (with the portable version) or starts swapping like mad after filling my 8 GB RAM and completely locking my PC (with the 64-bit version). With the Fuzzy algorithm it uses no more than 1 GB (32-bit)or 1.5 GB (64-bit). The debug log of the 32-bit version is here: Spoiler: calibre Debug log calibre 2.26 Portable isfrozen: True is64bit: False Windows-8-6.2.9200 Windows ('32bit', 'WindowsPE') 32bit process running on 64bit windows ('Windows', '8', '6.2.9200') Python 2.7.9 Windows: ('8', '6.2.9200', '', 'Multiprocessor Free') Successfully initialized third party plugins: DeDRM && BOL_NL && catawiki && Extract ISBN && KoboTouchExtended && KindleUnpack - The Plugin && Manage Series && Count Pages && Barnes & Noble && Quality Check && Obok DeDRM && Kobo Utilities && Webscription && KePub Output && Goodreads && Fantastic Fiction Adults && Fantastic Fiction && Find Duplicates Starting up... DEBUG: 0.0 obok::utilities.py - loading translations DEBUG: 0.0 obok::dialogs.py - loading translations DEBUG: 0.0 obok::config.py - loading translations DEBUG: 0.0 obok::action_err.py - loading translations DEBUG: 0.0 KoboUtilites::action.py - loading translations DEBUG: 0.0 KoboUtilites::dialogs.py - loading translations DEBUG: 0.0 KoboUtilites::action.py - loading translations DEBUG: 13.1 No Kobo Touch, Glo or Mini appears to be connected DEBUG: 13.1 rebuild_menus - self.supports_ratings=None Started up in 29.84 seconds with 140238 books SMART_DEV ( 0.00: 0.002) _attach_to_port try ip address "" on port 9090 SMART_DEV ( 0.91: 0.804) startup_on_demand listening on port 9090 SMART_DEV ( 0.97: 0.021) _attach_to_port try ip address "" on port 54982 SMART_DEV ( 1.02: 0.000) startup_on_demand broadcast socket listening on port 54982 SMART_DEV ( 2.08: 1.014) run All IP addresses {'{6517BCBE-7D0A-4CB1-8C84-14B3FFAFC80F}': [{'addr': '169.254.196.192'}], '{FB9E5A1C-970A-4FE3-A88F-546A6A940162}': [{'broadcast': '192.168.192.255', 'netmask': '255.255.255.0', 'addr': '192.168.192.3'}], '{F42A7CC3-6D02-11E3-824E-806E6F6E6963}': [{'broadcast': '127.255.255.255', 'netmask': '255.0.0.0', 'addr': '127.0.0.1'}], '{5B09173B-6A84-4D21-87C3-BD721EB0E535}': [{'addr': '169.254.101.53'}]} calibre 2.26 Portable isfrozen: True is64bit: False Windows-8-6.2.9200 Windows ('32bit', 'WindowsPE') 32bit process running on 64bit windows ('Windows', '8', '6.2.9200') Python 2.7.9 Windows: ('8', '6.2.9200', '', 'Multiprocessor Free') Successfully initialized third party plugins: DeDRM && BOL_NL && catawiki && Extract ISBN && KoboTouchExtended && KindleUnpack - The Plugin && Manage Series && Count Pages && Barnes & Noble && Quality Check && Obok DeDRM && Kobo Utilities && Webscription && KePub Output && Goodreads && Fantastic Fiction Adults && Fantastic Fiction && Find Duplicates Traceback (most recent call last): File "calibre_plugins.find_duplicates.action", line 195, in toolbar_button_clicked File "calibre_plugins.find_duplicates.action", line 156, in find_book_duplicates File "calibre_plugins.find_duplicates.duplicates", line 127, in run_book_duplicates_check File "calibre_plugins.find_duplicates.book_algorith ms", line 61, in run_duplicate_check File "calibre_plugins.find_duplicates.book_algorith ms", line 114, in convert_candidates_to_groups File "calibre_plugins.find_duplicates.book_algorith ms", line 199, in partition_using_exemptions MemoryError If any more info is needed, let me know.

05-11-2015, 05:48 PM	#533
LadyKate Fanatic Posts: 515 Karma: 1470724 Join Date: Jul 2013 Location: Quebec CA Device: android 4 (samsung tablet and asus tablet)	When I run find duplicates after a name change still shows as duplicate I'm wondering if there is a way to have find duplicates run as if newly installed again. The reason for this is I have several titles (about 85 or so) which were flagged as duplicates using title/author search but have had the title corrected. They are still marked as duplicate in the duplicate exemption list. While some are valid duplicate titles (two editions of a book or a different translator) others were in error (such as the name of the series being entered originally instead of the name of the book). Just curious about this.

05-11-2015, 07:53 PM	#534
BetterRed null operator (he/him) Posts: 20,568 Karma: 26954694 Join Date: Mar 2012 Location: Sydney Australia Device: none	@LadyKate - I would do this Click Show all book duplicate exemptions from the Find Duplicates drop-down menu Select all books in the Book List(ctrl/a) Click Remove selected exemptions from the Find Duplicates drop-down menu Run Find Duplicates again and redo the exemptions - hopefully far fewer Watch out for this ==>> Some plugins, can change calibre's search highlighting BR

07-24-2015, 06:40 PM	#535
LadyKate Fanatic Posts: 515 Karma: 1470724 Join Date: Jul 2013 Location: Quebec CA Device: android 4 (samsung tablet and asus tablet)	Is there an option for a CRC check vs the binary check A binary search of the library takes a long time. Would a CRC check be faster? I didn't see the option for a CRC vs the binary check. I'm not sure if it would be significantly faster or not.

08-11-2015, 08:09 PM	#538
LadyKate Fanatic Posts: 515 Karma: 1470724 Join Date: Jul 2013 Location: Quebec CA Device: android 4 (samsung tablet and asus tablet)	Behavior of Checking for dups automatically remove duplicate format I am trying to check for duplicates between two libraries but don't know how an option works. I've googled and tried to find info on this but failed. With my main library closed, I have a temp library that I use for adding new items open. In Dluplicate Search Options I have chosen Library Compare, Binary Compare. I want to choose when doing a Binary compare, automatically remove duplicate formats but don't know what will happen. Which library will the duplicate format be removed from? Which format will be removed? Is it only duplicate formats that are in separate records be removed? Anyone know? or can anyone point me at where to find this info? Sorry. I just made two test libraries to see how this works and found the option to remove the duplicate formats does not exist. Last edited by LadyKate; 08-11-2015 at 08:36 PM. Reason: missed something before testing

04-27-2015, 11:15 AM	#526
laly003 Junior Member Posts: 4 Karma: 10 Join Date: Mar 2014 Device: Nook HD+	Well, not working at all for me, like I mentioned, I re-imported an existing book..therefore same title/author/metadata/file, etc and it did not recognized it as a dup using binary compare...using title/author I am finding over 70 dups. I like using binary compare because I don't need to clean up the titles/authors first. This was working great last time I used it but that was last year, maybe summer time... Wish I had not upgraded versions

04-28-2015, 11:29 AM	#527
laly003 Junior Member Posts: 4 Karma: 10 Join Date: Mar 2014 Device: Nook HD+	Update: Disregard my post about binary compare not working. I have multiple Calibre libraries setup and I tried the search in different library and it worked so must be something wrong or odd about the library where it's not working...

05-10-2015, 08:58 PM	#531
truth1ness Zealot Posts: 126 Karma: 50000 Join Date: Mar 2015 Device: none	Great plugin. Is there any way to tell it to ignore a certain Tag? Fuzzy search is always catching my News downloads because the magazines are almost all identically named and it really clutters up the duplicates page. It'd be great if I could ignore everything tagged "news".

08-11-2015, 08:39 PM	#539
BetterRed null operator (he/him) Posts: 20,568 Karma: 26954694 Join Date: Mar 2012 Location: Sydney Australia Device: none	@LadyKate - if this related to the optional Find Duplicates plug-in then you should have posted this in its thread (there's an index at the top of the sub forum). If you can confirm it is then I'll move it. And assuming you are referring to the Find Duplicates PI, then I don't think it moves anything. You must do that via the built-in Copy to library feature. The Add Books settings control how the Copy to library feature handles 'duplicates'. BR Last edited by BetterRed; 08-11-2015 at 08:42 PM.

08-14-2015, 04:55 AM	#540
LadyKate Fanatic Posts: 515 Karma: 1470724 Join Date: Jul 2013 Location: Quebec CA Device: android 4 (samsung tablet and asus tablet)	Yes, Sorry, wrong place.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[GUI Plugin] Quality Check	kiwidude	Plugins	1184	04-17-2024 06:17 PM
[GUI Plugin] View Manager	kiwidude	Plugins	414	04-13-2024 01:41 PM
[GUI Plugin] Open With	kiwidude	Plugins	403	04-01-2024 08:39 AM
[GUI Plugin] Generate Cover	kiwidude	Plugins	811	03-16-2024 11:31 PM
[GUI Plugin] Plugin Updater Deprecated	kiwidude	Plugins	159	06-19-2011 12:27 PM

Advert

Advert