[GUI Plugin] Find Duplicates

[kiwidude]

Do you have a lot of duplicate authors? Feel free to PM me details if you prefer.

I do suspect I know what the issue is if that is indeed the case. I tried not to change the underlying approach which is to get a list of authors only and compare their names. Whereas the 1.9.7 version iterates across all the books, and then compares the authors from those books.

Iterating just across the author names would ordinarily be the faster approach. But the downside now is that I do not have the details of the books for that author, and instead have to run an additional search to retrieve them in the case where they are showing up as a duplicate.

For only a handful of duplicate authors I would not expect this to be a problem. If you have hundreds or thousands I can imagine this not scaling well...

[kiwidude]

This does have me wondering whether Find Duplicates needs some refactoring at some point to perform it's "Find Book Duplicates" and "Find Library Duplicates" searches as background jobs. For larger libraries searches can take a long time to run and it is pretty annoying to have the GUI blocked. Another one for the rainy day list...

[theducks]

Originally Posted by kiwidude

This does have me wondering whether Find Duplicates needs some refactoring at some point to perform it's "Find Book Duplicates" and "Find Library Duplicates" searches as background jobs. For larger libraries searches can take a long time to run and it is pretty annoying to have the GUI blocked. Another one for the rainy day list...

I use the old Cross Library Duplicates: Similar Title, Fuzzy Author (Looking for Koalas who write???) prior to doing a Copy To Library (AKA move) from my Intake Library
The viewed Library (Intake) Is on a Shared drive. The Target is on This PC

(Yes I know. Networked drives can be an issue, but I have not ever had an issue with a PULL from a remote. OTOH Pushed did barf frequently)
The Intake is fairly tiny (gets processed frequently). The Target has ~15K+ books (the libraries are on Rotating drives, although the Host has a SSD)
Time is under a minute
Could the 'slowness' others are seeing, be simply from drive trashing?

[chaley]

Originally Posted by kiwidude

I do suspect I know what the issue is if that is indeed the case. I tried not to change the underlying approach which is to get a list of authors only and compare their names. Whereas the 1.9.7 version iterates across all the books, and then compares the authors from those books.

This might be an instance of the performance problem I worried about in this post in the development thread. The original code I wrote used the underlying API, but in a way that required the reversal of the {id:authors} dict for every author query. Say there are 5000 authors. That would be 5000 reversals of a 5000 item dictionary every time the id/name pairs are fetched.

I suggest you try it with the changed code, which does the reversal once then uses the reversed dict. This should be no slower than the old method that ignored VLs.

[chaley]

Originally Posted by kiwidude

Iterating just across the author names would ordinarily be the faster approach. But the downside now is that I do not have the details of the books for that author, and instead have to run an additional search to retrieve them in the case where they are showing up as a duplicate.

The calibre db method new_api.books_for_field(field_name, item_id) might help here. For example, given author_id NNN it returns all the book ids that have that author. It is significantly faster than searching.

Then, to get full metadata for a book with book_id you could then use

new_api.get_proxy_metadata(book_id)

that returns a 'lazy' metadata object that evaluates metadata fields only when requested. If you need only a few metadata fields then the lazy fetch is much faster.

[kiwidude]

Thanks chaley (and theDucks for their feedback). Those tips above look promising, obviously calibre's underlying apis have changed a bit in the last 10 years unsurprisingly to add this new_api stuff which looks to be worthy of some investigation.

I did find one other performance bottleneck which is the set_marked_ids when it is called with 50k rows (taking 11 seconds on my machine). I was progressively adding print statements through calibre source to find out where the slowdown is, got as far as refresh_ids() being invoked by marked_changed().

So my plugins normally do these two pairs of calls, for example:

self.gui.current_db.set_marked_ids(marked_ids)
self.gui.search.set_search_string('marked:library_duplicate')

In this situation because I am issuing a search straight after calling set_marked_ids I wonder if there is some expensive part of this "refresh" going on that doesn't actually need to happen because of the search afterwards. Or maybe it is that refresh which makes the search so fast of course...

[kiwidude]

Originally Posted by chaley

This might be an instance of the performance problem I worried about in this post in the development thread. The original code I wrote used the underlying API, but in a way that required the reversal of the {id:authors} dict for every author query. Say there are 5000 authors. That would be 5000 reversals of a 5000 item dictionary every time the id/name pairs are fetched.

I suggest you try it with the changed code, which does the reversal once then uses the reversed dict. This should be no slower than the old method that ignored VLs.

You were bang on with this chaley, the feedback I had from alumoi was that a new version with your change fixed that issue. It was running for > 10 minutes with no results before the change, now it completes in 3 seconds.

The downside now is that it then takes an additional 15 seconds for set_marked_ids() as per my comments above to complete for them to see the results. If there isn't a quick win on that one I may just make this a new configuration option on the Find Library Duplicates dialog as to whether to display the results or not.

[chaley]

Originally Posted by kiwidude

I did find one other performance bottleneck which is the set_marked_ids when it is called with 50k rows (taking 11 seconds on my machine). I was progressively adding print statements through calibre source to find out where the slowdown is, got as far as refresh_ids() being invoked by marked_changed().

So my plugins normally do these two pairs of calls, for example:

self.gui.current_db.set_marked_ids(marked_ids)
self.gui.search.set_search_string('marked:library_duplicate')

The refresh_ids() is necessary to update the marked pins in the gui. It shouldn't be too slow. For example, in one library with 2500 books, selecting all the books then using "mark with text", the response is instantaneous.

Question: are you calling set_marked_ids in a loop with the dict growing each time, or once with a complete dict? If the former then that might explain it because all the work is redone for each call. If the latter then

Another test: how long does it take if you select all 50,000 books in the library then use Mark books to set a text mark? If this takes 11 seconds then we know the culprit. If it doesn't then there is something in the sequence of calls.

In this situation because I am issuing a search straight after calling set_marked_ids I wonder if there is some expensive part of this "refresh" going on that doesn't actually need to happen because of the search afterwards. Or maybe it is that refresh which makes the search so fast of course...

The search will use the marked dict to get the list of IDs, a very fast operation. It does depend on the marks being set, but it doesn't depend on the marks being refreshed in the GUI.

If you are willing, could you give me the metadata.db for the 50,000 book library? With that I could look at what it actually happening. It may be that we could add a "refresh=True" parameter to set_marked_ids(), telling it whether or not to do the actual refresh. My theory: if you do the search at some point after then the refresh_ids will occur there, involving only the books returned by the search.

[alumoi]

I know somebody's gonna hate me, but while doing a find metadata variation (with the 'alumoi special version') I ran into this:

calibre 6.5 embedded-python: True
Windows-10-10.0.19044-SP0 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.19044')
Python 3.10.1
Windows: ('10', '10.0.19044', 'SP0', 'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: Gather KFX-ZIP (from KFX Input) (1, 49, 0) && DeDRM (7, 2, 1) && Package KFX (from KFX Input) (1, 49, 0) && Barnes & Noble (1, 3, 0) && Clean Metadata (0, 0, 6) && Extract ISBN (1, 5, 1) && Fantastic Fiction (1, 5, 1) && Fantastic Fiction Adults (1, 2, 0) && Find Duplicates (1, 10, 2) && Goodreads (1, 7, 0) && ISFDB (3, 0, 0) && KFX metadata reader (from KFX Input) (1, 49, 0) && KFX Input (1, 49, 0) && Kindle Collections (1, 7, 29) && Obok DeDRM (7, 2, 1) && Quality Check (1, 12, 0) && Smashwords Metadata (1, 0, 2)
Traceback (most recent call last):
File "calibre_plugins.find_duplicates.dialogs", line 901, in _on_variation_list_item_changed
File "calibre_plugins.find_duplicates.dialogs", line 837, in _search_in_gui
File "calibre_plugins.find_duplicates.dialogs", line 905, in _decode_list_item
KeyError: 2926

while changing the author.

[kiwidude]

Originally Posted by alumoi

I know somebody's gonna hate me, but while doing a find metadata variation (with the 'alumoi special version') I ran into this

No hate I will make sure it is sorted for the next release.

[kiwidude]

Originally Posted by chaley

Question: are you calling set_marked_ids in a loop with the dict growing each time, or once with a complete dict? If the former then that might explain it because all the work is redone for each call. If the latter then

Definitely not in a loop, once at the end.

Originally Posted by chaley

Another test: how long does it take if you select all 50,000 books in the library then use Mark books to set a text mark? If this takes 11 seconds then we know the culprit. If it doesn't then there is something in the sequence of calls.

It is even slower using the GUI to mark books . Clearing is super fast. Mark books with text label takes about the 11 seconds to appear in the dialog, then another 17 seconds to apply the mark.

Originally Posted by chaley

If you are willing, could you give me the metadata.db for the 50,000 book library? With that I could look at what it actually happening. It may be that we could add a "refresh=True" parameter to set_marked_ids(), telling it whether or not to do the actual refresh. My theory: if you do the search at some point after then the refresh_ids will occur there, involving only the books returned by the search.

I will send it through to you...

[Graham Hurst]

Originally Posted by alumoi

I know somebody's gonna hate me, but while doing a find metadata variation (with the 'alumoi special version') I ran into this:

calibre 6.5 embedded-python: True
Windows-10-10.0.19044-SP0 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.19044')
Python 3.10.1
Windows: ('10', '10.0.19044', 'SP0', 'Multiprocessor Free')
Interface language: None
Successfully initialized third party plugins: Gather KFX-ZIP (from KFX Input) (1, 49, 0) && DeDRM (7, 2, 1) && Package KFX (from KFX Input) (1, 49, 0) && Barnes & Noble (1, 3, 0) && Clean Metadata (0, 0, 6) && Extract ISBN (1, 5, 1) && Fantastic Fiction (1, 5, 1) && Fantastic Fiction Adults (1, 2, 0) && Find Duplicates (1, 10, 2) && Goodreads (1, 7, 0) && ISFDB (3, 0, 0) && KFX metadata reader (from KFX Input) (1, 49, 0) && KFX Input (1, 49, 0) && Kindle Collections (1, 7, 29) && Obok DeDRM (7, 2, 1) && Quality Check (1, 12, 0) && Smashwords Metadata (1, 0, 2)
Traceback (most recent call last):
File "calibre_plugins.find_duplicates.dialogs", line 901, in _on_variation_list_item_changed
File "calibre_plugins.find_duplicates.dialogs", line 837, in _search_in_gui
File "calibre_plugins.find_duplicates.dialogs", line 905, in _decode_list_item
KeyError: 2926

while changing the author.

================================================


I concur, since the last version update of the plugin I have been unable to perform any Author "find metadata variations" searches for Authors . I am really hoping this get fixed soon please.



An example Error log below

calibre, version 6.7.0
ERROR: Unhandled exception: <b>KeyError</b>:43653

calibre 6.7 embedded-python: True
Windows-10-10.0.22621-SP0 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.22621')
Python 3.10.1
Windows: ('10', '10.0.22621', 'SP0', 'Multiprocessor Free')
Interface language: en_GB
Successfully initialized third party plugins: Gather KFX-ZIP (from KFX Input) (1, 49, 0) && Package KFX (from KFX Input) (1, 49, 0) && Author Book Count (2, 2, 2) && Fantastic Fiction (1, 5, 1) && Find Duplicates (1, 10, 1) && KFX metadata reader (from KFX Input) (1, 49, 0) && KFX Input (1, 49, 0) && Set KFX metadata (from KFX Output) (1, 64, 0) && KFX Output (1, 64, 0) && Kindle Collections (1, 7, 29) && KoboTouchExtended (3, 6, 3) && Modify ePub (1, 7, 3) && Obok DeDRM (6, 5, 4) && Quality Check (1, 12, 0) && Reading List (1, 14, 0)
Traceback (most recent call last):
File "calibre_plugins.find_duplicates.dialogs", line 901, in _on_variation_list_item_changed
File "calibre_plugins.find_duplicates.dialogs", line 837, in _search_in_gui
File "calibre_plugins.find_duplicates.dialogs", line 905, in _decode_list_item
KeyError: 43653

[kiwidude]

Release Notes:
https://github.com/kiwidude68/calibr...icates-v1.10.2

Releasing this version now so those impacted by the Metadata Variations bug (which I never reproduced) should be able to continue. If anyone has some exact steps to repro that problem let me know.

[Graham Hurst]

Originally Posted by kiwidude

Release Notes:
https://github.com/kiwidude68/calibr...icates-v1.10.2

Releasing this version now so those impacted by the Metadata Variations bug (which I never reproduced) should be able to continue. If anyone has some exact steps to repro that problem let me know.

There is not enough love in the world. It's not crashing anymore on the quick tests I've done.

[chaley]

Originally Posted by kiwidude

It is even slower using the GUI to mark books . Clearing is super fast. Mark books with text label takes about the 11 seconds to appear in the dialog, then another 17 seconds to apply the mark.

Thank you for the db. That helped enormously!

I found yet another 'hidden' linear search deep in the guts of the gui's view of the database. The code verified that a book_id exists in the gui display (has a gui row number) by searching for the book in a list. It being a list, the average number of "looks" to make the check is len(books)/2. It did this for every book_id in the list of ids passed to refresh_ids(). For your db with all 50,000 books selected that meant 1.25 billion lookups (!) to refresh the booklist. The fix is to keep a dict {book_id:row} so the check is one dict probe per book id.

I also found a place where on a mouse click calibre checked to see if the clicked-on row was in a selection, and if so checked some other things to see if drag-n-drop is being used. By refactoring the check I knocked 3 seconds off the right-click on a book.

Qt takes a long time to return the selected rows. I can't do anything about that.

The bottom line (all times measured on my machine, an Intel Core i7-10710U):

• It still takes around 4 seconds to select all 50,000 books with Ctrl-A. This time is all in Qt.
• Opening the context menu for a book is now instantaneous. It was around 4 seconds.
• It takes 4 seconds to open the mark with text dialog for the same reason it takes 4 seconds to select the books: Qt returning the selected rows.
• It takes 30 milliseconds after hitting OK to mark the books and refresh the gui. This took 12.2 seconds before the changes.

What took 24 seconds before now takes 8, and all of that 8 is in Qt selection management.

I will submit the changes to Kovid. The fix is a bit exotic so he might prefer an alternate approach. I will post here once we have a resolution in source.