[GUI Plugin] Find Duplicates - Page 47

Jeremy_D · 01-17-2020, 03:33 AM

Hi all,

I'm keen to have a go at porting this plugin to python3. According to the Calibre plugin stats, "Find Duplicates" is pretty popular and according to the py3 plugin port post it hasn't yet been done.

Is the original developer still floating about? If so are you ok with me working on a port? If not, does anyone else know if someone has already started work on this?

Also, does an existing public revision control (preferably git) repo already exist for this? Or should I just start a new one on GitHub?

Finally, assuming I do work on this, I'm a Linux only user, so to ensure it works on all platforms, are there other OS users who are interested in testing the py3 port?

BetterRed · 01-17-2020, 05:56 PM

Quote:

Originally Posted by Jeremy_D

Hi all,

I'm keen to have a go at porting this plugin to python3. According to the Calibre plugin stats, "Find Duplicates" is pretty popular and according to the py3 plugin port post it hasn't yet been done.

Is the original developer still floating about? If so are you ok with me working on a port? If not, does anyone else know if someone has already started work on this?

Also, does an existing public revision control (preferably git) repo already exist for this? Or should I just start a new one on GitHub?

Finally, assuming I do work on this, I'm a Linux only user, so to ensure it works on all platforms, are there other OS users who are interested in testing the py3 port?

The original developer (kiwidude) logs in occasionally, I suggest you send him a PM as a courtesy.

Use whatever version control you're comfortable with, but the distribution of the release versions must be via the Plugin Updater process - as per the stickies at the top of the Plugins sub-forum. The forum lacks a facility to assign a thread to a different member; so, that means having to ask one of the active mods (me, theducks, DoctorOhh, PeterT, kovid) to update the first post and the index - see the top of index.

Post test versions in a new messages. There'll be a few people to test new versions in current release calibre (i.e. 4.x) for Windows and MacOS, not so many for 4.99.x. You'll get more when Kovid calls for beta testers on what I assume will be calibre 5.

Regarding this plugin in particular:

It has a feature to exclude duplicate groups from consideration whenever the plugin is run.

Some time after that feature was implemented in FD, Kovid implemented the user accessible marking of books in the calibre core (which deprecated the Temp Marker plugin). This plugin makes use of the same mechanism. IIRC it keeps a list of excluded groups in the database, when the plugin is loaded (or on a library switch) it uses the list to re-mark the books - which confuses users.

The solutions that have come to mind is to avoid the existing temporary mark mechanism, and implement a 'plugins only' marking mechanism, and/or defer the re-marking of books to when the search for duplicates is executed.
And there's this problem ==>> How to fix Search showing all books with matches highlighted

I can't recall the circumstances when that happens, I don't think it relates to excluded groups.

Added: it either needs to restore the standard search highlight/filter setting back to what it was (but, when it would do that might be the question), or use a plugin specific approach

It would be good to get those issues resolved - probably after the py3 dust settles.

BR

Jeremy_D · 01-21-2020, 01:45 AM

Quote:

Originally Posted by BetterRed

The original developer (kiwidude) logs in occasionally, I suggest you send him a PM as a courtesy.

Thanks for that. I'll shoot him a PM shortly.

Quote:

Use whatever version control you're comfortable with, but the distribution of the release versions must be via the Plugin Updater process - as per the stickies at the top of the Plugins sub-forum. The forum lacks a facility to assign a thread to a different member; so, that means having to ask one of the active mods (me, theducks, DoctorOhh, PeterT, kovid) to update the first post and the index - see the top of index.

Ok great. I'll keep that in mind.

Quote:

Post test versions in a new messages. There'll be a few people to test new versions in current release calibre (i.e. 4.x) for Windows and MacOS, not so many for 4.99.x. You'll get more when Kovid calls for beta testers on what I assume will be calibre 5.

Do you mean posting it in a new message here? Or somewhere else?

Quote:

Regarding this plugin in particular:

It has a feature to exclude duplicate groups from consideration whenever the plugin is run.

Some time after that feature was implemented in FD, Kovid implemented the user accessible marking of books in the calibre core (which deprecated the Temp Marker plugin). This plugin makes use of the same mechanism. IIRC it keeps a list of excluded groups in the database, when the plugin is loaded (or on a library switch) it uses the list to re-mark the books - which confuses users.

The solutions that have come to mind is to avoid the existing temporary mark mechanism, and implement a 'plugins only' marking mechanism, and/or defer the re-marking of books to when the search for duplicates is executed.
And there's this problem ==>> How to fix Search showing all books with matches highlighted

I can't recall the circumstances when that happens, I don't think it relates to excluded groups.

Added: it either needs to restore the standard search highlight/filter setting back to what it was (but, when it would do that might be the question), or use a plugin specific approach

It would be good to get those issues resolved - probably after the py3 dust settles.

Ok great. I won't give any promises at this stage, but it would be nice to have a look at those. Although I'm fairly novice as far as programming goes, so porting to python3 without introducing any new bugs is probably the first step (at least as far as I'm concerned).

Speaking of which, I've already run 2to3 over this plugin code, but I'm still struggling a little to get a python3 instance of Calibre up and running so I can start doing some preliminary testing. Unfortunately, the dev instructions don't appear to work with Python3 (it's still running python2). On reflection that makes sense as the binary install would be pre-compiled. Anyway, I'll hopefully work that out and get back to this...

Thanks again for the warm welcome and friendly advice.

BetterRed · 01-21-2020, 02:51 AM

Quote:

Originally Posted by Jeremy_D

... Do you mean posting it in a new message here? Or somewhere else?

Attach the zip to a new post in this thread. Delete it when its replaced or released.

Quote:

Originally Posted by Jeremy_D

... Speaking of which, I've already run 2to3 over this plugin code, but I'm still struggling a little to get a python3 instance of Calibre up and running so I can start doing some preliminary testing. Unfortunately, the dev instructions don't appear to work with Python3 (it's still running python2). On reflection that makes sense as the binary install would be pre-compiled. Anyway, I'll hopefully work that out and get back to this...

Plug yourself into this thread ==>>https://www.mobileread.com/forums/sh...d.php?t=326394

I'm sure davidfor, JimmXinu etc will be happy to give you some pointers, they're both very familiar with calibre's development setup, kiwidudes plugins, and what's needed to make them py3 compliant.

BR

davidfor · 01-21-2020, 05:31 AM

Quote:

Originally Posted by BetterRed

Attach the zip to a new post in this thread. Delete it when its replaced or released.

Plug yourself into this thread ==>>https://www.mobileread.com/forums/sh...d.php?t=326394

And Attn plugin devs: porting to python 3. From memory, there is some discussion of running both the released and the Python 3 version on one machine. On my laptop, I installed the portable Python 3 beta and have some scripts to run it with the appropriate environment variables.

Quote:

I'm sure davidfor, JimmXinu etc will be happy to give you some pointers, they're both very familiar with calibre's development setup, kiwidudes plugins, and what's needed to make them py3 compliant.

Yes, ask and someone will answer if we know. If it is about how this plugin works, you probably should ask here. But, using either of the above threads should get the attention of everyone that is doing calibre development.

davidfor · 02-01-2020, 12:39 AM

I have made the changes to the plugin to make it Python 3 compatible. I don't use it much, but, it was there was an error each time I changed library. That was annoying me and causing some other problems.

I have tested some basic function in both the release version of calibre and the Python 3 betas. It all seems to work. But, as I don't use it much, I haven't tested everything. It loads, will find duplicates in both the current library and another library, but, I have only used the default options for everything. If someone who does use it in test it in either environment, that would be very helpful. And honestly, at this point, I am more concerned about breaking the plugin in the released version of calibre than the Python 3 version.

@Jeremy_D: Sorry, I had completely forgotten that you had picked this plugin to look at until I came to do this post.

sikm · 02-08-2020, 08:11 PM

For some reason this plugin isn't showing up in the main toolbar. As Calibre loads it does show up but when loaded it's not there - nor is Edit Book. If I try to modify the main toolbar then Find Duplicates and Edit Book both show as being there but when the library is loaded it's not there.
I don't recall changing anything but can't think what could be wrong.

BetterRed · 02-08-2020, 09:57 PM

Do you have a device attached, there are separate toolbars and menus when a device is attached

BR

kovidgoyal · 02-08-2020, 11:49 PM

there are probably too many icons on your toolbar, click the little dots at the edge of the toolbar to expand it and see the missing icons.

Section8 · 02-15-2020, 03:20 PM

Running Calibre 4.9.1 here. Thanks for the beta - I was getting an xrange error switching libraries and the beta seems to have fixed that.

I mostly use the plugin to weed out publisher variations. If I do Find Metadata variations->check "Publisher", and then press "Search", I get this error:

Code:

calibre, version 4.9.1
ERROR: Unhandled exception: <b>TypeError</b>:'cmp' is an invalid keyword argument for this function

calibre 4.9.1  embedded-python: False is64bit: True
Linux-4.19.97-gentoo-x86_64-Intel-R-_Core-TM-_i7-4790K_CPU_@_4.00GHz-with-gentoo-2.6 Linux ('64bit', 'ELF')
('Linux', '4.19.97-gentoo', '#2 SMP PREEMPT Sat Feb 1 18:45:22 CST 2020')
Python 3.6.9
Linux: ('Gentoo Base System', '2.6', '')
Interface language: None
Successfully initialized third party plugins: Count Pages (1, 9, 0) && Diaps Editing Toolbag (0, 3, 6) && EpubSplit (2, 9, 0) && Extract ISBN (1, 4, 4) && Find Duplicates (1, 6, 5) && Goodreads Sync (1, 14, 2) && KindleUnpack - The Plugin (0, 82, 1) && Kobo Books (1, 8, 0) && Modify ePub (1, 4, 0) && Quality Check (1, 9, 11)
Traceback (most recent call last):
  File "calibre_plugins.find_duplicates.dialogs", line 705, in _refresh_results
  File "calibre_plugins.find_duplicates.variation_algorithms", line 49, in run_variation_check
  File "calibre_plugins.find_duplicates.variation_algorithms", line 117, in _flatten_candidates_for_item
  File "calibre_plugins.find_duplicates.variation_algorithms", line 136, in _clean_dup_groups
TypeError: 'cmp' is an invalid keyword argument for this function

JSWolf · 02-15-2020, 06:17 PM

Quote:

Originally Posted by Section8

Running Calibre 4.9.1 here. Thanks for the beta - I was getting an xrange error switching libraries and the beta seems to have fixed that.

I mostly use the plugin to weed out publisher variations. If I do Find Metadata variations->check "Publisher", and then press "Search", I get this error:

Code:

calibre, version 4.9.1
ERROR: Unhandled exception: <b>TypeError</b>:'cmp' is an invalid keyword argument for this function

calibre 4.9.1  embedded-python: False is64bit: True
Linux-4.19.97-gentoo-x86_64-Intel-R-_Core-TM-_i7-4790K_CPU_@_4.00GHz-with-gentoo-2.6 Linux ('64bit', 'ELF')
('Linux', '4.19.97-gentoo', '#2 SMP PREEMPT Sat Feb 1 18:45:22 CST 2020')
Python 3.6.9
Linux: ('Gentoo Base System', '2.6', '')
Interface language: None
Successfully initialized third party plugins: Count Pages (1, 9, 0) && Diaps Editing Toolbag (0, 3, 6) && EpubSplit (2, 9, 0) && Extract ISBN (1, 4, 4) && Find Duplicates (1, 6, 5) && Goodreads Sync (1, 14, 2) && KindleUnpack - The Plugin (0, 82, 1) && Kobo Books (1, 8, 0) && Modify ePub (1, 4, 0) && Quality Check (1, 9, 11)
Traceback (most recent call last):
  File "calibre_plugins.find_duplicates.dialogs", line 705, in _refresh_results
  File "calibre_plugins.find_duplicates.variation_algorithms", line 49, in run_variation_check
  File "calibre_plugins.find_duplicates.variation_algorithms", line 117, in _flatten_candidates_for_item
  File "calibre_plugins.find_duplicates.variation_algorithms", line 136, in _clean_dup_groups
TypeError: 'cmp' is an invalid keyword argument for this function

Install Calibre 4.10.1 and try again. 4.10.1 change the version of QT to a newer version so the results might be different.

DNSB · 02-15-2020, 08:15 PM

Quote:

Originally Posted by Section8

Running Calibre 4.9.1 here. Thanks for the beta - I was getting an xrange error switching libraries and the beta seems to have fixed that.

I mostly use the plugin to weed out publisher variations. If I do Find Metadata variations->check "Publisher", and then press "Search", I get this error:

Code:

calibre, version 4.9.1
ERROR: Unhandled exception: <b>TypeError</b>:'cmp' is an invalid keyword argument for this function

calibre 4.9.1  embedded-python: False is64bit: True
Linux-4.19.97-gentoo-x86_64-Intel-R-_Core-TM-_i7-4790K_CPU_@_4.00GHz-with-gentoo-2.6 Linux ('64bit', 'ELF')
('Linux', '4.19.97-gentoo', '#2 SMP PREEMPT Sat Feb 1 18:45:22 CST 2020')
Python 3.6.9
Linux: ('Gentoo Base System', '2.6', '')

Under Windows 10 x64, calibre 4.10.1, I can't duplicate your error. However, since you are running Linux and using Python 3.6.9 while my system is using the embedded Python 2.7.16+, I'd suspect that may be the difference. Doing a quick Google search came up with multiple items such as: TypeError: 'cmp' is an invalid keyword argument for this function which is about 5 years old but likely still valid and Convert a cmp function to a key function (Python recipe).

Code:

calibre 4.10.1 [64bit]  embedded-python: True is64bit: True
Windows-10-10.0.18362-SP0 Windows ('64bit', 'WindowsPE')
('Windows', '10', '10.0.18362')
Python 2.7.16+
Windows: ('10', '10.0.18362', 'SP0', u'Multiprocessor Free')
Interface language: None

Analysing 2404 publisher for duplicates...
Completed duplicate analysis in: 0.195999860764
Found 249 duplicate groups

Basically, it looks as if the conversion to Python3 has some unfinished bits and pieces.

davidfor · 02-15-2020, 08:44 PM

Quote:

Originally Posted by Section8

Running Calibre 4.9.1 here. Thanks for the beta - I was getting an xrange error switching libraries and the beta seems to have fixed that.

I mostly use the plugin to weed out publisher variations. If I do Find Metadata variations->check "Publisher", and then press "Search", I get this error:

Code:

calibre, version 4.9.1
ERROR: Unhandled exception: <b>TypeError</b>:'cmp' is an invalid keyword argument for this function

calibre 4.9.1  embedded-python: False is64bit: True
Linux-4.19.97-gentoo-x86_64-Intel-R-_Core-TM-_i7-4790K_CPU_@_4.00GHz-with-gentoo-2.6 Linux ('64bit', 'ELF')
('Linux', '4.19.97-gentoo', '#2 SMP PREEMPT Sat Feb 1 18:45:22 CST 2020')
Python 3.6.9
Linux: ('Gentoo Base System', '2.6', '')
Interface language: None
Successfully initialized third party plugins: Count Pages (1, 9, 0) && Diaps Editing Toolbag (0, 3, 6) && EpubSplit (2, 9, 0) && Extract ISBN (1, 4, 4) && Find Duplicates (1, 6, 5) && Goodreads Sync (1, 14, 2) && KindleUnpack - The Plugin (0, 82, 1) && Kobo Books (1, 8, 0) && Modify ePub (1, 4, 0) && Quality Check (1, 9, 11)
Traceback (most recent call last):
  File "calibre_plugins.find_duplicates.dialogs", line 705, in _refresh_results
  File "calibre_plugins.find_duplicates.variation_algorithms", line 49, in run_variation_check
  File "calibre_plugins.find_duplicates.variation_algorithms", line 117, in _flatten_candidates_for_item
  File "calibre_plugins.find_duplicates.variation_algorithms", line 136, in _clean_dup_groups
TypeError: 'cmp' is an invalid keyword argument for this function

I missed one of the sorts in the Python 3 changes and it still had the old parameters. But, that's what betas are for. It is fixed in the attached beta.

Section8 · 02-16-2020, 06:47 PM

Thsnk you - that fixed it.

dunhill · 03-08-2020, 01:44 PM

I would like to make my contribution to translate it into Spanish, since it is being remodeled.

02-01-2020, 12:39 AM	#696
davidfor Grand Sorcerer Posts: 24,907 Karma: 47303748 Join Date: Jul 2011 Location: Sydney, Australia Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos	Beta - Python 3 compatible I have made the changes to the plugin to make it Python 3 compatible. I don't use it much, but, it was there was an error each time I changed library. That was annoying me and causing some other problems. I have tested some basic function in both the release version of calibre and the Python 3 betas. It all seems to work. But, as I don't use it much, I haven't tested everything. It loads, will find duplicates in both the current library and another library, but, I have only used the default options for everything. If someone who does use it in test it in either environment, that would be very helpful. And honestly, at this point, I am more concerned about breaking the plugin in the released version of calibre than the Python 3 version. @Jeremy_D: Sorry, I had completely forgotten that you had picked this plugin to look at until I came to do this post. Last edited by davidfor; 02-15-2020 at 08:46 PM. Reason: Removed beta as there is a later version.

02-08-2020, 08:11 PM	#697
sikm Member Posts: 14 Karma: 10 Join Date: Oct 2011 Location: Melbourne, Australia Device: kindle	Calibre Plugin - Find Duplicates For some reason this plugin isn't showing up in the main toolbar. As Calibre loads it does show up but when loaded it's not there - nor is Edit Book. If I try to modify the main toolbar then Find Duplicates and Edit Book both show as being there but when the library is loaded it's not there. I don't recall changing anything but can't think what could be wrong.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[GUI Plugin] Quality Check	kiwidude	Plugins	1184	04-17-2024 06:17 PM
[GUI Plugin] View Manager	kiwidude	Plugins	414	04-13-2024 01:41 PM
[GUI Plugin] Open With	kiwidude	Plugins	403	04-01-2024 08:39 AM
[GUI Plugin] Generate Cover	kiwidude	Plugins	811	03-16-2024 11:31 PM
[GUI Plugin] Plugin Updater Deprecated	kiwidude	Plugins	159	06-19-2011 12:27 PM

01-17-2020, 03:33 AM	#691
Jeremy_D Junior Member Posts: 5 Karma: 10 Join Date: Jan 2020 Location: Australia Device: kindle (very old)	Hi all, I'm keen to have a go at porting this plugin to python3. According to the Calibre plugin stats, "Find Duplicates" is pretty popular and according to the py3 plugin port post it hasn't yet been done. Is the original developer still floating about? If so are you ok with me working on a port? If not, does anyone else know if someone has already started work on this? Also, does an existing public revision control (preferably git) repo already exist for this? Or should I just start a new one on GitHub? Finally, assuming I do work on this, I'm a Linux only user, so to ensure it works on all platforms, are there other OS users who are interested in testing the py3 port?

02-08-2020, 09:57 PM	#698
BetterRed null operator (he/him) Posts: 20,568 Karma: 26954694 Join Date: Mar 2012 Location: Sydney Australia Device: none	Do you have a device attached, there are separate toolbars and menus when a device is attached BR

02-08-2020, 11:49 PM	#699
kovidgoyal creator of calibre Posts: 43,858 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	there are probably too many icons on your toolbar, click the little dots at the edge of the toolbar to expand it and see the missing icons.

02-16-2020, 06:47 PM	#704
Section8 Addict Posts: 254 Karma: 2092424 Join Date: Oct 2011 Location: Arlington, TX Device: Kindle PW4, Moon+ Reader on a cheap Android tablet	Thsnk you - that fixed it.

03-08-2020, 01:44 PM	#705
dunhill Guru Posts: 723 Karma: 228142 Join Date: Sep 2017 Location: Argentina Device: moon+ reader, kindle paperwhite	I would like to make my contribution to translate it into Spanish, since it is being remodeled.