[GUI Plugin] Find Duplicates

[Rellwood]

Marked books are losing their marked icon once the search field is cleared....

Instead of the icon remaining once I clear out the field (so I can check the duplicates against the books in the library together) the icon is now being removed.

I submitted a bug report, but thought it should be known here as well.

Calibre 3.9
Plugin version 1.6.3

[Northguy]

I would like to add a dutch variant to the Fuzzy Compare of the Find Duplicates plug-in. Is it possible to change English words like:

the -> de
and -> en
a -> een

somewhere in a file in order to better tweak the fuzzy file compare? Does someone know where I can raise a feature request or where a Github of this plugin is maintained?

[BetterRed]

Originally Posted by Northguy

Does someone know where I can raise a feature request

You just did, but the originator of the plugin is no longer active, the only changes made since he left have been to maintain compatibility with the calibre core, and an option to not pop messages.

Originally Posted by Northguy

or where a Github of this plugin is maintained?

Doesn't exist. Latest source is in the plugin's ZIP. You can change it and use the modified version via the install plugin from the file option.

I've done that with a couple of Kiwidude's plugins, as have others. I renamed them to MyXxxxx and removed the released Xxxxx plugins.

BR

[LadyKate]

Originally Posted by BetterRed

You just did, but the originator of the plugin is no longer active, the only changes made since he left have been to maintain compatibility with the calibre core, and an option to not pop messages.


Doesn't exist. Latest source is in the plugin's ZIP. You can change it and use the modified version via the install plugin from the file option.

I've done that with a couple of Kiwidude's plugins, as have others. I renamed them to MyXxxxx and removed the released Xxxxx plugins.

BR

Perhaps, if the source is available, the OP could make, test and release a Dutch version of the plugin with of course all the credits due the original release.

[davidfor]

Originally Posted by LadyKate

Perhaps, if the source is available, the OP could make, test and release a Dutch version of the plugin with of course all the credits due the original release.

As stated in the post you quoted, the OP is no longer around to maintain the plugins. Each plugin is a zip file and contains the full code for the plugin. If you want to make changes you can. And if you want to make them available to other people, you can. Most of kiwidude's plugins are being maintained by the community.

Adding a language is possible, but a reasonable amount of work. This plugin was released before language support was added for plugins. To do this now, isn't hard. It means adding a few lines to load the language files and then finding all the strings in the plugin that should be translated and marking them. Then you run another program to generate the language files and do the translations. Not hard, but takes a bit of time. If you do it, I'm sure at least one of the non-English speakers will supply a translation.

[odinokij]

Originally Posted by Northguy

I would like to add a dutch variant to the Fuzzy Compare of the Find Duplicates plug-in. Is it possible to change English words like:

the -> de
and -> en
a -> een

somewhere in a file in order to better tweak the fuzzy file compare? Does someone know where I can raise a feature request or where a Github of this plugin is maintained?

I have modified the Find Duplicates plugin in order to:

- Take into account articles, prepositions... not only in english but also in russian, esperanto, spanish, french, german and dutch(*)
- Take into account plurals
- Improve detection in volumes with more than a book in the title
- Improve Fuzzy authors in order to consider alias: "Name Lastname (Alias)"
- Detect duplicates when the series is in the title as "Series #. Title" or "(Series #) Title"
- In Identical Authors, avoid consider as identical if the number of authors is different, i.e., "Author1 & Author2 != Author1"
- Optionally disable the exemptions list
- (...)

(*) Note: I don't speak german nor dutch, so if you find any fail in these languages, please report so I can fix it.

I share the modified plugin in the next link:

https://1drv.ms/u/s!AobKqg_eHGvlicteuckLgFlxmqE2zw


You can import the plugin in calibre by:

Ctrl+P / Advanced-Plugins / Load plugin from file --> Select "duplicates++.zip" file



I hope it will also be useful for you,

Odinokij.

[Northguy]

Originally Posted by odinokij

(*) Note: I don't speak german nor dutch, so if you find any fail in these languages, please report so I can fix it.

I hope it will also be useful for you,

Odinokij.

Hey great! ...

I started working on the 'matching.py' of the original plug-in, but your version looks a lot more elaborated. When comparing to the original matching.py, I have some questions:
1) In general: I do not speak spanish/portuguese, so I do not understand your comments

2) In def fuzzy_it(text, patterns=None):, why didn't you change (tweaks.get('title_sort_articles', r'^(a|the|an)\s+'), ''),

3) In def get_title_tokens I added something like:

'NL', 'ebook', 'e-Book' and 'druk' as possible alternatives in

(r'(?i)[({\[](\d{4}|ebook|e-book|NL|omnibus|anthology|hardcover|paperback|mass \s*market|edition|ed\.)[\])}]', ''),

4) In def get_title_tokens I think we should need to add 'een', because the words 'de', 'het' and 'een' represent articles as used in the Dutch language.

tokens_du = ('een', 'de', 'het', 'van', 'met', 'naar')

[odinokij]

Originally Posted by Northguy

Hey great! ...

I started working on the 'matching.py' of the original plug-in, but your version looks a lot more elaborated. When comparing to the original matching.py, I have some questions:
1) In general: I do not speak spanish/portuguese, so I do not understand your comments

2) In def fuzzy_it(text, patterns=None):, why didn't you change (tweaks.get('title_sort_articles', r'^(a|the|an)\s+'), ''),

3) In def get_title_tokens I added something like:

'NL', 'ebook', 'e-Book' and 'druk' as possible alternatives in


4) In def get_title_tokens I think we should need to add 'een', because the words 'de', 'het' and 'een' represent articles as used in the Dutch language.

Thanx, I will take a look to your suggestions and update the version. I'll make you know when it's ready.

Kind regards,
Odinokij

[odinokij]

Originally Posted by Northguy

Hey great! ...

I started working on the 'matching.py' of the original plug-in, but your version looks a lot more elaborated. When comparing to the original matching.py, I have some questions:
1) In general: I do not speak spanish/portuguese, so I do not understand your comments

2) In def fuzzy_it(text, patterns=None):, why didn't you change (tweaks.get('title_sort_articles', r'^(a|the|an)\s+'), ''),

3) In def get_title_tokens I added something like:

'NL', 'ebook', 'e-Book' and 'druk' as possible alternatives in


4) In def get_title_tokens I think we should need to add 'een', because the words 'de', 'het' and 'een' represent articles as used in the Dutch language.

Done, thanks for the suggestions.

The version is ready in the same link.

[Northguy]

Thanks odinokij....

[Rellwood]

Was wondering if anyone would be willing to add custom columns in the metadata comparisons, and/or having that feature used with multiple libraries?

Also, it would be nice if there was some way of setting the plugin to tag the desired duplicate for removal by comparing against set rules?

Example, if two books are found to be duplicate, and the plugin had a rule set to tag the duplicate to remove/keep if column X is filled. Making it easy to decide which to keep and which to remove without having to go through them individually.

[odinokij]

I have released a new update of the duplicates++ plugin (based on the original Find Duplicates plugin v.1.6.3 by Grant Drake). It includes:

- Finding duplicates by series and authors
- New algorithms fzzEx (extended fuzzy) for titles/series and authors
- Accelerated "Mark current group as exempt" (uncheck option "autosave new exemptions" and use menu "Save new exemptions". The new exemptions are also automatically saved if the end of the duplicated list is reached)
- Several bugs are fixed


The link is the same:
https://1drv.ms/u/s!AobKqg_eHGvlicteuckLgFlxmqE2zw


I hope it may be as useful for you as it's for me

Odinokij.

[BetterRed]

@odinokij - any reason why your version shouldn't be the 'official' version ?

BR

[odinokij]

Originally Posted by BetterRed

@odinokij - any reason why your version shouldn't be the 'official' version ?

BR

I don't know how to publish the plugin in calibre as official plugin.

The official "Find Duplicates" plugin was developped by Grant Drake but it has never been update since June 2017.

In my first version I added changes to the fuzzy algorithms to consider the articles and prepositions in other languages... (you can read above)

Greetings,
Odinokij

[davidfor]

Originally Posted by odinokij

I don't know how to publish the plugin in calibre as official plugin.

There are notes in the plugin index thread about how to do this. But, it is basically just creating a thread with the plugin in the first post and asking a mod to add it to the index.

But...

The official "Find Duplicates" plugin was developped by Grant Drake but it has never been update since June 2017.

In my first version I added changes to the fuzzy algorithms to consider the articles and prepositions in other languages... (you can read above)

As this is a modification of an existing plugin, it is a little different.

Since kiwidude basically retired from the forum, others, including myself, have been maintaining his plugins as needed. For most of the plugins, this has been just whatever changes have been needed to keep them working when there were changes in calibre. For other plugins, there has been more work done. In general, the work has been done by someone who uses the plugin and wants to keep it working. When the changes have been made, we post a beta in the thread for testing. If there are no problems, we the ask a moderator to update the released version. This seems to be working for the small number of changes that have been made to most of the plugins.

If you are interested in doing more with this plugin, I would suggest PMing kiwidude and asking him if he was OK with you taking over the plugin. He is still around but might take a few days to respond to a PM. He is happy that people are still using and maintaining them. The advantage of doing this is that you can start a separate thread and manage the release yourself.