Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Plugins

Notices

Reply
 
Thread Tools Search this Thread
Old 03-02-2023, 10:56 PM   #1081
komali_2
Junior Member
komali_2 began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Jun 2021
Location: Taiwan, sovereign nation and #1
Device: kindle pw4
Quote:
Originally Posted by random733 View Post
Is there any working version with the auto merging feature? komali2s version (1.8.5) here unfortunately does not work with the latest Calibre version (linux and windows) - keeps requiring a reboot without the plugin being fully installed.

I understand the concerns about the dangers to the integrity of the library. But I think you can manage the risk with a backup. And in the end, everyone is responsible for actually using this function (which is also explicitly declared as dangerous).

Hey, sorry about that, I got caught up in some big work projects.

It's ok that kiwidude doesn't want to include a fork's functionality, that's very normal in the open source community for people to have strong opinions - such things are required for really cool projects like theirs to get created for free. I don't actually disagree with their logic either, combining duplicates carries high risk of screwing up your ebooks and metadata and losing copies! You have to be careful.

But, there's still usecases. I had one, so I made the tweak to the plugin. Others have usecases too. I'm working right now to get it working with up to date versions of calibre. There was someone else that kindly emailed me to mention another bug that I'm looking at.

As for @kiwidude, don't worry, I won't be making a PR, though i appreciate you linking https://github.com/kiwidude68/calibr...icates-v1.10.7 your github repo, there's lots of really good information there. The primary motivation for me to create my tweak was to try doing some open source work outside of web dev, thanks for creating resources to help people do that! As your code is GPL-3, I'm going to change my repo to be a fork of yours with the same license. I really like GPL-3 as well.
komali_2 is offline   Reply With Quote
Old 03-03-2023, 01:55 AM   #1082
komali_2
Junior Member
komali_2 began at the beginning.
 
Posts: 6
Karma: 10
Join Date: Jun 2021
Location: Taiwan, sovereign nation and #1
Device: kindle pw4
Sorry to double reply, but it looks like there's a time limit on editing a post.

The fork of kiwidude's excellent repo for finding duplicates that I made a while back has been updated to (hopefully) work on newer versions of Calibre, and also includes the massive improvements kiwidude made to their code in the meantime. https://github.com/komali2/calibre-find-duplicates

Remember that this is only possible because of all of their hard work, credit goes to them!
komali_2 is offline   Reply With Quote
Old 03-03-2023, 03:18 AM   #1083
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 20,763
Karma: 27405072
Join Date: Mar 2012
Location: Sydney Australia
Device: none
Moderator Notice

@komali_2: See this regarding edits and PMs.

BR

Last edited by BetterRed; 03-03-2023 at 01:36 PM.
BetterRed is online now   Reply With Quote
Old 04-07-2023, 11:06 AM   #1084
feuille
Connoisseur
feuille will become famous soon enoughfeuille will become famous soon enoughfeuille will become famous soon enoughfeuille will become famous soon enoughfeuille will become famous soon enoughfeuille will become famous soon enough
 
Posts: 55
Karma: 666
Join Date: May 2020
Location: Germany
Device: android smartphone + tablet
Key error in module book_algorithms.py

Hello kiwidude,

when Calibre was running in debug mode today, I noticed that an key error is reported in the book_algorithms.py module:

File "calibre_plugins.find_duplicates.book_algorith ms", line 295, in _find_candidate_by_file_size
KeyError: 'mtime'

Statement 295 in version (1, 10, 7) is: mtime = stat_metadata['mtime'].

However, the plugin continued to run and returned a result.

Regards,
feuille is offline   Reply With Quote
Old 04-14-2023, 05:52 AM   #1085
kiwidude
Calibre Plugins Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,661
Karma: 2162064
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
Quote:
Originally Posted by feuille View Post
Hello kiwidude,

when Calibre was running in debug mode today, I noticed that an key error is reported in the book_algorithms.py module:

File "calibre_plugins.find_duplicates.book_algorith ms", line 295, in _find_candidate_by_file_size
KeyError: 'mtime'

Statement 295 in version (1, 10, 7) is: mtime = stat_metadata['mtime'].

However, the plugin continued to run and returned a result.

Regards,
Hi fueille,

The only way I can see this happening is if you somehow have invalid metadata in your library for a book format. Perhaps you deleted a book format from disk, but calibre still thinks it exists.

I will tweak the code so the exception won't appear in future, but with the way it is written it would just skip that book format anyways which is what it should do, it is just extra noise in the log.
kiwidude is offline   Reply With Quote
Old 04-14-2023, 05:59 AM   #1086
kiwidude
Calibre Plugins Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,661
Karma: 2162064
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
Find Duplicates v1.10.8 Released

Release Notes
https://github.com/kiwidude68/calibr...icates-v1.10.8
kiwidude is offline   Reply With Quote
Old 06-11-2023, 09:11 PM   #1087
ownedbycats
Custom User Title
ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.
 
ownedbycats's Avatar
 
Posts: 8,963
Karma: 62040409
Join Date: Oct 2018
Location: Canada
Device: Kobo Libra H2O, formerly Aura HD
Question: Is it possible to run a binary compare on just a few selected books, instead of the entire library?
ownedbycats is online now   Reply With Quote
Old 06-12-2023, 05:17 AM   #1088
kiwidude
Calibre Plugins Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,661
Karma: 2162064
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
Quote:
Originally Posted by ownedbycats View Post
Question: Is it possible to run a binary compare on just a few selected books, instead of the entire library?
Yeah just make a virtual library with those books on it, Find Duplicates respects virtual libs.
kiwidude is offline   Reply With Quote
Old 07-22-2023, 03:22 PM   #1089
Wiggo
Leftutti
Wiggo ought to be getting tired of karma fortunes by now.Wiggo ought to be getting tired of karma fortunes by now.Wiggo ought to be getting tired of karma fortunes by now.Wiggo ought to be getting tired of karma fortunes by now.Wiggo ought to be getting tired of karma fortunes by now.Wiggo ought to be getting tired of karma fortunes by now.Wiggo ought to be getting tired of karma fortunes by now.Wiggo ought to be getting tired of karma fortunes by now.Wiggo ought to be getting tired of karma fortunes by now.Wiggo ought to be getting tired of karma fortunes by now.Wiggo ought to be getting tired of karma fortunes by now.
 
Wiggo's Avatar
 
Posts: 415
Karma: 1153449
Join Date: Feb 2019
Location: Bavaria
Device: iPad Pro, Kobo Libra 2
Quote:
Originally Posted by capink View Post
Unfortunatly, this is not possible. Custom columns in calibre do not support ampersands as a separator. I opened a bug report a while back about this issue, but it was not addressed because of technical difficulties.



The solution for your problem is to do away with ampersands as separator and fall back to using commas. You can do this by modifying the template function as below

Code:
def evaluate(self, formatter, kwargs, mi, locals, val, col_name, user_cat_prefix):
    new_val = ''
    if hasattr(mi, '_proxy_metadata'):
        all_cats = mi._proxy_metadata.user_categories
        cats = {k:v for k,v in all_cats.items() if k.startswith(user_cat_prefix)}
        SEP = mi.metadata_for_field(col_name)['is_multiple'].get('list_to_ui', '')
        new_val = set()
        if SEP:
            val_ = val.split(SEP)
        else:
            val_ = [val]  
        for user_cat, v in cats.items():
            repl = user_cat.lstrip(user_cat_prefix)
            for user_cat_item, src_cat in v:
                if src_cat == col_name:
                    for item in val_[:]:
                        if item == user_cat_item:
                            new_val.add(repl)
                            val_.remove(item)
                        else:
                            new_val.add(item)
        if new_val:
            return ', '.join(list(new_val))
    return val
Note, however, that this will break author names with commas in them.
Unfortunately the problem is back. Two names separated by "&" are treated as one again

Edit: Nevermind. I swapped to this method, which works a bit better in my eyes.

https://www.mobileread.com/forums/sh...77#post4333077

Last edited by Wiggo; 07-23-2023 at 05:19 AM.
Wiggo is offline   Reply With Quote
Old 09-12-2023, 02:02 AM   #1090
kjdavies
Zealot
kjdavies is no e-book dilettante.kjdavies is no e-book dilettante.kjdavies is no e-book dilettante.kjdavies is no e-book dilettante.kjdavies is no e-book dilettante.kjdavies is no e-book dilettante.kjdavies is no e-book dilettante.kjdavies is no e-book dilettante.kjdavies is no e-book dilettante.kjdavies is no e-book dilettante.kjdavies is no e-book dilettante.
 
Posts: 111
Karma: 53342
Join Date: Jun 2013
Device: Sony PRS-600
Quote:
Originally Posted by kiwidude View Post
By comparison, with this plugin I can test 40000 books in under a second and once my exemptions are in place any future comparisons will take negligible time to perform and maintain.
Incidentally, I've taken to updating my files with a couple of new 'identifiers'.

One is SHA, the other is called 'CRC' but lies a little (it's actually CRC32-<type>-<size>). Your lovely plugin here makes it very easy to reliably dedupe based on this information.

I still use a binary dupe check on occasion to look for files with different extensions (I found one set that had 'XML' and 'SRD' extensions -- one was the actual format, the other indicated the data set), but just having this is very handy.

I can then, in a dedupe cleanup library, present the 'CRC identifier' in the tag browser. They're grouped by first character, giving me 16 groups of CRC values to quickly skim looking for 'more than 1' entries and cull. It's a little more convenient for me than the options built into the plugin.
kjdavies is offline   Reply With Quote
Old 01-24-2024, 04:37 AM   #1091
ownedbycats
Custom User Title
ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.ownedbycats ought to be getting tired of karma fortunes by now.
 
ownedbycats's Avatar
 
Posts: 8,963
Karma: 62040409
Join Date: Oct 2018
Location: Canada
Device: Kobo Libra H2O, formerly Aura HD
How does "Similar" in Find Metadata Variations work? I can't remember the last time I even got a match with it, so I'm not even entirely sure it does work.
ownedbycats is online now   Reply With Quote
Old 01-24-2024, 10:36 PM   #1092
kiwidude
Calibre Plugins Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,661
Karma: 2162064
Join Date: Oct 2010
Location: Australia
Device: Kindle Oasis
Quote:
Originally Posted by ownedbycats View Post
How does "Similar" in Find Metadata Variations work? I can't remember the last time I even got a match with it, so I'm not even entirely sure it does work.
As per the description on the dialog. For a similar title it removes punctuation characters like \(){}<>'";,:#-._ and certain common words from the title like a/the/an

For similar authors it strips initials, punctuation and ignores order of the names so John Smith and Smith John will match.
kiwidude is offline   Reply With Quote
Old 02-11-2024, 01:44 PM   #1093
maripopa
Junior Member
maripopa began at the beginning.
 
Posts: 2
Karma: 10
Join Date: Feb 2024
Device: kindle
Duplicate finder (binary compare)

Hello guys,
I am using calibre as an e-book/e-documents for many years.
Now I am facing a new challenge:
I merged two libraries (medium size, both). The new resulted library contains more duplicates than I expected initially.
The problem:
I run binary compare to find real doc duplicates (without option to delete duplicates formats during operations). The result is a quite big list/groups.

Can I save the result for processing the metadata/formats merge AFTER, without need to rerun the duplicate finder again (my this case is a long operation (the library is medium size 500GB and ~200k entries))

Any help/suggestion welcomed
Thanks for your time!
maripopa is offline   Reply With Quote
Old 02-11-2024, 03:23 PM   #1094
BetterRed
null operator (he/him)
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 20,763
Karma: 27405072
Join Date: Mar 2012
Location: Sydney Australia
Device: none
Quote:
Originally Posted by maripopa View Post
Can I save the result for processing the metadata/formats merge AFTER, without need to rerun the duplicate finder again (my this case is a long operation (the library is medium size 500GB and ~200k entries))
How about putting the duplicates into a Reading List ==>> plugin

BR
BetterRed is online now   Reply With Quote
Old 02-12-2024, 04:22 AM   #1095
luciaisacat
Enthusiast
luciaisacat began at the beginning.
 
luciaisacat's Avatar
 
Posts: 37
Karma: 10
Join Date: May 2012
Device: android
Quote:
Originally Posted by maripopa View Post
Can I save the result for processing the metadata/formats merge AFTER, without need to rerun the duplicate finder again (my this case is a long operation (the library is medium size 500GB and ~200k entries)
In these cases, I just add a temporary tag to the resulting titles so that I could call it back when I wish to work on those entries. Too simple?
L
luciaisacat is offline   Reply With Quote
Reply

Tags
cross library duplicates, in library duplicates

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[GUI Plugin] Generate Cover kiwidude Plugins 829 07-14-2024 06:58 AM
[GUI Plugin] Quality Check kiwidude Plugins 1195 07-08-2024 02:59 AM
[GUI Plugin] View Manager kiwidude Plugins 415 05-11-2024 03:28 AM
[GUI Plugin] Open With kiwidude Plugins 403 04-01-2024 08:39 AM
[GUI Plugin] Plugin Updater **Deprecated** kiwidude Plugins 159 06-19-2011 12:27 PM


All times are GMT -4. The time now is 06:28 PM.


MobileRead.com is a privately owned, operated and funded community.