[GUI Plugin] Find Duplicates

[Merischino]

thanks kiwidude, i will. (I did call that merge discussion in my original post ot... sorry for taking us off topic)

[Sidetrack]

I'm sure this has been asked before, but what are the chances of adding the ability to check for series spread out across two libraries?

[LadyKate]

A question about this plugin. The binary search, does that ignore the bookmark file that is inserted when the book is opened and looked at?
Is there any chance of a duplicate finder that would compare the book part but not the tags and/or bookmark file?

[kiwidude]

@Sidetrack - if it has been asked for before then it was a long time ago, then again I do have goldfish memory and been a long time since I updated this plugin. It's an interesting idea, I will have a think about it the next time I look at this plugin.

@LadyKate - a binary search is an *exact* match type search on the underlying files, so yes bookmarks screw things up. Personally I turned off calibre bookmarks a long, long time ago because (a) I don't read books using the calibre viewer, and (b) I was getting annoyed with the contents and timestamps changing every time I opened one up to look at it.

All I can suggest is turning off the feature (in the Preferences in the calibre ebook viewer application itself), and then using the Quality Check and Modify ePub plugins to find ePubs with Bookmarks and remove the bookmarks respectively.

[halfcore]

kiwidude,

Is there a way to do a Title/Author search between two libraries that will find all the books from the authors in the 2nd library and mark all tittles by those authors in the first. I tried the library search with exact author search and Ignore title but it only give me a log of match. I would like to mark those matched books as well for removal or move to another library. FYI, I am trying to remove all the erotic books from the main library for general consumption.

Thanks,

halfcore

[AnselmD]

to be able to find duplicate covers would be great.

[theducks]

Originally Posted by AnselmD

to be able to find duplicate covers would be great.

that would take Image analysis (and a hefty CPU )
All covers are stored inside calibre with the same name: cover.jpeg

switch to Grid view with the Details and Tag Browser hidden for maximum # of covers showing. Page through the Library.
When you see a candidate, select the image, Tap I for information or turn off the Grid (and decide )

[Sabardeyn]

With regards to duplicate covers...

I have no idea what would be involved, but finding approximate similarity might be acceptable for images so something like an MD5 checksum comparison might work. The results wouldn't be error free, particularly for a series of books with a rigidly enforced layout/logo, but it would substantially cut down on the number of possible duplicates. Eyeballing a couple thousand covers is not reasonable.

Of course, it also depends on what kind of duplication the OP is looking for. Either multiple books with the same cover image, or multiple cover images within a single ebook.

(Personally this feature would be of no use to me as I explode anthologies and short story collections. I prefer story titles over book titles. So I am going to have multiple books with the same cover.)

[BetterRed]

Originally Posted by AnselmD

to be able to find duplicate covers would be great.

@AnselmD - I use a defunct program called DupDetector from Prism Systems, its a bit clunky UI wise, but its the best free duplicate image finder I know of. There are a few programs out there with the same name - make sure you get the one from Prism.

Its been a couple of years since I last looked for similar programs, nothing came close, including some add-ons (plugins) to high end expensive commercial products.

I don't have it installed on my 'calibre' system so I cant test it on a library right now. But I've used it on friends systems to dedup tens of thousands of images scattered around their disk drives. It analyses the image properties not the file properties, edge analysis, shape detection, colour histograms that sort of thing. And it detects mirrors, flips, monochrome etc.

The only thing I'm unsure of is whether it could handle 1,000s of images all with the same name, theoretically it shouldn't care but... So don't use it or anything like it without doing a backup first - because they all have options to delete the duplicates etc.

As theducks said these things ain't super fast, but the last system I deduped was a fairly modest Toshiba laptop. It probably took me a day, it it had tens of thousands of high res images (8-12MP+) on it. I can't recall how much disk space I recovered, but along with the crud and bloat I removed it should mean the owner will get another couple years use from it.

BR

[AnselmD]

Originally Posted by AnselmD

to be able to find duplicate covers would be great.

Thanks to all to your answers.

With your answers i have an idea how to solve my special use case.

I importet the ebooks from gutenberg. The most of them has an identical cover image (thousands). I wanted to find these one automtically and start to find better covers or at least delete this covers. I do not like to do it manually.

So i think i will do it like this: find all binary duplicates with some external tool. Delete all the duplicate covers. After this, i can find all books which has "no cover" with calibre and do whatever i like.

If the images are not binary duplicate, i will try to install Dup Detector with wine at Linux.

Thnx a lot!!

[theducks]

Originally Posted by AnselmD

Thanks to all to your answers.

With your answers i have an idea how to solve my special use case.

I importet the ebooks from gutenberg. The most of them has an identical cover image (thousands). I wanted to find these one automtically and start to find better covers or at least delete this covers. I do not like to do it manually.

So i think i will do it like this: find all binary duplicates with some external tool. Delete all the duplicate covers. After this, i can find all books which has "no cover" with calibre and do whatever i like.

If the images are not binary duplicate, i will try to install Dup Detector with wine at Linux.

Thnx a lot!!

KISS
Leave the details pane open.

Use Grid View and eyeball Mk I,

Get the Search the internet PI. Right click (assumes the PI was set to use the Context Menu) on the 'bad image'
I use the Search Google for Images
When I find a 'good one', I just drag it over the old one on details

Having a 2nd Monitor makes this a LOT easier

[BetterRed]

@kiwidude - for your information/attention - https://www.mobileread.com/forums/sho...d.php?t=224527 - whatever.

ETA : Good news at https://www.mobileread.com/forums/sho...6&postcount=14

BR

[JohnnyBook]

Originally Posted by AnselmD

Thanks to all to your answers.

With your answers i have an idea how to solve my special use case.

I importet the ebooks from gutenberg. The most of them has an identical cover image (thousands). I wanted to find these one automtically and start to find better covers or at least delete this covers. I do not like to do it manually.

Thnx a lot!!

I have been using a program for years called "uniquefiler" it is designed for Images, but will do all file types.

It is Lightning fast at exact matches (Binary duplicates), and Duplicate file names, and will do image comparisons also, finding close matches (typically different sizes of the same picture or cropped pictures or pictures with slight variations) with the ability to set how close % you want to search for.

Its also probably defunct ware, but I am pretty sure it is availible out there somewhere.

EDIT: the website is still there. http://www.uniquefiler.com/ I use V1.4

[AnselmD]

Originally Posted by theducks

KISS
Leave the details pane open.

Use Grid View and eyeball Mk I,

Get the Search the internet PI. Right click (assumes the PI was set to use the Context Menu) on the 'bad image'
I use the Search Google for Images
When I find a 'good one', I just drag it over the old one on details

Having a 2nd Monitor makes this a LOT easier

I needed some time to understand "eyeball Mk I"
I tried Internet PI, it is nice to find Images. But for this special case it is too much work. I put ~ 15000 books from Project Gutenberg into calibre. Maybe it better to download all in bulk and Use Grid View and eyeball Mk I to throw away the bad ones. Or generate the covers....

[AnselmD]

Originally Posted by JohnnyBook

I have been using a program for years called "uniquefiler" it is designed for Images, but will do all file types.

It is Lightning fast at exact matches (Binary duplicates), and Duplicate file names, and will do image comparisons also, finding close matches (typically different sizes of the same picture or cropped pictures or pictures with slight variations) with the ability to set how close % you want to search for.

Its also probably defunct ware, but I am pretty sure it is availible out there somewhere.

EDIT: the website is still there. http://www.uniquefiler.com/ I use V1.4

This seems to be nice tool, i will try it (but for another purpose ).
BTW, i found 2 groups of exact images. As i can remember the 1st group had a few hundred duplicates, the 2nd more than 9000.