[GUI Plugin] Find Duplicates

[kiwidude]

The User Category when clicked on is a search, so you can use that as the basis for a restriction. However you still need to do that extra step of clicking on the "Restrict to" dropdown so that any searches that plugins like Find Duplicates and Quality Check do are restricted to just those books in that "search".

[Philosopher]

Ah yes - now I see.

Thanks so much. Works like a charm.

[drMerry]

Just another request.
I'd like to show all duplicates with a restriction on group-level

I'd like to find duplicate books with VARIOUS formats.

At this moment I do a duplicate search, restrict to all duplicates and search for:

(formats:pdf and not formats:epub) or (formats:epub and not formats:pdf)

This gives me all books that do have an epub and no pdf (or vise versa).
I than can merge all duplicates without loosing books.

The problem is, I have to do this on all my duplicates, so this search will also show the duplicates that only contain pdf or only epub.

Is there a way to show all duplicates and add a search-restriction on the group/duplicate-level
(in that case I can also look for books with explicit title differences e.g.

title:- or not title:-

)

All these restrictions are restrictions that do not make sense in advance, but can be very use-full to select some specific duplicates.

[kiwidude]

@drMerry - you are back to discussing merging again, aka "filters" and any other number of times you have sneakily tried to broach this topic...

I'm going to have to modify this plugin at some point soon to address the keyboard shortcut changes that need to be made after Kovid's (welcome) additions in Calibre aroung that. I'm not promising anything about the above but I will take a look into it.

[Starson17]

Originally Posted by drMerry

At this moment I do a duplicate search, restrict to all duplicates and search for:

(formats:pdf and not formats:epub) or (formats:epub and not formats:pdf)

This gives me all books that do have an epub and no pdf (or vise versa).
I than can merge all duplicates without loosing books.[/CODE])

All these restrictions are restrictions that do not make sense in advance, but can be very use-full to select some specific duplicates.

I'm not sure I fully understand what you are trying to do here, but if you want to merge without losing books, are you aware of this type of search:

formats:#1

It will find all book records that have a single format. I use it often with

formats:pdf

to find all entries that have only a single pdf entry, etc.

[kiwidude]

@Starson17 - I was going to suggest that but it would still only be a subset of what he wants.

What he wants to do is find duplicate pairs that he can immediately merge without thinking. So a book that has PDF and a book that has EPUB can be merged.

However two books which both have EPUB cannot be merged without figuring out which wins. So the formats:#1 would not work for him in this situation

Also it would not catch say a book with RTF, PDF and a book with EPUB - these can be safely merged as well, even though the count of their formats differ.

[Starson17]

Originally Posted by kiwidude

What he wants to do is find duplicate pairs that he can immediately merge without thinking. So a book that has PDF and a book that has EPUB can be merged.

Unless one has good metadata and one has bad metadata

I understand the issue. He has the same problem with using:

(formats:pdf and not formats:epub) or (formats:epub and not formats:pdf)

Two books with only a pdf format and no others will both show up and can't be merged without closer inspection. I was just commenting on the form of his search, which would get pretty long if he starts adding more formats. I wasn't sure if he was familiar with the "#" search for is_multiple fields which can be useful for searches like that.

I wasn't suggesting a solution, as there currently isn't one AFAIK. To find books that don't have overlapping formats, you have to use a search that simultaneously knows about the formats in multiple book records (select "A" if this is true of "B"), but the search system only looks at one book record ("A") at a time.

[kiwidude]

Indeed, so given that the search system cannot meet his needs, he really only has two options I can see:
(1) Extend the Find Duplicates plugin to filter the results
(2) Give up on the idea of trying to filter the results, and instead write a plugin that does more intelligent merging.

And I totally agree that merging formats just because they are of different types regardless of their source is a recipe for a library mess. Still, some users really don't care that much. It isn't the way I personally manage my library which is why I am only lukewarm to the idea, but there are people out there who don't share my philosophy on content quality. Which is completely fair enough given how time consuming it can be.

[Starson17]

Originally Posted by kiwidude

Indeed, so given that the search system cannot meet his needs, he really only has two options I can see:
(1) Extend the Find Duplicates plugin to filter the results
(2) Give up on the idea of trying to filter the results, and instead write a plugin that does more intelligent merging.

Or (3) Extend the search system.

I've occasionally resorted to full SQL searching in the metadata.db file to solve tricky search problems.

With respect to 1), the Find Duplicates plugin takes a large group of records, and breaks it into multiple sub groups, some of which may overlap. A "filter" option would preferably be able to do the same thing - break one of the first groups down into one or more additional subgroups that may overlap. (It's not really "filtering" at this point, but finding matching groups within the results.) If there are four books that have fuzzy matched author-titles from a FD search, a) may have only a PDF, b) may have an EPUB and a MOBI, c) may have an EPUB and a TXT and d) may have only a MOBI. One subgroup is a), b). Another is a), c), d). Either could be merged without format loss. Add in a fifth with only an EPUB and the number of groups escalates.

With respect to 2) and 3), one needs the FD output anyway. Both provide advantages beyond the Find Dupes merger situation.

[drMerry]

Originally Posted by kiwidude

@drMerry - you are back to discussing merging again, aka "filters" and any other number of times you have sneakily tried to broach this topic...

Hmm,

maybe I need to look better to earlier requests I did myself and the reactions before I request for more

@ all other reactions.
Thank you all for thinking about this.

The case of content merging is no point, because I have own created identifiers (I have a 'procedure' to add this info to the title when scanning for duplicate titles and remove them afterwards.) to separate the content. So, if a book is said to be duplicated on title, I know it is also on content base.

While the plugin-creation is just volunteer work, and time-consuming, take your time or even don't implement it at all.
It are just suggestions I would use often if available. But I still love the current functionality.

[drMerry]

Got a nice bug.

I selected show all at once
highlight authors in tag browser
unchecked sort by number of dups (it was checked, so really UNCHECKED it)
Changed search mode from similar title+identical author to ignore title + similar author.

This last step was done by selecting ignore title (similar author is automatically selected at that moment).

My results are in the screenshot????

No duplicates searching ignore title and identical author????? I'm sure this should give results in the first place, but I asked similar authors, no identical....

[kiwidude]

@drMerry - nice spot. I've made the fix, but the release can wait until I figure out the keyboard shortcut changes.

[drMerry]

Originally Posted by kiwidude

@drMerry - nice spot. I've made the fix, but the release can wait until I figure out the keyboard shortcut changes.

Quickly done
It's not a critical bug. If the next release also contains the shortcut changes, it makes a nice update at once. (new major number?)

[kiwidude]

Changes in this release:

• Fix bug for when switching to an ignore title search where author search was previously set to ignore.
• Remove customisation of shortcuts on tab, to use Calibre's centrally managed shortcuts instead.

Requires Calibre 0.8.18. If you have previously assigned shortcuts for features in this plugin you will have to reassign them after upgrading.

[ElMiko]

What are the search parameters that I should be using (if this is even possible) for Find Duplicates to flag the following duplicates:

title:                                        author:
Cotton Malone 06 - The Emperor's Tomb         Berry, Steve
The Emperor's Tomb                            Steve Berry

As always, thanks!