[GUI Plugin] Find Duplicates

[DMee]

What an amazing addition!!! Many thanks yet again :-)))

Trouble is "Similar" authors produces nothing and "Soundex" or "Fuzzy" produce loads with some errors hidden amongst lots of "cruft" I beleive you call it :-)

e.g. Arthur Charles Clark was buried 10 screens down - is it possible to manually adjust the settings to limit A.Bronte and A.Brown clashing but pick up Arthur C. Clark and Arthur Charles Clark ??

[kiwidude]

Ok, there seem to be a few things here. In the first case you are wanting to make a direct edit of books. I have made the plugin window modal deliberately, because if you were to make changes in the calibre window it will make the results displayed in the plugin window out of date. The primary use case I had in mind was where you would be perfectly happy renaming the authors via the plugin window. However in your case you had found a completely kakked metadata situation, and renaming the "A.N. Author" was not your desired action. I don't see that modality changing to be honest, I think it is an extreme situation and would cause too many headaches if I were to change it.

However your next point about that was that as you clicked cancel you found the plugin cleared the search so you could no longer see those rows to edit them outside the plugin. The intention on my part to restore whatever search the user had displayed before they ran the plugin, to ensure that data is refreshed visually. However I can see that in your situation you would instead want those results to be kept on screen. That is something I will take a look at changing - if a suer has "Show books on screen" checked, there is no issue with leaving them on screen when Close is clicked.

Your final KeyError message looks a concern as that looks a genuine bug. It looks like somehow the plugin was dealing with stale data. However to fix it I am going to need to be able to replicate it - if you or someone else comes across it again and can give me some steps it would be much appreciated.

[kiwidude]

The rules of what "Similar", "Soundex" and "Fuzzy" do are no different for Authors between the new screen and the old Find duplicate books approach.

Similar does not "do nothing" - it will find you variations like "Arthur C Clark", "Arthur C. Clark" and "Clark, Arthur C." for instance. Fuzzy is the loosest search, it tries to flatten names out by taking the author last name and the first initial of their first name, in order to catch "A. Clark" and "Arthur Clark". However as you have found it will produce a lot of false positives from this approach.

[DMee]

1) I see where you're coming from re Modal form
2) Yes, if you could leave search active then for weird cases like mine its easily fixed (see you're so good at plugins it does something you didn't plan! ;-) )
3)OK, here's what I did before getting error - dunno if recreatable.....

Plugin flags

John Doe & Jr.
Tom Smith & Iii

Authors Jr. and Iii are considered "Similar"

I Close plugin

I type Author:"Jr." or Author:"Iii" into search

I realise there are a few where the suffix has been taken as surname

I open Tag Browser - Manage Authors - search for Jr. and then Iii

In both cases I "Find" all cases and ensure the Author and Author Sort are corrected.

I close Manage Authors and rerun Find Duplicates - Metadata

Authors - Similar.... Search - boom error, close Calibre, reopen, works fine for every type of search on every item.


4) My "does nothing" wasn't slagging off plugin - its my bad shorthand for "Didn't produce any results i.e. I had no "Similar" authors, but I have loads Soundex and Fuzzy Authors :-)

5) In Find Duplicates you can set a length to the Soundex - that was what I meant by tweak in Metadata :-)

Keep up the amazing work :-)

[pelikano]

Hi, I'm using the plugin and it works great. But is there a way to split the duplicates in two groups: unicates(first found or original) and duplicates(all the other copies)
So I can delete all the duplicates at once?

[theducks]

Originally Posted by pelikano

Hi, I'm using the plugin and it works great. But is there a way to split the duplicates in two groups: unicates(first found or original) and duplicates(all the other copies)
So I can delete all the duplicates at once?

But what if the first found is not the best version

[DMee]

Originally Posted by pelikano

Hi, I'm using the plugin and it works great. But is there a way to split the duplicates in two groups: unicates(first found or original) and duplicates(all the other copies)
So I can delete all the duplicates at once?

The first may not be the best - however, once you've decided on the best (Highlight line, click V to view, repeat....), you can then click the one you want, then hold CTRL and click all the other duplicates, then rightclick, Edit Metadata, Merge books, and pick which option suits (usually the first)

If you will always pick the first merge option then pick best book, CTRL click rest and press M

[pelikano]

I'm aware of

The first may not be the best

but I can MAKE the first one the best. There are more lovely plugins to enrich the metadata. I know the procedure to merge the meta-data but I have 4196 duplicates....Althought it is a rainy sunday I prefer an automated procedure.
It should be lovely if the split function comes with the plugin...

[DMee]

The metadata isn't the problem. What if one file is a crappy OCR that hasn't worked well and the second is a beautifully formatted copy. Just because first one has better metadata doesn't mean it's the one you want to keep :-)

[pelikano]

I think in opportunities not in threaths. But I just count and sorted out 50 duplicates(all with isbn, thanks to the plugin "Extract ISBN") and the book quality was the same. I just want to get rid of the doubles.

[DMee]

:-) unfortunately there's no "which one reads best " plugin. Maybe there's your opportunity ;-)

I'm just doing same so sympathise but there's no easy way :-(

[pelikano]

Hi DMee, I'm not looking for a "which one reads best" plugin. Ik just would like to suggest a feature request for the duplicate plugin. All the functionality is already present in the plugin. The only extended functionality I need is a marking the doubles in a group or label. So I can decide not showing them in the database, check them or delete all at once.

[kiwidude]

@pelikano - to be honest I am not exactly sure what you are asking for?

Originally you asked to somehow separate the first match in each duplicate group. As others have explained I can't see how that is useful, because neither the formats nor the metadata associated with any particular book in a duplicate group can be assumed to be the one you should "keep", be it the first, last or any other. The only way to decide which one to keep is to manually investigate the associated formats and judiciously merging/deleting until you only have one book left. There are no shortcuts in this process without screwing up the quality of your library. I'm afraid you created your 4196 duplicates, so it is you that has to clean up your mess .

Then in your last post you talk about "marking the doubles in a group or label". Again I don't know what you mean by that. The plugin already "marks" each group behind the scenes temporarily for the purposes of stepping through the result groups so you can resolve them, seeing only the duplicates in that group. You can step through the results and defer decisions on on group, and you can just run the plugin again to bring the results back up. So what is it that it doesn't do?

[DMee]

@pelikano

The plugin shows all the duplicates, group by group - you tweak whether its one group at a time or every duplicate group in the settings. Once you've dealt with a group reclicking the "Find Duplicates" button on the toolbar will highlight the next group of duplicates to deal with.

[kiwidude]

Originally Posted by DMee

3)OK, here's what I did before getting error - dunno if recreatable.....

Thanks for that. I have managed to replicate it. If you remove an author that would have been a duplicate candidate and then re-run the metadata variations search it will blow up. It is because of stale data in the calibre database. I query the database to ask for a list of authors, but that will still include authors which actually aren't linked to any books any more (e.g. an author like III which you had just eliminated).

I'll PM chaley - it is something we have discussed previously and I thought had been "fixed" in such a way that I did not have to do a "cleanup" call before I do every query but obviously not. At least I can see why it is happening...