Ignore Duplicates

[anthony.burton4]

Is there any way to disable this check?

I find it a constant irritation when adding books.

Thanks,

Tony

[kiwidude]

The only way is to turn on AutoMerge in Preferences ->Adding books. Choose the behaviour you want from the dropdown as to what you want to happen when it detects a duplicate.

You will still get a dialog popping up after the merge that you cannot turn off unfortunately. I say unfortunately because it is about as useful as tits on a bull - telling you only the title of a book without the author or formats merged etc. Plus since there is no "undo" or review process the dialog is shutting the gate after the horse has bolted anyways.

Likewise if you have automerge turned off (which I would never do, as I disagree violently that duplicate matching only on title is in any way useful), then the dialog is next to useless, telling you only the title and not the author so you have no idea if it is correctly detecting a duplicate or not to make a decision to merge it. In most circumstances you are better off telling it to ignore the duplicate and create the new record - after all the risk of not having a book added to your library just because the title matches one by another author is not particularly useful imho.

You cannot turn off all duplicate checking completely, at least not without modifying the source.

[Gary_M_Mugford]

Kiwidude,

Is there a particular reason why the duplicate pre-add search doesn't list the title AND author in the result info line? My pace on adding books has slowed considerably and I do what you do. Add and then search for same author and then figure out whether I added it before (happens with various articles I've been adding from various computers. Seems I've written a lot of stuff over the years and stored backups ALL over the place [G] PLUS there are only so many variations on titles for articles from niche subjects like Bridge, whether by me or some other person).

At any rate, if you or Kovid ever decided to list the title AND author(s) ... I have more than a half-dozen books called Masquerade, for example ... it would remove one of the last remaining nits I have with the program.

Well, until the Toshiba Thrive arrives in two weeks and I have to cope with having two readers to attach, one of them brand new. [G]

[kiwidude]

@Gary - it's a question more for Kovid than for me. Most of my work around duplicates has been what I can do from a plugin without touching the Calibre source code myself.

As for why it is the way it is - only Kovid can say for sure but I suspect it is just the case that his dev time is spent on other priorities, and if someone were to submit a patch to improve it then he would consider it. Other than make suggestions (like enhancements to automerge which Starson17 did) I've not attempted to change that part of the codebase myself. As I said above I have some disagreement with a number of aspects of the behaviour, which means the scope of the changes I would make to it might involve more debate that I haven't the time to get into. However I can't see that displaying the authors for the titles it is saying are duplicates as being too controversial (just that it might involve changing a number of bits of code to enable that).

[Starson17]

Originally Posted by Gary_M_Mugford

I have more than a half-dozen books called Masquerade, for example ... it would remove one of the last remaining nits I have with the program.

Why aren't you just adding them, ignoring the Duplicates warning, then running kiwidude's duplicate finder? That' gives you lots of "hands on" control over what is considered a duplicate. If you don't want to manually review all the books with Find Duplicates, why not just turn on Automerge? If the title of "Masquerade" has the same author, it won't get merged unless it's a new format. If it's a different author, you'll get a new record.

I never found the duplicates warning to be of much use. That's why I wrote Merge and Automerge.

I suspect that Kovid isn't changing this because he probably thinks the combination of Merge, Automerge and the Find Duplicates plugin gives you all the control you need if the basic title-matching algorithm of Duplicates isn't enough.

[Gary_M_Mugford]

Starson,

While I truly appreciate the automatic tools you've created and the after-the-fact tools from KiwiDude, I've got a couple of issues that an immediate confirmation of authorship would make life easier for me. The first is due to having entered in a LOT of books with lots still lying around (I use the database for real paper books, too). But it's been awhile and my immediate memory of what's gone in, flags. I'm old. I'd like to know whether the title/author is unique before going through the after-the-fact examination. But I understand this is a sore spot with our leader and unlikely to be fixed in anything approaching the near future. I can't turn automerge on because I also use Calibre as a repository of my various articles from bridge magazines and sports publications. Not all of my work, especially in the bridge area. But I do wish to keep versions, just not multiple copies of the same version. (I adjusted articles for different publications. i.e. what I wrote for Bridge World appeared in the souvenir book for the Indian National Championships a year later in expanded form. That sort of thing). Plus there are articles like Mike Lawrence's 2 Over 1 Notes that are the not the same as MY 2 Over 1 Notes. It all gets messy.

But it's NOT your problem. You've done great work that satisfies the needs of 99.9 percent of the Calibre user-base. I've known about Kovid's touchiness on the subject and just wondered if any of you insiders knew why he has adopted that stance. KiwiDude explained as well as it can be explained. And I accept that.

GM

[Starson17]

Originally Posted by Gary_M_Mugford

While I truly appreciate the automatic tools you've created and the after-the-fact tools from KiwiDude, I've got a couple of issues that an immediate confirmation of authorship would make life easier for me.

Thanks, however, I'm not looking for appreciation - I'm just trying to understand your use case. Do you add multiple books at once, or only one at a time? I agree that telling you the author would be useful for those who rely on Duplicates and add only one book at a time, I'm just not sure why you would want to do that?

Remember, the Duplicates Warning code only spots exact match titles and it ignores authors. Moreover, it provides no individual book processing. It's all or nothing - you have to add all duplicate titles or not add them, regardless of whether the author matches or not. That's why IMHO, Duplicates is only useful when adding a single book - so you can choose what to do. Since it doesn't do any "similar" title processing, even the single book add use case is limited.

The first is due to having entered in a LOT of books with lots still lying around (I use the database for real paper books, too). But it's been awhile and my immediate memory of what's gone in, flags. I'm old.

Automerge was written to allow you to add lots of books, all at once, and when the title and author are unique, it always creates a new record. When the author is not unique and the title matches very closely (punctuation, capitalization and leading articles are ignored) it will try to add it to the matching book record, if it's a new format, and then do what you tell it to do if it can't.

I'd like to know whether the title/author is unique before going through the after-the-fact examination.

What will you do if the title/author is unique vs. not unique? Specify if you are adding multiple books or only single books. Specify if you are adding different formats or only one format. Perhaps examples would help.

But I understand this is a sore spot with our leader and unlikely to be fixed in anything approaching the near future.

I'm not sure why you say that. AFAIK, Kovid is neutral on this, although it's probably low priority to change if he thinks the existing tools meet the user needs. That's why I'm trying to figure out how they fail to meet your needs. I understand that the Duplicates Warning is of limited value. That's just because it was an early bit of legacy code that has been superseded by Automerge, Merge and Duplicate Finder code.

I can't turn automerge on because I also use Calibre as a repository of my various articles from bridge magazines and sports publications. Not all of my work, especially in the bridge area. But I do wish to keep versions, just not multiple copies of the same version. (I adjusted articles for different publications. i.e. what I wrote for Bridge World appeared in the souvenir book for the Indian National Championships a year later in expanded form. That sort of thing). Plus there are articles like Mike Lawrence's 2 Over 1 Notes that are the not the same as MY 2 Over 1 Notes. It all gets messy.

Explain a bit more on this. The "Mike Lawrence's 2 Over 1 Notes that are the not the same as MY 2 Over 1 Notes" books would never be processed with Automerge. They have different authors. (it will appear in Duplicates Warning.) Suppose you add a new version of an article that has you as the author. Suppose Duplicates Warning triggers and tells you that you are the author of the existing record. (It won't if you have "version 1" etc. or "Written for ..." in the title. - It won't trigger Automerge in that case either.) What will you do with the information from Duplicate Warning? There's no way to look at the book. Do you just remember different articles and do different things based on your memory or does the title have some info about the version?

I've known about Kovid's touchiness on the subject and just wondered if any of you insiders knew why he has adopted that stance. KiwiDude explained as well as it can be explained. And I accept that.

I'm still not sure what "touchiness" or "stance" you're referring to. I'm familiar with the code in that area, and I may have some time in a week or so to get up to speed and possibly adjust it, I'm just trying to figure out the use case. I can't imagine that Kovid would be touchy about this kind of change.

[unboggling]

What would make me ecstatic is an unambiguous option in automerge to *always* create a new record in all cases, formats same or different, authors and/or titles matching or not.

[kovidgoyal]

Originally Posted by unboggling

What would make me ecstatic is an unambiguous option in automerge to *always* create a new record in all cases, formats same or different, authors and/or titles matching or not.

Turn off automerge.

[unboggling]

Originally Posted by kovidgoyal

Turn off automerge.

I do. It's always off. I've been losing books anyway. Tracked it down to this so far: if authors and titles fuzzily match, they are the same format, and they have the same isbn, one of them disappears during a save-out/add-in operation. That is with saying yes to the initial dialog confirmation to add duplicates, with automerge off.

Edit: That is one reason I've been spending a lot of time ensuring isbn's are exactly correct per edition lately. Otherwise, for my purposes, I'd probably totally ignore isbns since they are not otherwise useful to me.

[kovidgoyal]

If you have automerge off and select add duplicates anyway during an add operation, the files are *always* added. If you feel they are not, open a bug report and attach a file that on being added twice to calibre results in one book entry with automerge off.

[unboggling]

Okay. What I'll do first when I have time is set up an experiment with debug on and see what happens in various cases. But I doubt that behavior would show as a bug. I had just assumed there was something else in calibre that didn't like identical isbns for identical formats and that was normal behavior, so I was trying to work around it.

Edit: More precisely, something that didn't like exact title and author matches for the same format. Which is what first led me to trying to use more precise ISBNs as workaround.

[Starson17]

Originally Posted by unboggling

I do. It's always off. I've been losing books anyway. Tracked it down to this so far: if authors and titles fuzzily match, they are the same format, and they have the same isbn, one of them disappears during a save-out/add-in operation. That is with saying yes to the initial dialog confirmation to add duplicates, with automerge off.

That's a bug if it can be recreated, but it's got very strange symptoms that don't match any code I'm familiar with. The only "fuzzy" title/same author testing during Adding Books is in Automerge, but Automerge ignores ISBN.

Can you advise how you are doing the "save-out/add-in operation?" Is it a Save to Disk, a Send to Device, a Connect to Folder, or a Copy to Library operation?

[unboggling]

Save to disk, opfs included. Add to different library using internal metadata not filename. If I remember correctly, it happened with exact author/title matches (I don't know why I said fuzzy before, sorry, that was a broadening assumption). But it happened often enough that it propelled me toward the precise isbn solution. That was months ago. Then a couple weeks ago it dropped an exact title/author/isbn/format match during save from libraryA then add to libraryB, automerge off, add dupes yes. I'll try to replicate it again and get back to you with exact documentation.

Edit. The 2 identicals with the isbn match were both in Library A, only one showed up in Library B.

[Starson17]

Originally Posted by unboggling

Save to disk, opfs included. Add to different library using internal metadata not filename. If I remember correctly, it happened with exact author/title matches (I don't know why I said fuzzy before, sorry, that was a broadening assumption). But it happened often enough that it propelled me toward the precise isbn solution. That was months ago. Then a couple weeks ago it dropped an exact title/author/isbn/format match during save from libraryA then add to libraryB, automerge off, add dupes yes. I'll try to replicate it again and get back to you with exact documentation.

The isbn part is what confuses me. I'm not aware of any isbn comparison matching going on during any Adding Books processing. Automerge uses only fuzzy title and exact author matching. Duplicates Warning uses only exact title matching. It doesn't matter where you get the metadata - filename or internal or opf override - it's what's in the metadata that's important and whether Automerge is off/on.