How to merge and eliminate duplicates

[clittle]

Hi,

I have a lot of duplicate books and others where the same book but different formats are listed separately.

Is there any easy way to merge formats and delete duplicates?

Thanks

[Manichean]

Select all entries of one book and right click. There should be several merge options available.

[theducks]

Originally Posted by clittle

Hi,

I have a lot of duplicate books and others where the same book but different formats are listed separately.

Is there any easy way to merge formats and delete duplicates?

Thanks

1) Select the Destination entry FIRST, then select additional entries. M (for merge with delete) or Right-click for more options.

[Starson17]

Originally Posted by theducks

1) Select the Destination entry FIRST, then select additional entries. M (for merge with delete) or Right-click for more options.

That's the manual merge method, which is usually the best. OTOH, if he has lots of these duplicates and all have approximately the same metadata, another option is to do it automatically. To do that he can turn on the autosort/automerge option in Export/Import|Adding Books, then copy the entire library into a new library. This process will check each book as it is copied into the new library and when it finds a book that has the same author and nearly the same title as a book that was previously copied, Calibre will copy the new format into the previous record. This method is not suitable for cases where the author/title differ significantly or where the metadata of the first record is worse than the metadata for later books.

[whodean]

In the automerge option method you describe, Starson, how would I determine which version would be copied?

[Starson17]

Originally Posted by whodean

In the automerge option method you describe, Starson, how would I determine which version would be copied?

It's automatic. You don't have control. The first version of each format for each book sent into the new library is kept and any duplicates of that format are ignored.

[Calliastra]

Originally Posted by Starson17

That's the manual merge method, which is usually the best. OTOH, if he has lots of these duplicates and all have approximately the same metadata, another option is to do it automatically. To do that he can turn on the autosort/automerge option in Export/Import|Adding Books, then copy the entire library into a new library. This process will check each book as it is copied into the new library and when it finds a book that has the same author and nearly the same title as a book that was previously copied, Calibre will copy the new format into the previous record. This method is not suitable for cases where the author/title differ significantly or where the metadata of the first record is worse than the metadata for later books.

Starson, I really appreciate you taking the time to help!! I have a similar problem and am not sure how to tackle it. I started out with a large number of ebooks, probably about 12-15K. I imported them into Calibre and now I have almost 40K and loads of duplicates. The problem with going down the list and deleting the one or two (or more!) extras is that the DB is really bogging down. I am a very new user, but am a programmer/software tester so I understand the lingo. Can you give me a short set of instructions and then perhaps I can techwrite them into a more complete help item? From what I've googled up, it looks like this is a common question.

Part of what I am wondering if it would be worth organizing the books properly (author, title, series) or downloading metadata or any other prework that one could do that would make the duplicate matching process more effective or streamlined.

P.S. Happy to help out in testing or other tech stuff as needed too since I am currently out of work.

[kiwidude]

Originally Posted by Calliastra

Starson, I really appreciate you taking the time to help!! I have a similar problem and am not sure how to tackle it. I started out with a large number of ebooks, probably about 12-15K. I imported them into Calibre and now I have almost 40K and loads of duplicates. The problem with going down the list and deleting the one or two (or more!) extras is that the DB is really bogging down. I am a very new user, but am a programmer/software tester so I understand the lingo. Can you give me a short set of instructions and then perhaps I can techwrite them into a more complete help item? From what I've googled up, it looks like this is a common question.

I will butt in since this is a topic of great interest to me currently. Firstly, have you read the Duplicate Detection thread in this forum? That discusses some changes and additions to Calibre we are in the process of making. Feedback on that thread as to what sounds useful or not is always welcomed (particularly as the plugin which will "find" duplicates has not been written yet and there's a few ways we can approach it).

As to "instructions", from a Calibre perspective Starson has given you what you need to do if you decide to try that approach. You just need to be aware of the implications:
- It will only find duplicates where the authors exactly match. There is no "fuzzy matching" on authors.
- You really have very little control over which version will be kept if you have duplicates of a format. As Starson says above it is done by order of "selection" - but if you are doing a bulk library all at once that "selection order" won't mean too much. You could maybe sort by date or something but unless you investigate each book one by one you won't know which version to keep and it could be pot luck. And if you were doing it one by one controlling selections, you wouldn't need Starson's approach and would just use Merge instead

Part of what I am wondering if it would be worth organizing the books properly (author, title, series) or downloading metadata or any other prework that one could do that would make the duplicate matching process more effective or streamlined.

There's a few other threads in the forum if you look around at approaches people have taken. At the moment I have my own tool outside of Calibre that does fuzzy matches of authors and/or titles, doing direct sql queries against the Calibre database. Other people have their own tools/scripts, some of which were made available. Hopefully we will have a Calibre plugin soon (I've offered to write it but anyone is welcome to beat me to it), but we need to make decisions about it before I start and that discussion should be kept to the other thread.

Certainly the 1.0 version may "only" have the exact same comparison logic Starson's automerge functionality has - of exact match on author, fuzzy on title. In which case in terms of cleanup preparation getting any author dups sorted is going to greatly increase the success of any dup search on top. If you dont want to resort to sql, just use the tag browser on the left to look down your authors list and with it's alphabetical sorting you can hopefully spot a lot of the common issues like typos, initials, spacings, abbreviations of names etc. Stuff like "E.E.Doc Smith", "E. E. 'Doc' Smith", "E. E. Smith" etc etc - rename the "wrong" author variations and get them down to one.

Then at least if you decide to try Starson's described method above (not caring for instance about which EPUB to keep if you have two of them) you are in the best position to do so.

That's my 2p for what its worth.

[Starson17]

Originally Posted by kiwidude

I will butt in since this is a topic of great interest to me currently.

As far as I'm concerned - your help is always appreciated.

Firstly, have you read the Duplicate Detection thread in this forum? That discusses some changes and additions to Calibre we are in the process of making. Feedback on that thread as to what sounds useful or not is always welcomed (particularly as the plugin which will "find" duplicates has not been written yet and there's a few ways we can approach it).

Yes, comments are welcome. The new automerge code is now in Kovid's hands, but he'll always consider improvements.

As to "instructions", from a Calibre perspective Starson has given you what you need to do if you decide to try that approach. You just need to be aware of the implications:
- It will only find duplicates where the authors exactly match. There is no "fuzzy matching" on authors.

Correct.

- You really have very little control over which version will be kept if you have duplicates of a format.

I'm not completely sure how much control you have for Copy to Library. We'd have to test it or ask Kovid. I suspect, however, that the order of the processing for Copy to Library will be according to the sorted selection order. As each selected book is processed, each format for that book is compared to the contents of the new library being constructed by the CTL code. Merging is done with the automerge code (assuming automerge is on). In that case, the first book the CTL code handles will be the final book. (Unlike the manual Merge cod I wrote, automerge has never merged metadata.)

Note that the new code in the linked thread only applies to automerge of incoming books. I did not replicate that code to automerge for Copy to Library. My suggestion to Calliastra (the OP) was to consider using Copy to Library. (In that scenario, there is always only one "identical book" since automerge in it's current form silently ignores duplicate formats. It's guaranteed to merge up all books with identical titles and similar titles. It can't currently make duplicate records with automerge on.

As Starson says above it is done by order of "selection" - but if you are doing a bulk library all at once that "selection order" won't mean too much. You could maybe sort by date or something but unless you investigate each book one by one you won't know which version to keep and it could be pot luck.

Yes.

There's a few other threads in the forum if you look around at approaches people have taken. At the moment I have my own tool outside of Calibre that does fuzzy matches of authors and/or titles, doing direct sql queries against the Calibre database. Other people have their own tools/scripts, some of which were made available. Hopefully we will have a Calibre plugin soon (I've offered to write it but anyone is welcome to beat me to it), but we need to make decisions about it before I start and that discussion should be kept to the other thread.

I assume you've seen the other duplicate finding threads I contributed to. They all have simple SQL queries for duplicate titles or duplicate author/title

Certainly the 1.0 version may "only" have the exact same comparison logic Starson's automerge functionality has - of exact match on author, fuzzy on title.

That's fine by me, but remember that the reason I didn't fuzzy match more aggressively is that automerge is AUTOmatic. An error in automerge meant a lost book format. If we are doing duplicate finding with manual review on a dialog screen with merging manually controlled to find the best format or not merge, we can be much more aggressive.

[kiwidude]

Originally Posted by Starson17

That's fine by me, but remember that the reason I didn't fuzzy match more aggressively is that automerge is AUTOmatic. An error in automerge meant a lost book format. If we are doing duplicate finding with manual review on a dialog screen with merging manually controlled to find the best format or not merge, we can be much more aggressive.

I'll reply to this in detail on the Duplicate Detection thread as to why I suggested it may be this way in the first release. There are some reasonings for my comment that I would like feedback on, but I don't want to split the discussion across threads. I absolutely agree that "fuzzier" searches on both author and title should be in at least its longer term goals. Whether such additions are required for the first version of the plugin is open for debate imho.