Help With Duplicate Book Titles

[pghaworth]

I'm trying to clean up my Calibre library and merge formats. Is there any way I can tell Calibre to look for and bring up only duplicate [or similar titles], other than sorting by title?

I've tried a couple of things but nothing is getting me where I need to go yet. All help and ideas appreciated.

Thanks a bunch!

[Starson17]

Originally Posted by pghaworth

I'm trying to clean up my Calibre library and merge formats. Is there any way I can tell Calibre to look for and bring up only duplicate [or similar titles], other than sorting by title?

I find duplicate titles by opening up the metadata.db file in sqlitespy and using an sql query.

Alternatively, this will also do it:

calibre-debug -c "from calibre.library.database2 import LibraryDatabase2; db = LibraryDatabase2('/path/to/library/folder');dupes = db.conn.get('select title from books group by title having count(*) > 1;');print dupes;">dupes.txt

Copy that long line, change the path to point to your library folder and paste it into a command window (terminal window, dos box, whatever you call it) and it will produce a file in the current directory called dupes.txt with all your duplicate titles in it. It's a list of dupe titles in unicode format.

[pghaworth]

Thanks. I don't think this will get me where I need to go but maybe I'm not understanding it's full use. I was hoping to do this from inside Calibre.

If all else fails I will use this method but Calibre is such a clever program I feel sure there's something, that will bring a list of my duplicates up in Calibre, that I'm just missing or don't know how to do.

[Starson17]

Originally Posted by pghaworth

Thanks. I don't think this will get me where I need to go but maybe I'm not understanding it's full use. I was hoping to do this from inside Calibre.

Sort by title and manually looking is the only way inside the GUI. There is no "press a button to show duplicates" function. I've posted my two workarounds.

[pghaworth]

Yes, and I found your answer in another thread, much older, as well. I'm not very good with either of the options in your answer. It's a lot like reading Latin for me at this point. I think I will have to limp along in Calibre at this point unless I learn Latin. Thanks again.

[peterselizabeth]

pghaworth, have you tried selecting one book. right click, choose similar books in the context menu, then choose books by the same author. All books by that author will show in the list and its a matter of just selecting the dups and merging or deleting the ones with the same format? It may not be much better than sorting by title, but it depends on how many books you have. Good luck.

[pghaworth]

Hmmm... this may be just what I'm looking for. My list is ginormous and to go thru book by book is daunting, even sorted by titles. I'll give this a try and see if it gets me where I need to go any quicker. Thanks!

[Starson17]

Originally Posted by pghaworth

Yes, and I found your answer in another thread, much older, as well. I'm not very good with either of the options in your answer. It's a lot like reading Latin for me at this point. I think I will have to limp along in Calibre at this point unless I learn Latin. Thanks again.

Duplicates were very annoying to me. That "much older" thread preceded the Merge function, which I wrote so I could merge them once I found them.

It also preceded the autosort/automerge function I wrote to avoid creating duplicates in the first place. Note that the autosort/automerge function is located in Preferences|Add/Save|"If books with similar authors and titles found, merge automatically" and it's off by default. That means you will often create duplicates if you add two different formats for the same book. You may want to turn it on to prevent them from being formed in the first place. I seldom see them any more.

[pghaworth]

I don't want to turn it on because I want the different versions to show and be added for the time being. I want to be able to view my device versions and I have set up things so that I can differentiate.

I do have a question for you tho' that is unrelated to above. I have an old Kindle and a new one and I have versions of a given book for each device. Can I merge those two azw file records but keep the device integrity?

[Starson17]

Originally Posted by Starson17

Sort by title and manually looking is the only way inside the GUI. There is no "press a button to show duplicates" function. I've posted my two workarounds.

OK, I lied. Just last week I wrote some special code that implements the autosorting/automerge function when doing a Copy to Library. That code was released minutes ago as 0.7.17.

You can do this:

1) create an empty library
2) make sure that the autosort/automerge option is on (the one with the long name referred to above in Preferences|Add/Save)
3) select all your books and right click then Copy to Library

This should cause all of your duplicates to be merged during the transfer. You should check carefully, as I've only tested it once. I hesitated to mention it as it hadn't even been released until I checked a minute ago.

If you're happy with the merge, use the new library.

Yes, Calibre changes minute by minute.

Full disclosure: When I say "I wrote ... code" what I really mean is I submit code that works (sort of) then Kovid tells me he's "refactored" it. That's a polite way of saying he's taken my crappy code and cleaned it up so it meets the high standards he keeps for Calibre.

[Starson17]

Originally Posted by pghaworth

I don't want to turn it on because I want the different versions to show and be added for the time being. I want to be able to view my device versions and I have set up things so that I can differentiate.

If you want the dupes, why do you want to find them?

I do have a question for you tho' that is unrelated to above. I have an old Kindle and a new one and I have versions of a given book for each device. Can I merge those two azw file records but keep the device integrity?

I'm not sure what you're asking, but if you're asking about keeping two files with the same extension in one record, the answer is no. I know little about the Kindle or azw files.

[pghaworth]

LOL Good one.

Calibre is an amazing program - I continue to be impressed daily. I will never be without it again in my life. The only real problem with it, in fact, is the ID10T error. I wish Kovid could take my code and clean it up.

I'll see if I can digest your instructions and apply without trashing my library.

[pghaworth]

Originally Posted by Starson17

If you want the dupes, why do you want to find them?

If I have an epub and an azw, for instance, I want to temporarily keep them so that I can choose when one gets merged into the other. I did a lot of metadata clean-up of my records - well over 1K books - and then upgraded. So if I bring in a record with incomplete data then I can just tell it which one to merge into. I like control.

Originally Posted by Starson17

I'm not sure what you're asking, but if you're asking about keeping two files with the same extension in one record, the answer is no. I know little about the Kindle or azw files.

Yes, two records, one extension. I have identifiers for which device it goes to. I don't want a separate library so for those records I will have to have duplicate titles.

[Starson17]

Originally Posted by pghaworth

I'll see if I can digest your instructions and apply without trashing my library.

It's perfectly safe to try it, but don't give up your original library until you're sure the new one is in good shape. Remember, it will merge together duplicate books. A "duplicate book" is one which has the identical author and nearly the same title (ignoring capitalization, spaces and certain indefinite articles - "a," "an" or "the" at the beginning of the title.)

[pghaworth]

How does it decide which book to merge into which? Plus, with my different devices that have the same file name it would be really bad to merge those.