Library full of Duplicates

[hotwired]

I have 3-4 of each book with a number in parenthesis after it. For example in my Stephen King folder I might have
A Face in the Crowd (869)
A Face in the Crowd (2)

Approximately 85% of my library is duplicates like this basically from sloppy "quick" conversion using Calibre (my sloppines, not Calibres). However, when using the find duplicates plugin, and trying every parameter possible, it shows NO DUPLICATES. I tried ccCleaner and that finally found them all but it's only finding them by folder. Only ONE folder has .mobi file in it while the DUPLICATE folder has .epubs so I can't just check off random duplicates in cccleaner because I might be deleting the only folder containing the .mobi which is my preferred format.

Thanks for any tips or alternate pieces of software ... I could take a week off work and do alot of "point, click and delete" but I'd rather not if I don't have to! Thanks all!!

[DaltonST]

Originally Posted by hotwired

I have 3-4 of each book with a number in parenthesis after it. For example in my Stephen King folder I might have
A Face in the Crowd (869)
A Face in the Crowd (2)

Approximately 85% of my library is duplicates like this basically from sloppy "quick" conversion using Calibre (my sloppines, not Calibres). However, when using the find duplicates plugin, and trying every parameter possible, it shows NO DUPLICATES. I tried ccCleaner and that finally found them all but it's only finding them by folder. Only ONE folder has .mobi file in it while the DUPLICATE folder has .epubs so I can't just check off random duplicates in cccleaner because I might be deleting the only folder containing the .mobi which is my preferred format.

Thanks for any tips or alternate pieces of software ... I could take a week off work and do alot of "point, click and delete" but I'd rather not if I don't have to! Thanks all!!

Hotwired:

The Find Duplicates plug-in uses metadata.db. Your library's metadata.db apparently no longer reflects reality due to (forbidden) manual changes to the the library's OS directory structure and contents.

Try this:

[1] Stop manually changing the library's directory structure and contents.

[2] Click Library > Library Maintenance > Check Library and then let Calibre thoroughly check your library, show you what is wrong, and then fix it. Repeat this step as many times as needed until your library is "clean".

[3] Using Bulk Edit Metadata for 100% of your books, "Automatically Set Author Sort", "Change Title to Title Case", and then finally "Update Title Sort".

[4] Using the Quality Check plug-in, standardize the Author Names as FN LN or LN,FN (your choice), but make all of the books comply with your choice.

[5] Using the Quality Check plug-in, standardize the Author Initials.

[6] Run Find Duplicates again.


DaltonST

[JSWolf]

Originally Posted by hotwired

I have 3-4 of each book with a number in parenthesis after it. For example in my Stephen King folder I might have
A Face in the Crowd (869)
A Face in the Crowd (2)

Approximately 85% of my library is duplicates like this basically from sloppy "quick" conversion using Calibre (my sloppines, not Calibres). However, when using the find duplicates plugin, and trying every parameter possible, it shows NO DUPLICATES. I tried ccCleaner and that finally found them all but it's only finding them by folder. Only ONE folder has .mobi file in it while the DUPLICATE folder has .epubs so I can't just check off random duplicates in cccleaner because I might be deleting the only folder containing the .mobi which is my preferred format.

Thanks for any tips or alternate pieces of software ... I could take a week off work and do alot of "point, click and delete" but I'd rather not if I don't have to! Thanks all!!

A slightly different solution is after you clean up your database using DaltonST's directions, don't do step 6 just yet. Instead...

1. Sort by title
2. Pick one of the different formats with the correct metadata
3. On the other formats, right click and select Open containing folder.
4. On the picked entry, select and then press E. This will open the metadata editor.
5. Drag/drop the other format into the Medata editor dialog in the upper right
6. Once you see the other format, close Windows Explorer and delete that format's entry
7. Do this for every format of the same eBook
8. Now run a find duplicate

[hotwired]

Thankfully I have my books backed up. But I just ran the "check library" feature. "Mark all for fix" wouldn't do anything so I marked all for deletion and it did what was promised. Wiped out my entire library. For whatever reason this plugin doesn't even come close to doing ANYTHING that I've seen it described as doing. Very strange. I know it's partly me doing the old fashioned thing (physically placing books in the folder that is my library). This is how I learned so it's tough for me to move over to the "database" way of thinking. I'll just restore from backup, then literally, click on every folder (4,000 +) and delete the extras unless I find something else that works. Thank you though for your suggestions. If anything I've said here rings a bell feel free to point me in the right direction though!

[DoctorOhh]

Moderator Notice
Moved posts to their own thread. Check library feature is not part of the Find Duplicates plugin. It appears your duplicates were not part of your calibre library just files in that folder. The Find duplicates plugin does not check the folder level for duplicate books.

[DoctorOhh]

Originally Posted by hotwired

But I just ran the "check library" feature. "Mark all for fix" wouldn't do anything so I marked all for deletion and it did what was promised. Wiped out my entire library.

You told the Ceck library feature of calibre to delete all items in your library folder that were not listed in your library's database. It did as expected. The only books / files in that folder should have been already added to your library database. This feature allows you to ensure your library matches the files in the folder.

Originally Posted by hotwired

For whatever reason this plugin doesn't even come close to doing ANYTHING that I've seen it described as doing.

The Find duplicates plugin finds duplicates in your calibre library database not in the folder structure. Check library feature is a feature of calibre not part of a plugin.

Originally Posted by hotwired

Very strange. I know it's partly me doing the old fashioned thing (physically placing books in the folder that is my library).

Partly you is an understatement. Storing files in your calibre library folder can lead to exactly what you discovered.

Originally Posted by hotwired

This is how I learned so it's tough for me to move over to the "database" way of thinking. I'll just restore from backup, then literally, click on every folder (4,000 +) and delete the extras unless I find something else that works.

Once you have your calibre library is restored you can create a new (empty) library then use the Copy to library feature to move all of your books to a new library. Once all your books are moved whatever is in the previous folder structure are files that were not part of your calibre library database.

[DaltonST]

Originally Posted by hotwired

Thankfully I have my books backed up. But I just ran the "check library" feature. "Mark all for fix" wouldn't do anything so I marked all for deletion and it did what was promised. Wiped out my entire library. For whatever reason this plugin doesn't even come close to doing ANYTHING that I've seen it described as doing. Very strange. I know it's partly me doing the old fashioned thing (physically placing books in the folder that is my library). This is how I learned so it's tough for me to move over to the "database" way of thinking. I'll just restore from backup, then literally, click on every folder (4,000 +) and delete the extras unless I find something else that works. Thank you though for your suggestions. If anything I've said here rings a bell feel free to point me in the right direction though!

Hotwired:

You need to start over from the beginning with Calibre.

[1] Create a fresh, new, empty Calibre Library, and switch to it.

[2] Configure Calibre. See the attached image.

[3] Start using the "Add Books" button that you placed on your ToolBar in Step 2, above. Do not ever use Windows Explorer to add, change or delete books.

[4] When you Add Books, and Calibre tells you that it found duplicates, choose wisely as to what Calibre should do or not do.

I suggest that you first add a few small samples of your 4,000+ books so you can test-and-tune your Preferences to suit your needs.

Adding 4,000+ books might take a few hours depending on your computer system, so add them in reasonably sized chunks.

It would be a good idea to read the Calibre User Manual found at https://manual.calibre-ebook.com/



DaltonST

[BetterRed]

Originally Posted by DaltonST

It would be a good idea to read the Calibre User Manual found at https://manual.calibre-ebook.com/

to starting over etc

Also a good idea to watch the Video Tutorials

The Grand Tour and the Editor videos are the only instructional videos I ever recommend period, generally I loathe such things. Watch a segment, pause, try it, repeat segment until you get it right, next segment...

For me at least, Kovid gets the pitch, structure, content, and pace just right.

BR

[theducks]

FWIW I would do it slightly different

1) I would make sure duplicate detection is turned on in
Preferences: Adding Books: When using Copy to Library
Preferences: Adding Books: Automerge:
2) Create a new (Empty Library).: (your choice) Overwrite or Ignore
<suggest a backup of your old Library, just in case>
3)Select some books in old library: (right-click)Copy to Library Delete, so the list gets smaller): REPEAT
True Dups, get merged. Multiple formats get merged
All that is missing is what source was the best quality

[DaltonST]

Ducks, his current library metadata.db is empty because he never used Add Books. He only used Windows Explorer. Hence, my instructions.

DaltonST

[theducks]

Originally Posted by DaltonST

Ducks, his current library metadata.db is empty because he never used Add Books. He only used Windows Explorer. Hence, my instructions.

DaltonST

[hotwired]

Thanks so much everyone. You've really got me on the right track. The whole "starting over" advice is where I'll start. I wasn't getting that the duplicate finder was only finding duplicates in the DB (Calibre) not physical duplicates. This all makes perfect sense now.

[Stevenro]

Originally Posted by DaltonST

Hotwired:

The Find Duplicates plug-in uses metadata.db. Your library's metadata.db apparently no longer reflects reality due to (forbidden) manual changes to the the library's OS directory structure and contents.

Try this:

[1] Stop manually changing the library's directory structure and contents.

[2] Click Library > Library Maintenance > Check Library and then let Calibre thoroughly check your library, show you what is wrong, and then fix it. Repeat this step as many times as needed until your library is "clean".

[3] Using Bulk Edit Metadata for 100% of your books, "Automatically Set Author Sort", "Change Title to Title Case", and then finally "Update Title Sort".

[4] Using the Quality Check plug-in, standardize the Author Names as FN LN or LN,FN (your choice), but make all of the books comply with your choice.

[5] Using the Quality Check plug-in, standardize the Author Initials.

[6] Run Find Duplicates again.


DaltonST

Duplicate Files Deleter is an effective tool to delete the selected duplicate or original files.

[Stevenro]

I suggest to use Duplicate Files Deleter its a helpful program.

[theducks]

Originally Posted by Stevenro

I suggest to use Duplicate Files Deleter its a helpful program.

Using ANY OS level app on the Library folder is a recipe for disaster. Calibre tracks what and where in the DB. Using the OS to make ANY changes leaves the DB out of the picture.
Please read the FAQ (Help has the link)