Deleting duplicate folders

faniediv · 07-04-2014, 04:38 AM

My Calibre library is very big. The problem is that I have thousands of duplicate files in the library. It does not show as duplicate books, they are just there in the directory. In other words calibre only recognise one book although there are more than one file. How do I get rid of them? In one post it was suggested that the calibre library is exported, and calibre will then only export the files it uses.
I cannot find this function anywhere in the newer versions of calibre. How do I do this?
I have used “check library” under “Library Maintenance” without success.
None of the options under Quality Check seems to be applicable.
Advise will be appreciated

eschwartz · 07-04-2014, 05:02 AM

Create/Move Library.

First option in the Library dropdown.

faniediv · 07-05-2014, 02:13 AM

Thanks. Just tried it. Moved library to new location. Took many hours. All duplicates were moved as well! There must be another way?

itimpi · 07-05-2014, 02:42 AM

Quote:

Originally Posted by faniediv

Thanks. Just tried it. Moved library to new location. Took many hours. All duplicates were moved as well! There must be another way?

That suggests that the files ARE referenced in the Calibre metadata database and thus visible via the Calibre GUI. Your original post implied that there were folders referencing files not visible via the GUI - is this not the case?

Removing duplicate books that are visible in Calibre, or combining entries for the same book to a single entry is a completely different issue. Another interpretation of your original post is that you want do something about additional formats stored for a book that you no longer want.

A few screen shots illustrating the problem might help clarify things.

BetterRed · 07-05-2014, 03:16 AM

I've never deduped a library via the Library->Create/Move.

But I have done it via Copy to Library->NEW EMPTY (delete after copy), if you have the settings in Add books set up as per attachment then after the copy (all books) the NEW EMPTY library will be duplicate free, any duplicates and spurious folders remain in the old library

Not sure if Move Library uses the Add Books settings in same way

BR

faniediv · 07-05-2014, 04:03 AM

Quote:

Originally Posted by itimpi

That suggests that the files ARE referenced in the Calibre metadata database and thus visible via the Calibre GUI. Your original post implied that there were folders referencing files not visible via the GUI - is this not the case?

Removing duplicate books that are visible in Calibre, or combining entries for the same book to a single entry is a completely different issue. Another interpretation of your original post is that you want do something about additional formats stored for a book that you no longer want.

A few screen shots illustrating the problem might help clarify things.

No, Calibre do not see them as duplicates. I can check under writer, only one book. under title, only one book. Run duplicate finder: no duplicates.
But there are up to 3 folders for that book. Calibre only using one.
Each folder contains all the formats complete
Will do screenshot if necessary

faniediv · 07-05-2014, 04:06 AM

Quote:

Originally Posted by BetterRed

I've never deduped a library via the Library->Create/Move.

But I have done it via Copy to Library->NEW EMPTY (delete after copy), if you have the settings in Add books set up as per attachment then after the copy (all books) the NEW EMPTY library will be duplicate free, any duplicates and spurious folders remain in the old library

Not sure if Move Library uses the Add Books settings in same way

BR

Thanks will try this now. Takes many hours due to size

itimpi · 07-05-2014, 04:08 AM

Quote:

Originally Posted by faniediv

No, Calibre do not see them as duplicates. I can check under writer, only one book. under title, only one book. Run duplicate finder: no duplicates.
But there are up to 3 folders for that book. Calibre only using one.
Each folder contains all the formats complete
Will do screenshot if necessary

When you tell Calibre to move the library it should only move folders/files referenced in the Calibre metadata database (and thus displayed via the GUI). Not sure therefore how there can be multiple folders per book entry in Calibre after the move.

Adoby · 07-05-2014, 06:02 AM

Quote:

Originally Posted by faniediv

Thanks. Just tried it. Moved library to new location. Took many hours. All duplicates were moved as well! There must be another way?

You made some mistake. Try again. Make sure there are no books at the destination before you start. Calibre will only move/copy books it knows about, and that you can see in the gui. If there are no duplicates in the gui, there will be no duplicates in the newly created library folders. If there are duplicates in the gui, there will be in the new library folders as well.

faniediv · 07-05-2014, 06:51 AM

Quote:

Originally Posted by Adoby

You made some mistake. Try again. Make sure there are no books at the destination before you start. Calibre will only move/copy books it knows about, and that you can see in the gui. If there are no duplicates in the gui, there will be no duplicates in the newly created library folders. If there are duplicates in the gui, there will be in the new library folders as well.

uH, mistake how?? There are no options to make mistakes, just move library. I do not find any other selectable options. Of course the new destination should be empty.
If what you say is true, calibre is using my duplicate folders somehow, but I can't figure out how. Identical authors and titles but only one book in calibre, however you search.

In any case, I am using the option now as suggested by BetterRed. If it doesn't work, thats it. The whole process takes about 6 hours, after that I will just live with it as is.

chaley · 07-05-2014, 09:34 AM

If check library does not report extra folders then one of two things is true:
1) they are in the calibre database somewhere.
2) you are mixing case sensitive and case-insensitive file systems.

For number 1, you can look for the paths by creating a composite custom column (column built from other columns) that contains the full paths of every format file for the books calibre knows about. The template to use is {:'formats_paths()'}. After you have created the column, search for one of the books you think is not in the database.

For number 2, I am not sure what to do other than not to do that. For example, if under Linix you create a file "foo" and "Foo" then there are two files with different names. If you later look at the containing folder with windows then strange things might happen.

faniediv · 07-05-2014, 10:02 AM

Thanks, if the current process does not work, I will try your advice. I just realised that I have used synctoy for years to synchronise between main library and backups. This can be where all the duplicates come from?
The folders mostly are identical regarding case. Some folders do differ:fire & custard; fire and custard, but they are in the minority.
I have no idea how to insert a screen copy. Hope this works

faniediv · 07-05-2014, 10:04 AM

Try again:

chaley · 07-05-2014, 10:19 AM

That screen shot makes debugging a lot easier.

First, I assume that the screen shot is of a calibre library and not a backup folder. If so, then the number in parenthesis is the calibre database ID for that book, at least when the book was created. If check library does not report that one of (for example) Grailblazers with ID 213 or 1331 is extra, then there is every reason to believe that records exist in calibre's db with those ids.

Fortunately it is easy to search in calibre for a book with a given id. Using Grailblazers again as an example, the two search expressions are:

Code:

id:=213
id:=1331

or if you want them both at once:

Code:

id:=213 or id:=1331

The result of the search should be one or two books, at which point you can look at the metadata for them.

Adoby · 07-05-2014, 10:43 AM

Or just sort on book title (click the column header) and see if there are two entries for Grailblazers in the gui.

If there is two versions, then you have at some point added a duplicate of an already existing book to your library, and calibre helpfully keeps track of both versions for you.

You will have to handle these duplicates manually, and decide which version of the book to keep or possibly merge formats, and which to delete or archive, possibly by moving the unwanted version to a special library for duplicates. Could be nice to keep in case you discover that you decided to keep the wrong version...

If there is only one Grailblazers book in the gui, and you don't have any books hidden or filtered from the gui view, then a move of the library should get rid of the spurious folders. You may want to examine the books left behind in the old library, to see if you want to keep those versions.

07-04-2014, 04:38 AM	#1
faniediv Member Posts: 15 Karma: 10 Join Date: Sep 2009 Location: Pretoria South Africa Device: kindle dx	Deleting duplicate folders My Calibre library is very big. The problem is that I have thousands of duplicate files in the library. It does not show as duplicate books, they are just there in the directory. In other words calibre only recognise one book although there are more than one file. How do I get rid of them? In one post it was suggested that the calibre library is exported, and calibre will then only export the files it uses. I cannot find this function anywhere in the newer versions of calibre. How do I do this? I have used “check library” under “Library Maintenance” without success. None of the options under Quality Check seems to be applicable. Advise will be appreciated

07-05-2014, 03:16 AM	#5
BetterRed null operator (he/him) Posts: 20,553 Karma: 26954694 Join Date: Mar 2012 Location: Sydney Australia Device: none	I've never deduped a library via the Library->Create/Move. But I have done it via Copy to Library->NEW EMPTY (delete after copy), if you have the settings in Add books set up as per attachment then after the copy (all books) the NEW EMPTY library will be duplicate free, any duplicates and spurious folders remain in the old library Not sure if Move Library uses the Add Books settings in same way BR Attached Thumbnails Last edited by BetterRed; 07-05-2014 at 03:23 AM.

07-05-2014, 10:19 AM	#14
chaley Grand Sorcerer Posts: 11,734 Karma: 6690881 Join Date: Jan 2010 Location: Notts, England Device: Kobo Libra 2	That screen shot makes debugging a lot easier. First, I assume that the screen shot is of a calibre library and not a backup folder. If so, then the number in parenthesis is the calibre database ID for that book, at least when the book was created. If check library does not report that one of (for example) Grailblazers with ID 213 or 1331 is extra, then there is every reason to believe that records exist in calibre's db with those ids. Fortunately it is easy to search in calibre for a book with a given id. Using Grailblazers again as an example, the two search expressions are: Code: id:=213 id:=1331 or if you want them both at once: Code: id:=213 or id:=1331 The result of the search should be one or two books, at which point you can look at the metadata for them.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Library Folders - Duplicate Authors (, vs _) Since February	paw3pals	Calibre	15	04-01-2012 04:39 PM
Seeing Duplicate Folders for the Same Author	Nyssa	Calibre	2	01-22-2012 12:26 PM
Finding and Deleting Duplicate Files of different formats	dpayment	General Discussions	19	10-19-2011 03:02 PM
Calibre creates duplicate folders	BeccaPrice	Library Management	5	07-04-2011 10:55 AM
Deleting duplicate collections	Gazman	Introduce Yourself	3	01-25-2009 10:19 AM

07-04-2014, 05:02 AM	#2
eschwartz Ex-Helpdesk Junkie Posts: 19,422 Karma: 85397180 Join Date: Nov 2012 Location: The Beaten Path, USA, Roundworld, This Side of Infinity Device: Kindle Touch fw5.3.7 (Wifi only)	Create/Move Library. First option in the Library dropdown.

07-05-2014, 02:13 AM	#3
faniediv Member Posts: 15 Karma: 10 Join Date: Sep 2009 Location: Pretoria South Africa Device: kindle dx	Thanks. Just tried it. Moved library to new location. Took many hours. All duplicates were moved as well! There must be another way?

07-05-2014, 09:34 AM	#11
chaley Grand Sorcerer Posts: 11,734 Karma: 6690881 Join Date: Jan 2010 Location: Notts, England Device: Kobo Libra 2	If check library does not report extra folders then one of two things is true: 1) they are in the calibre database somewhere. 2) you are mixing case sensitive and case-insensitive file systems. For number 1, you can look for the paths by creating a composite custom column (column built from other columns) that contains the full paths of every format file for the books calibre knows about. The template to use is {:'formats_paths()'}. After you have created the column, search for one of the books you think is not in the database. For number 2, I am not sure what to do other than not to do that. For example, if under Linix you create a file "foo" and "Foo" then there are two files with different names. If you later look at the containing folder with windows then strange things might happen.

07-05-2014, 10:02 AM	#12
faniediv Member Posts: 15 Karma: 10 Join Date: Sep 2009 Location: Pretoria South Africa Device: kindle dx	Thanks, if the current process does not work, I will try your advice. I just realised that I have used synctoy for years to synchronise between main library and backups. This can be where all the duplicates come from? The folders mostly are identical regarding case. Some folders do differ:fire & custard; fire and custard, but they are in the minority. I have no idea how to insert a screen copy. Hope this works

07-05-2014, 10:04 AM	#13
faniediv Member Posts: 15 Karma: 10 Join Date: Sep 2009 Location: Pretoria South Africa Device: kindle dx	Try again:

07-05-2014, 10:43 AM	#15
Adoby Handy Elephant Posts: 1,736 Karma: 26785668 Join Date: Dec 2009 Location: Southern Sweden, far out in the quiet woods Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus	Or just sort on book title (click the column header) and see if there are two entries for Grailblazers in the gui. If there is two versions, then you have at some point added a duplicate of an already existing book to your library, and calibre helpfully keeps track of both versions for you. You will have to handle these duplicates manually, and decide which version of the book to keep or possibly merge formats, and which to delete or archive, possibly by moving the unwanted version to a special library for duplicates. Could be nice to keep in case you discover that you decided to keep the wrong version... If there is only one Grailblazers book in the gui, and you don't have any books hidden or filtered from the gui view, then a move of the library should get rid of the spurious folders. You may want to examine the books left behind in the old library, to see if you want to keep those versions.