Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Reply
 
Thread Tools Search this Thread
Old 07-04-2014, 05:38 AM   #1
faniediv
Member
faniediv began at the beginning.
 
Posts: 11
Karma: 10
Join Date: Sep 2009
Location: Pretoria South Africa
Device: kindle dx
Deleting duplicate folders

My Calibre library is very big. The problem is that I have thousands of duplicate files in the library. It does not show as duplicate books, they are just there in the directory. In other words calibre only recognise one book although there are more than one file. How do I get rid of them? In one post it was suggested that the calibre library is exported, and calibre will then only export the files it uses.
I cannot find this function anywhere in the newer versions of calibre. How do I do this?
I have used “check library” under “Library Maintenance” without success.
None of the options under Quality Check seems to be applicable.
Advise will be appreciated
faniediv is offline   Reply With Quote
Old 07-04-2014, 06:02 AM   #2
eschwartz
Irrational Optimist
eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.
 
eschwartz's Avatar
 
Posts: 8,354
Karma: 14774587
Join Date: Nov 2012
Location: The Beaten Path, USA, Roundworld, This Side of Infinity
Device: Kindle Touch fw5.3.7 (Wifi only)
Create/Move Library.

First option in the Library dropdown.
eschwartz is offline   Reply With Quote
Old 07-05-2014, 03:13 AM   #3
faniediv
Member
faniediv began at the beginning.
 
Posts: 11
Karma: 10
Join Date: Sep 2009
Location: Pretoria South Africa
Device: kindle dx
Thanks. Just tried it. Moved library to new location. Took many hours. All duplicates were moved as well! There must be another way?
faniediv is offline   Reply With Quote
Old 07-05-2014, 03:42 AM   #4
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,106
Karma: 780247
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
Quote:
Originally Posted by faniediv View Post
Thanks. Just tried it. Moved library to new location. Took many hours. All duplicates were moved as well! There must be another way?
That suggests that the files ARE referenced in the Calibre metadata database and thus visible via the Calibre GUI. Your original post implied that there were folders referencing files not visible via the GUI - is this not the case?

Removing duplicate books that are visible in Calibre, or combining entries for the same book to a single entry is a completely different issue. Another interpretation of your original post is that you want do something about additional formats stored for a book that you no longer want.

A few screen shots illustrating the problem might help clarify things.
itimpi is offline   Reply With Quote
Old 07-05-2014, 04:16 AM   #5
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 3,987
Karma: 2925589
Join Date: Mar 2012
Location: NSW Australia
Device: none
I've never deduped a library via the Library->Create/Move.

But I have done it via Copy to Library->NEW EMPTY (delete after copy), if you have the settings in Add books set up as per attachment then after the copy (all books) the NEW EMPTY library will be duplicate free, any duplicates and spurious folders remain in the old library

Not sure if Move Library uses the Add Books settings in same way

BR
Attached Thumbnails
Click image for larger version

Name:	Capture.JPG
Views:	40
Size:	86.0 KB
ID:	124925  

Last edited by BetterRed; 07-05-2014 at 04:23 AM.
BetterRed is online now   Reply With Quote
Old 07-05-2014, 05:03 AM   #6
faniediv
Member
faniediv began at the beginning.
 
Posts: 11
Karma: 10
Join Date: Sep 2009
Location: Pretoria South Africa
Device: kindle dx
Quote:
Originally Posted by itimpi View Post
That suggests that the files ARE referenced in the Calibre metadata database and thus visible via the Calibre GUI. Your original post implied that there were folders referencing files not visible via the GUI - is this not the case?

Removing duplicate books that are visible in Calibre, or combining entries for the same book to a single entry is a completely different issue. Another interpretation of your original post is that you want do something about additional formats stored for a book that you no longer want.

A few screen shots illustrating the problem might help clarify things.
No, Calibre do not see them as duplicates. I can check under writer, only one book. under title, only one book. Run duplicate finder: no duplicates.
But there are up to 3 folders for that book. Calibre only using one.
Each folder contains all the formats complete
Will do screenshot if necessary
faniediv is offline   Reply With Quote
Old 07-05-2014, 05:06 AM   #7
faniediv
Member
faniediv began at the beginning.
 
Posts: 11
Karma: 10
Join Date: Sep 2009
Location: Pretoria South Africa
Device: kindle dx
Quote:
Originally Posted by BetterRed View Post
I've never deduped a library via the Library->Create/Move.

But I have done it via Copy to Library->NEW EMPTY (delete after copy), if you have the settings in Add books set up as per attachment then after the copy (all books) the NEW EMPTY library will be duplicate free, any duplicates and spurious folders remain in the old library

Not sure if Move Library uses the Add Books settings in same way

BR
Thanks will try this now. Takes many hours due to size
faniediv is offline   Reply With Quote
Old 07-05-2014, 05:08 AM   #8
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,106
Karma: 780247
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
Quote:
Originally Posted by faniediv View Post
No, Calibre do not see them as duplicates. I can check under writer, only one book. under title, only one book. Run duplicate finder: no duplicates.
But there are up to 3 folders for that book. Calibre only using one.
Each folder contains all the formats complete
Will do screenshot if necessary
When you tell Calibre to move the library it should only move folders/files referenced in the Calibre metadata database (and thus displayed via the GUI). Not sure therefore how there can be multiple folders per book entry in Calibre after the move.
itimpi is offline   Reply With Quote
Old 07-05-2014, 07:02 AM   #9
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,124
Karma: 5735944
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Ubuntu Linux, Cybook Opus, Motorola Xoom with Mantano Premium
Quote:
Originally Posted by faniediv View Post
Thanks. Just tried it. Moved library to new location. Took many hours. All duplicates were moved as well! There must be another way?
You made some mistake. Try again. Make sure there are no books at the destination before you start. Calibre will only move/copy books it knows about, and that you can see in the gui. If there are no duplicates in the gui, there will be no duplicates in the newly created library folders. If there are duplicates in the gui, there will be in the new library folders as well.
Adoby is offline   Reply With Quote
Old 07-05-2014, 07:51 AM   #10
faniediv
Member
faniediv began at the beginning.
 
Posts: 11
Karma: 10
Join Date: Sep 2009
Location: Pretoria South Africa
Device: kindle dx
Quote:
Originally Posted by Adoby View Post
You made some mistake. Try again. Make sure there are no books at the destination before you start. Calibre will only move/copy books it knows about, and that you can see in the gui. If there are no duplicates in the gui, there will be no duplicates in the newly created library folders. If there are duplicates in the gui, there will be in the new library folders as well.
uH, mistake how?? There are no options to make mistakes, just move library. I do not find any other selectable options. Of course the new destination should be empty.
If what you say is true, calibre is using my duplicate folders somehow, but I can't figure out how. Identical authors and titles but only one book in calibre, however you search.

In any case, I am using the option now as suggested by BetterRed. If it doesn't work, thats it. The whole process takes about 6 hours, after that I will just live with it as is.
faniediv is offline   Reply With Quote
Old 07-05-2014, 10:34 AM   #11
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 5,907
Karma: 1217216
Join Date: Jan 2010
Location: France
Device: Many android devices
If check library does not report extra folders then one of two things is true:
1) they are in the calibre database somewhere.
2) you are mixing case sensitive and case-insensitive file systems.

For number 1, you can look for the paths by creating a composite custom column (column built from other columns) that contains the full paths of every format file for the books calibre knows about. The template to use is {:'formats_paths()'}. After you have created the column, search for one of the books you think is not in the database.

For number 2, I am not sure what to do other than not to do that. For example, if under Linix you create a file "foo" and "Foo" then there are two files with different names. If you later look at the containing folder with windows then strange things might happen.
chaley is offline   Reply With Quote
Old 07-05-2014, 11:02 AM   #12
faniediv
Member
faniediv began at the beginning.
 
Posts: 11
Karma: 10
Join Date: Sep 2009
Location: Pretoria South Africa
Device: kindle dx
Thanks, if the current process does not work, I will try your advice. I just realised that I have used synctoy for years to synchronise between main library and backups. This can be where all the duplicates come from?
The folders mostly are identical regarding case. Some folders do differ:fire & custard; fire and custard, but they are in the minority.
I have no idea how to insert a screen copy. Hope this works

faniediv is offline   Reply With Quote
Old 07-05-2014, 11:04 AM   #13
faniediv
Member
faniediv began at the beginning.
 
Posts: 11
Karma: 10
Join Date: Sep 2009
Location: Pretoria South Africa
Device: kindle dx
Try again:
faniediv is offline   Reply With Quote
Old 07-05-2014, 11:19 AM   #14
chaley
"chaley", not "charley"
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 5,907
Karma: 1217216
Join Date: Jan 2010
Location: France
Device: Many android devices
That screen shot makes debugging a lot easier.

First, I assume that the screen shot is of a calibre library and not a backup folder. If so, then the number in parenthesis is the calibre database ID for that book, at least when the book was created. If check library does not report that one of (for example) Grailblazers with ID 213 or 1331 is extra, then there is every reason to believe that records exist in calibre's db with those ids.

Fortunately it is easy to search in calibre for a book with a given id. Using Grailblazers again as an example, the two search expressions are:
Code:
id:=213
id:=1331
or if you want them both at once:
Code:
id:=213 or id:=1331
The result of the search should be one or two books, at which point you can look at the metadata for them.
chaley is offline   Reply With Quote
Old 07-05-2014, 11:43 AM   #15
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,124
Karma: 5735944
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Ubuntu Linux, Cybook Opus, Motorola Xoom with Mantano Premium
Or just sort on book title (click the column header) and see if there are two entries for Grailblazers in the gui.

If there is two versions, then you have at some point added a duplicate of an already existing book to your library, and calibre helpfully keeps track of both versions for you.

You will have to handle these duplicates manually, and decide which version of the book to keep or possibly merge formats, and which to delete or archive, possibly by moving the unwanted version to a special library for duplicates. Could be nice to keep in case you discover that you decided to keep the wrong version...

If there is only one Grailblazers book in the gui, and you don't have any books hidden or filtered from the gui view, then a move of the library should get rid of the spurious folders. You may want to examine the books left behind in the old library, to see if you want to keep those versions.
Adoby is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Library Folders - Duplicate Authors (, vs _) Since February paw3pals Calibre 15 04-01-2012 05:39 PM
Seeing Duplicate Folders for the Same Author Nyssa Calibre 2 01-22-2012 01:26 PM
Finding and Deleting Duplicate Files of different formats dpayment General Discussions 19 10-19-2011 04:02 PM
Calibre creates duplicate folders BeccaPrice Library Management 5 07-04-2011 11:55 AM
Deleting duplicate collections Gazman Introduce Yourself 3 01-25-2009 11:19 AM


All times are GMT -4. The time now is 05:27 AM.


MobileRead.com is a privately owned, operated and funded community.