Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Reply
 
Thread Tools Search this Thread
Old 07-26-2013, 01:56 PM   #1
Rbneader
Addict
Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.
 
Posts: 315
Karma: 1195795
Join Date: Mar 2012
Device: None
Calibre doesn't see books, randomly deletes them

Hello!

I'm having a very strange problem and would love some help.

Over the past few weeks, I've noticed that Calibre has
1) Missed some books that are still in the library folders. The Calibre program doesn't see them, but they do exist in the Calibre Library folder.

and separately (maybe it's related, IDK)

2) has started randomly deleting books.

I'm using Calibre Portable if that helps.

I can't figure out how to replicate either of these issues, and I'm leery of trying because this has already deleted 5 of the No. 1 Ladies Detective Agency books while I wasn't looking.

Anyone have any ideas on how to prevent this from happening in the future? At this point I think I'm going to have to redownload all the missing books, which is a pain but doable. I definitely don't want to worry about this happening again in the future though. Is Calibre subject to come kind of corruption, or ID confusion problem?
Rbneader is offline   Reply With Quote
Old 07-26-2013, 01:58 PM   #2
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,094
Karma: 780247
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
Never heard of such behaviour!

You are not by any chance using a cloud service to store your Calibre library? Some of them (in particular GoogleDrive) can interfere with Calibre.
itimpi is offline   Reply With Quote
Old 07-26-2013, 02:11 PM   #3
Rbneader
Addict
Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.
 
Posts: 315
Karma: 1195795
Join Date: Mar 2012
Device: None
No, this is all on my local machine.

I'm completely baffled by this too. I'm in the middle of exporting my books up and setting up an alternate Calibre installation to try and stop whatever is happening. But how this happened is still a headscratcher. I mean, I thought Calibre uses the file structure as a database, so how did it not see the folders that were there? And where did my books go? Was there some kind of ID confusion and Calibre auto-merged things?

Last edited by Rbneader; 07-26-2013 at 02:13 PM.
Rbneader is offline   Reply With Quote
Old 07-26-2013, 02:48 PM   #4
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,094
Karma: 780247
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
Calibre will only see files/folders that are referenced in its metadata.db database. You can run the Calibre Library Maintenance function to identify mismatches between the database and the underlying files in the Calibre library folder.
itimpi is offline   Reply With Quote
Old 07-26-2013, 03:06 PM   #5
Rbneader
Addict
Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.
 
Posts: 315
Karma: 1195795
Join Date: Mar 2012
Device: None
Quote:
Originally Posted by itimpi View Post
Calibre will only see files/folders that are referenced in its metadata.db database. You can run the Calibre Library Maintenance function to identify mismatches between the database and the underlying files in the Calibre library folder.

Ahah! This gives a lengthy list of 'Extra Titles', which sounds like problem #1. There doesn't seem to be any way to fix them except for deletion, though. Or re-importing.
Rbneader is offline   Reply With Quote
Old 07-26-2013, 03:27 PM   #6
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 3,820
Karma: 2698649
Join Date: Mar 2012
Location: NSW Australia
Device: none
Quote:
Originally Posted by Rbneader View Post
I thought Calibre uses the file structure as a database, so how did it not see the folders that were there?
Aaah, Calibre does NOT use the file structure as its database. I'm wondering if you've been changing the files and folders via your file manager.

Calibre uses the file system as a place to put books, much like a bricks & mortar library uses its shelves. When your librarian wants to see if the library has a book, whether its out on loan, where it is in the library etc, then reference is made to the library index/catalogue

The <library>/<author>/<title> folders you see are the equivalent of the library shelves, the metadata.db file in the <library> folder is the equivalent of the catalogue/index - they are closely coupled via the Calibre GUI code.

There is one 'golden rule' with Calibre, which is, don't make DIRECT changes to the names of folders or files within a library, or remove or add any folders or files to a library via your file manager or via the command line etc.

The Calibre GUI provides rich set of features (some would say too rich) to change library & author names, book titles etc, add additional formats directly or via conversion etc etc. Ordinarily there is no need to visit the library folders but if you do then just follow the 'golden rule' and all will be fine.

If you digress from the 'golden rule' then things will go wrong and the library wont function properly. Its like if people went into a bricks & mortar library, and took books without checking them out, moved books around on the shelves, swapped covers, put extraneous books on the shelves.

Did you watch the Calibre video - http://www.mobileread.com/forums/sho...d.php?t=118596 - if not then I recommend it.

BR
BetterRed is offline   Reply With Quote
Old 07-26-2013, 05:06 PM   #7
Rbneader
Addict
Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.
 
Posts: 315
Karma: 1195795
Join Date: Mar 2012
Device: None
Quote:
Originally Posted by BetterRed View Post
Aaah, Calibre does NOT use the file structure as its database. I'm wondering if you've been changing the files and folders via your file manager.
Nope.
Rbneader is offline   Reply With Quote
Old 07-26-2013, 05:40 PM   #8
theducks
Grand Sorcerer
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 15,086
Karma: 5939999
Join Date: Aug 2009
Location: (The original) Silicon Valley, USA
Device: Galaxy Tab 2, Astak Pocket Pro, K4NT
Internal HD or
Removable Media? Disable caching (a way to improve performance, but risk data). The DB data and the folder names need to stay in lock-step
theducks is offline   Reply With Quote
Old 07-27-2013, 04:36 AM   #9
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 3,820
Karma: 2698649
Join Date: Mar 2012
Location: NSW Australia
Device: none
Quote:
Originally Posted by Rbneader View Post
Is Calibre subject to come kind of corruption, or ID confusion problem?
Not that I know of, and I can't recall anyone reporting random disappearing books

If you open a book format file in an editor program, and that editor program does NOT lock the book format file and then whilst editing said book you change the author or title then that can cause the database and the folder names to get out of lock step

IIRC Sigil does/did not lock the epubs being edited. The Irfanview image editor is another program that does not lock the files it has open.

Word locks the files it is editing, and Calibre will throw an exception if it detects the lock when you try to change author or title. PDF xChnage also locks files it has open which Calibre detects

Whenever I've found errors in the second stage of the Check Library (Extra Titles etc), then they've always attributable to me changing an author or title whilst editing a format or cover file with a non-locking editor or me changing or moving things in the library folders.

I don't get many of these errors since I trained myself to avoid the pitfalls, such as getting the author and book titles 100% correct as soon as the book arrives before doing anything else.

BR
BetterRed is offline   Reply With Quote
Old 02-14-2014, 04:41 PM   #10
Rbneader
Addict
Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.
 
Posts: 315
Karma: 1195795
Join Date: Mar 2012
Device: None
This is happening again. It happened when I chose the 'Copy and delete' option to move books from one library to another. Don't know why it worked for 50 books and not the 51st, but one of the books that I know I had is missing. I didn't edit the book in Sigil or open it in another program.

At this point I think I need to stop relying on Calibre for library management, because it's deleted at least 10 of my books. And those are just the ones I've caught!

ETA: The library check doesn't find any errors. The book file and metadata are both gone. The only reason I found this was because I put the book in question on my ereader 2 days ago, before copying it to the new library, so when I looked for it in my Calibre library and didn't see it I knew there was a problem.

Anyone have any ideas on how to find other books Calibre deleted?

Last edited by Rbneader; 02-14-2014 at 05:01 PM.
Rbneader is offline   Reply With Quote
Old 02-14-2014, 08:03 PM   #11
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 3,820
Karma: 2698649
Join Date: Mar 2012
Location: NSW Australia
Device: none
Quote:
Originally Posted by Rbneader View Post

Anyone have any ideas on how to find other books Calibre deleted?
Try Recycle bin/Rubbish tin/Trash can, or a find file by name - on windows, agent ransack is pretty good.


BR

Last edited by BetterRed; 02-14-2014 at 10:16 PM.
BetterRed is offline   Reply With Quote
Old 02-14-2014, 10:21 PM   #12
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,124
Karma: 5721184
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Ubuntu Linux, Cybook Opus, Motorola Xoom with Mantano Premium
It is overwhelmingly more likely that it is some form of user error that caused calibre to loose track of books, than it is a bug or error in calibre that is the cause.

Somehow files in the calibre library has been updated, renamed, restored or otherwise changed in a way that made the database metadata.db out of sync with the books stored in the library. The paths and the names stored in metadata.db don't match the actual files. This is discovered only when calibre attempts to do something with the books. So there may very well be a very long time span between the error that caused the problem and the discovery of the problem. That is why it is good to run the database maintenance tools now and then, especially after the library has been updated or changed using any other tools than the GUI.

Possible causes are partial restores of backups, overwriting one library with another. Perhaps a restore of an old metadata.db without replacement of the books. Or a partial restore of book files from backups. Copying the library between different operating systems and/or filesystems that have different case sensitivity. Or using some utilities that updated files or paths in the library.

When calibre is backed up and restored it should be handled as a single monolithic entity. Never partially restored or restored over an existing library.

The problems should hopefully be easy to fix by running the calibre maintenance tools, using the GUI to copy the entire library to a new library, and manually add the books left behind in the old library to the new library. But metadata may be lost.

Please note that the problem will most likely come back again and again, if the assumption is that calibre was the cause. The same user error will then most likely be repeated again. So do try to figure out the actual cause.

Last edited by Adoby; 02-14-2014 at 10:28 PM.
Adoby is offline   Reply With Quote
Old 02-14-2014, 10:41 PM   #13
theducks
Grand Sorcerer
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 15,086
Karma: 5939999
Join Date: Aug 2009
Location: (The original) Silicon Valley, USA
Device: Galaxy Tab 2, Astak Pocket Pro, K4NT
Synching the Library folder while Calibre is active.

Multiple users/sessions connected to the Library

Multiple OS used on the same Library (file name handling differences)

Using a Network drive for the Library. (Some folk get away with it. Others don't)
theducks is offline   Reply With Quote
Old 02-17-2014, 10:48 AM   #14
Rbneader
Addict
Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.
 
Posts: 315
Karma: 1195795
Join Date: Mar 2012
Device: None
Quote:
Originally Posted by Adoby View Post
It is overwhelmingly more likely that it is some form of user error that caused calibre to loose track of books, than it is a bug or error in calibre that is the cause.

Somehow files in the calibre library has been updated, renamed, restored or otherwise changed in a way that made the database metadata.db out of sync with the books stored in the library. The paths and the names stored in metadata.db don't match the actual files. This is discovered only when calibre attempts to do something with the books. So there may very well be a very long time span between the error that caused the problem and the discovery of the problem. That is why it is good to run the database maintenance tools now and then, especially after the library has been updated or changed using any other tools than the GUI.

Possible causes are partial restores of backups, overwriting one library with another. Perhaps a restore of an old metadata.db without replacement of the books. Or a partial restore of book files from backups. Copying the library between different operating systems and/or filesystems that have different case sensitivity. Or using some utilities that updated files or paths in the library.

When calibre is backed up and restored it should be handled as a single monolithic entity. Never partially restored or restored over an existing library.

The problems should hopefully be easy to fix by running the calibre maintenance tools, using the GUI to copy the entire library to a new library, and manually add the books left behind in the old library to the new library. But metadata may be lost.

Please note that the problem will most likely come back again and again, if the assumption is that calibre was the cause. The same user error will then most likely be repeated again. So do try to figure out the actual cause.
You didn't read my post.

It's not that Calibre lost the filepaths. It's that Calibre has lost the files themselves. The book file is gone. I know how to fix Calibre losing the filepaths. What I don't know is how to stop Calibre from unpredictably deleting the book files themselves.

Also, I ran all the Library Check tools, which found nothing wrong.

The book was there in Library A, I picked 'Copy to Library B, delete after moving', then looked for it the next day and it wasn't in any of my libraries. So I'm pretty sure it was something Calibre did, probably when it copied the book over.

The most logical idea seems to me to be that somehow the copying went wrong, Calibre didn't realize that, and then deleted the copy from Library A like normal. This would fit the observed behavior so far. I'm not sure how to reproduce it, though. The book in question was one of many books that was copied over at the same time, and the rest seem to have made it through alright.

Until this is figured out I'm setting up a backup folder structure that will be my actual library. It's less convenient, but a library that unpredictably deletes files is a library I can't rely on. I'm also setting up a whole new portable Calibre installation, in case it was just something about my old installation.

Last edited by Rbneader; 02-17-2014 at 10:51 AM.
Rbneader is offline   Reply With Quote
Old 02-17-2014, 10:49 AM   #15
Rbneader
Addict
Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.
 
Posts: 315
Karma: 1195795
Join Date: Mar 2012
Device: None
Quote:
Originally Posted by theducks View Post
Synching the Library folder while Calibre is active.

Multiple users/sessions connected to the Library

Multiple OS used on the same Library (file name handling differences)

Using a Network drive for the Library. (Some folk get away with it. Others don't)
The last 3 don't apply here. Is the first item talking about synching to Dropbox/another online backup site, or about something else?
Rbneader is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[Old Thread] Why doesn’t calibre let me store books in my own directory structure? darknessangel Calibre 68 08-24-2014 01:16 AM
Calibre content-server doesn't seem to see new books offby1 Calibre 7 05-05-2013 11:58 AM
Calibre doesn't recognize .azw3 books Breed92014 Devices 6 04-15-2012 03:50 PM
Calibre 7.28 doesn't List Nook Books Kavon Calibre 0 11-15-2010 12:20 PM
K2 self-deletes 2 books at once BarryTX Amazon Kindle 14 04-09-2010 04:30 PM


All times are GMT -4. The time now is 01:48 AM.


MobileRead.com is a privately owned, operated and funded community.