Calibre doesn't see books, randomly deletes them

[DoctorOhh]

Originally Posted by Rbneader

Is the first item talking about synching to Dropbox/another online backup site, or about something else?

If the receiving library was in Dropbox and you did not pause syncing it is very possible that one or more books may not have made it to their destination. If you are going to do any bulk operations with a library in Dropbox (or other cloud) then pause syncing before doing anything.

Originally Posted by Rbneader

Until this is figured out I'm setting up a backup folder structure that will be my actual library.

Backups are always an excellent idea.

Originally Posted by Rbneader

It's less convenient, but a library that unpredictably deletes files is a library I can't rely on. I'm also setting up a whole new portable Calibre installation, in case it was just something about my old installation.

Calibre does not delete any files at anytime unless directed by the user. You are experiencing OS level issues or cloud sync software issues.

Good luck with your work around.

[theducks]

Originally Posted by Rbneader

The last 3 don't apply here. Is the first item talking about synching to Dropbox/another online backup site, or about something else?

Dropbox or any other connection that can write (or block a calibre write) to the Library folder.

Write behind (delayed) caching is another culprit.

Antivirus scans can block a change. Exclude the Library folder from a 'realtime' scan.

IMHO another problem with using 'the cloud' : Everything in my library has been scanned BEFORE it arrived. With the cloud, it seems possible to inject a contaminated copy if real time scanning has been excluded for structure integrity purposes . (like preventing your problem?)

4+ years running Calibre. Every book loss case has been identified to: ME Something (stupid) I did.

[Rbneader]

Originally Posted by DoctorOhh

If the receiving library was in Dropbox and you did not pause syncing it is very possible that one or more books may not have made it to their destination. If you are going to do any bulk operations with a library in Dropbox (or other cloud) then pause syncing before doing anything.

Good luck with your work around.

That's what I thought. I don't sync my library online anywhere, so that's not it.

[Rbneader]

Originally Posted by theducks

Dropbox or any other connection that can write (or block a calibre write) to the Library folder.

Write behind (delayed) caching is another culprit.

Antivirus scans can block a change. Exclude the Library folder from a 'realtime' scan.

IMHO another problem with using 'the cloud' : Everything in my library has been scanned BEFORE it arrived. With the cloud, it seems possible to inject a contaminated copy if real time scanning has been excluded for structure integrity purposes . (like preventing your problem?)

I do have security software that might have decided to scan at exactly that moment. I'll tell it to not worry about my ebooks.

I don't sync my library anywhere, so it's not an issue with that.

If Calibre can't handle the standard Windows caching methods, than that is a problem with Calibre. I don't think that's the case, though. An OS feature that's been standard for years shouldn't be able to cause book loss except with egregiously inflexible coding. There would be many, many more reports of this issue if that was the case, so I don't think it's related to caching.

My workaround will at least keep books from being lost, so that's something. It's still incredibly frustrating. Hopefully I'll be able to reproduce it.

Could it be related to multiple libraries? I do have several different Calibre libraries and move books between them fairly often using Calibre's standard move-book functionality. If swapping books around isn't something that Calibre can handle well that's at least an explanation that can be worked around.

[theducks]

Calibre does a set of transactions: folder(s) DB, cover, format(s)
Anything that seriously impacts their creation is the fault of that system as a normal Windows has no problem.
Millions of Users vs Your install.

[BetterRed]

Rbneader - If you have a book open in a 3rd party program when you move books between libraries things can go awry. Some 3rd party programs (eg Microsoft's Word and Trackers Xchange) will lock the files they have open and calibre will pop an error when you try to do the move, but many programs don't lock the files they have open and then...

BR

[Rbneader]

Originally Posted by BetterRed

Rbneader - If you have a book open in a 3rd party program when you move books between libraries things can go awry. Some 3rd party programs (eg Microsoft's Word and Trackers Xchange) will lock the files they have open and calibre will pop an error when you try to do the move, but many programs don't lock the files they have open and then...

BR

I know I didn't open the disappearing book in any other program. I might have had another book I was working on open, but that shouldn't cause a locking issue with the disappearing book. Unless maybe their IDs were the same behind the scenes or something like that. It would be strange if that was the case, but this whole thing is strange.

[BetterRed]

Originally Posted by Rbneader

I know I didn't open the disappearing book in any other program. I might have had another book I was working on open, but that shouldn't cause a locking issue with the disappearing book. Unless maybe their IDs were the same behind the scenes or something like that. It would be strange if that was the case, but this whole thing is strange.

Might you have changed an Author Name or Title when a format file was open that will result in a move, if a file was open in another program - eg Sigil IIRC it doesn't lock files - that can result in lost edits, but not lost books.

when you change a book title (which triggers a rename of the book folder) then IIRC calibre creates a folder with the new name, then it creates hardlinks to the files in the new folder, and then it trashes (shift/del) the old folder. But it then again it might actually be NTFS that does all that...

Do you have any gadgets/widgets/belles [sic] or whistles that might interfere with that process - I can't imagine what, but...

BR

[Adoby]

Calibre is very careful when moving files.

Copy first, then delete. If there was any problems with the copy, that would crash calibre or generate an error, before the delete. So I still say that it isn't calibre that caused your problems. If it was thousands would have reported problems a long time ago and it would have been fixed.

It is also unlikely to be a hardware error, since you seem to be able to use your computer and the harddrives.

Unfortunately that only(?) leaves user error. That you yourself somehow inadvertently deleted the books. At times like that it is good to have recent backups.

I don't trust myself or hardware, so I let my computer automatically backup my calibre library (and other stuff I don't want to loose) every hour to an extra harddrive on my computer, and daily to a NAS with redundancy. And to external USB hard drives now and then. And I have most of my library on my reading device. But I do trust calibre to not delete any books, unless I tell it to.

[BetterRed]

@Rbneader - I just noticed that you started this thread back in July last year, I'm not sure if the problems you raised then were resolved. Following my post of 07-27-2013, 08:36 AM the thread was quiet until your post of 02-14-2014, 08:41 PM

I'm intrigued as to what happened in the intervening 6 months - was calibre working OK or did you stop using it or... whatever.

Around October last year there was an issue regarding temporary folders being created and not being removed when deleting (moving ?) books and formats. I can't recall precise details nor whether this created actual problems or if it was just a nuisance issue. And IIRC around August there was an issue about the truncation of file names being altered that caused some consternation, again I can't recall details or the ramifications.

So I'm wondering if your current problems are a continuation of the problems you were having in July, or whether they might be related to those couple of issues that occurred in Aug & Oct. And I'm hoping people with better memories than mine can throw in there 2 cents worth.

BR

[eschwartz]

The temporary folders issue (when deleting a book) only happened on unusual filesystem mounts, like when I reported this using a school computer. It is a fat client running LTSP, so the filesystem is I believe mounted from the server computer. And it only kicked in after the book gets deleted, which if I understand correctly should be after the copy was made. Something to do with the new database backend having a new method of sending files to the trash.

The filename truncation was an issue involving a lack of proper support for long filenames (a Windows problem) and it meant that people expected to find their books under the usual {author}\{title} - {author}.{format} location, but didn't, since the unusually long field was given to the filesystem in shortened form. calibre kept perfect track of it, it just didn't play nice with people "looking under calibre's skirt" due to a Windows limitation.

As far as I am aware, neither should cause any problems with the books disappearing.

[BetterRed]

Originally Posted by eschwartz

The temporary folders issue (when deleting a book) only happened on unusual filesystem mounts, like when I reported this using a school computer.

It is a fat client running LTSP, so the filesystem is I believe mounted from the server computer. And it only kicked in after the book gets deleted, which if I understand correctly should be after the copy was made. Something to do with the new database backend having a new method of sending files to the trash.

I had them on a standard windows/ntfs rig, on a fixed internal drive. I also had a problem with hundreds of them turning up in the recycle bin

Both problems were repeatable but neither caused any loss of data or library corruption - Check Library saw them as extra author folders - but I first noticed them in my backup logs.

Both problems were fixed

Originally Posted by eschwartz

The filename truncation was an issue involving a lack of proper support for long filenames (a Windows problem) and it meant that people expected to find their books under the usual {author}\{title} - {author}.{format} location, but didn't, since the unusually long field was given to the filesystem in shortened form. calibre kept perfect track of it, it just didn't play nice with people "looking under calibre's skirt" due to a Windows limitation.

As far as I am aware, neither should cause any problems with the books disappearing.

I have a vague memory that the file name truncation length changed from from 36 to 34 or something like that - it seemed to disturb a few people - although I can't recall whether they were of a material or aesthetic kind.

I can't recall them leading to disappearing books either - but there's a first time for everything.

I don't have any formal reconciliation audit controls on my libraries so a 'book' or three could easily disappear from the two larger ones without me noticing.

BR

[eschwartz]

Originally Posted by BetterRed

I had them on a standard windows/ntfs rig, on a fixed internal drive. I also had a problem with hundreds of them turning up in the recycle bin

Both problems were repeatable but neither caused any loss of data or library corruption - Check Library saw them as extra author folders - but I first noticed them in my backup logs.

Both problems were fixed

Well, the problem as Kovid explained it was:

For some reason, your OS is raising permission denied (EPERM) errors
when calibre tries to delete a non-empty directory, instead of
ENOTEMPTY, which is causing the problem. Do you have your calibre
library on some kind of special filesystem?

and

In any case, I have changed the code to not rely on the OS sending the
proper error code, it will now manually check if the directory is empty.
That should fix your problem.

Whatever the problem was, that is what caused it for me.

And I got them in the parent directory to my library folder (in "Calibre Libraries") -- also no loss of data, just a pain in the neck to clean up after. Either way, it shouldn't cause errors until after the book was copied, so...

I have a vague memory that the file name truncation length changed from from 36 to 34 or something like that - it seemed to disturb a few people - although I can't recall whether they were of a material or aesthetic kind.

Aesthetic, definitely. The change was deliberate and calibre still does it.