Calibre arbitrarily removing my books

[C-novice]

Originally Posted by BetterRed

- wise non-move!


Agreed 110%.

• Is the library synced to cloud storage - e.g. dropbox, google drive etc?
  
• What OS are you using (Windows, OSX or Linux) and what file system is the library on?
  
• Is the library on the system drive ?

Maybe the folders/files weren't deleted, but moved - have you tried searching for the lost files. On windows I would run Recuva (3rd party util) to look for any clues.

It's hard to believe some other program would be stealing/deleting books and covers randomly - except maybe a dodgy backup file synch process.

BR

Thanks for following up BR .

No. The Library is not synced to any cloud storage at all -or anything else.

I'm using Linux Mint, Cinnamon 17.2, 64bit; library is on file-system Ext4.

Yes, the Library is on the computer's hard drive.

You mean search the computer for the missing file? If so, then, as I said, my first thought upon realizing that a book is no longer showing up in the Library is to go to where I store all my original book files and re-add it to Calibre. So I guess, I'd find the file anyway since it's also somewhere else outside of Calibre's library.

Well, I do back up very often using the system's [Linux Mint] Back up tool. But then why only my book files as I haven't noticed any other files missing? -I probably need to do a thorough check on this now.

[C-novice]

Those kinds say that the DB is NOW out of sync with the file system.

You renamed the Author or title, but there was something interfering with the transaction (copy and delete) . (cloud sync, user has file open, A/V scan...)

I'm not surprised some 'out of syncing' has happened. See, I buy mostly epubs and I don't download directly to Calibre. I download to my Books folder. Then from Calibre, I add them to my Calibre Library.

I edit the meta data to add the blurb because most times they don't get added with the book and also because I added a few custom columns like, Read -Yes/No, cost of book, word count, where purchased, etc. As well, if the title or author's first and last names are not separated by a space, I'll correct that. And, oh yes, I convert them to make sure -I the last while I got to thinking that step is probably unnecessary. That would create double files in the DB, wouldn't it?

Anyway, I then save them back to my Book folder, connect my reader and copy/move the files to my reader. That way all the metadata I added in Calibre show up for each book in my reader.

No cloud syncing, no AV -only GUFW, it won't mess with that stuff, would it now?

I'm not sure what you meant by copy and delete.

Looking closely at those errors may give you additional clues
Non-ASCII characters, Multi-OS use of the same Library (OS file name issues).

Non-ASCIIs -don't know how to check for those.
No. The library is used on only one operating system. I'm not dual booting -never did. Thing is though, this system was a Win 8.1, I replaced it with Linux.

Unless the fact that I copy/pasted my Calibre Library folder from a back up usb I made on Win to Linux and continued using it may be causing an issue now? But that would be the same for all the other files I carried over to Linux and they're okay.

Having the Library on Network drives can cause these issues (Or leaving the sync active when using Calibre)

Not using a network drive(s).

On the other hand, I wonder if the old version of Calibre [1.25] is causing some bother now that I'm thinking of all possibilities. It is still installed and uninstalling it will only uninstall the current version anyway so I left it since I hadn't uninstalled it before installing the newer version when I just started.

[theducks]

Originally Posted by C-novice

I'm not sure what you meant by copy and delete.

When you rename in Calibre, it does NOT do a Move with rename. I Copies the book with its new name, then deletes the old. Slower, but safer.

Unless the fact that I copy/pasted my Calibre Library folder from a back up usb I made on Win to Linux and continued using it may be causing an issue now? But that would be the same for all the other files I carried over to Linux and they're okay.

A Paste (overwrite) would mess things up, like those extra files. A Total Replace is the way to go

On the other hand, I wonder if the old version of Calibre [1.25] is causing some bother now that I'm thinking of all possibilities. It is still installed and uninstalling it will only uninstall the current version anyway so I left it since I hadn't uninstalled it before installing the newer version when I just started.

Calibre has not made a structural change in a long time. So 1.x to 2.x should not be an issue

[C-novice]

Originally Posted by theducks

A Paste (overwrite) would mess things up, like those extra files. A Total Replace is the way to go

Hmm. . ., overwrite? Even though when I moved my Calibre Library folder to Linux /home, nothing, including none of my book files, was there yet?

I do admit, this is beyond me at the moment. Yet, I am very glad for all your help and glad too, that Calibre still works.

Thanks again.

[theducks]

Originally Posted by C-novice

Hmm. . ., overwrite? Even though when I moved my Calibre Library folder to Linux /home, nothing, including none of my book files, was there yet?

I do admit, this is beyond me at the moment. Yet, I am very glad for all your help and glad too, that Calibre still works.

Thanks again.

I was referring to pasting INTO an existing (non-empty) Library

There are steps you need to take (because case sensitive) first thing after changing OS. The simplest, would be
Library Maintenance: Rebuild

Which effectively readjusts the DB file/path to that book so case is no longer an issue

[C-novice]

Originally Posted by theducks

I was referring to pasting INTO an existing (non-empty) Library

There are steps you need to take (because case sensitive) first thing after changing OS. The simplest, would be
Library Maintenance: Rebuild

Which effectively readjusts the DB file/path to that book so case is no longer an issue

I see. Okay. Thanks a bunch for explaining all this to me.

[BetterRed]

Originally Posted by theducks

When you rename in Calibre, it does NOT do a Move with rename. I Copies the book with its new name, then deletes the old. Slower, but safer.

On my Windows 10 system with NTFS HDD a change of title to a book that has a 1,000MB+ format file takes the same time as a book with a 100KB format file, sub 1 second. If I copy that 1,000MB file it will take a 10-20 seconds at best.

If what you say is true, that the files are copied, Kovid must have found a way to overclock my hard drive

I suspect you're confusing Rename (change Author and/or Title) with Copy to Library (Delete after copy) the latter is always a copy because the other library could be on another drive.

BR

[BetterRed]

Originally Posted by C-novice

Yes, the Library is on the computer's hard drive.

Is it same same drives as Linux and software, reason I ask is; if it is then its unlikely that a hardware fault would only effect calibre library data and not more critical things that Linux itself needs.

Originally Posted by C-novice

You mean search the computer for the missing file?

Yes to see if the lost files have been copied to some obscure location. Many (most ?) 'calibre lost my books' issues are caused by the folders/files being moved by someone/something - often as a result of mouse drag and drop slips.

Search for the calibre file name 'author - title', you might find the 'missing' files in a 'how did it get there' location - eg your music library Or, search for the book format you lost (e.g. EPUB, MOBI or AZW3 etc) and sort the results on location

Originally Posted by C-novice

Well, I do back up very often using the system's [Linux Mint] Back up tool. But then why only my book files as I haven't noticed any other files missing? -I probably need to do a thorough check on this now.

Not familiar with Mint Backup - but if you backup the calibre library(ies) along with everything else then "why only my book files" indeed

Suggest you resolve the existing anomalies from Check Library, and then run it regularly, you can do that from the command line see calibredb --check_library command. I run it at end-of-day, and look at the CSV next day, if something does go awry I usually (always ?) know who to blame

BR

[theducks]

Originally Posted by BetterRed

On my Windows 10 system with NTFS HDD a change of title to a book that has a 1,000MB+ format file takes the same time as a book with a 100KB format file, sub 1 second. If I copy that 1,000MB file it will take a 10-20 seconds at best.

If what you say is true, that the files are copied, Kovid must have found a way to overclock my hard drive

I suspect you're confusing Rename (change Author and/or Title) with Copy to Library (Delete after copy) the latter is always a copy because the other library could be on another drive.

BR

I don't think so. IIRC Kovid explained why there could be stragglers left behind, because the book was open (locked) elsewhere

[BetterRed]

Originally Posted by theducks

I don't think so. IIRC Kovid explained why there could be stragglers left behind, because the book was open (locked) elsewhere

I just added a 3.09 Gigabyte ISO to a book, it took several minutes to copy the ISO into the book folder (from elsewhere on the same drive), in fact long enough for me to go the bathroom, and put the kettle on. When it finished I renamed the book and it took less than a second.

Please explain

BR

[C-novice]

Originally Posted by BetterRed

Is it same same drives as Linux and software, reason I ask is; if it is then its unlikely that a hardware fault would only effect calibre library data and not more critical things that Linux itself needs.

Yes, the same drive as Linux and software. Has only the 1 drive.

Yes to see if the lost files have been copied to some obscure location. Many (most ?) 'calibre lost my books' issues are caused by the folders/files being moved by someone/something - often as a result of mouse drag and drop slips.

I don't have any lost books at the moment so a search only turn up those in my Book folder. [I'm the only one who uses this computer]

Search for the calibre file name 'author - title', you might find the 'missing' files in a 'how did it get there' location - eg your music library Or, search for the book format you lost (e.g. EPUB, MOBI or AZW3 etc) and sort the results on location


Not familiar with Mint Backup - but if you backup the calibre library(ies) along with everything else then "why only my book files" indeed

Suggest you resolve the existing anomalies from Check Library, and then run it regularly, you can do that from the command line see calibredb --check_library command. I run it at end-of-day, and look at the CSV next day, if something does go awry I usually (always ?) know who to blame

I'll see what happens when I run Check Library again when I'm done.

BR

As ever, thanks to you guys for helping me.

[eschwartz]

Originally Posted by BetterRed

I just added a 3.09 Gigabyte ISO to a book, it took several minutes to copy the ISO into the book folder (from elsewhere on the same drive), in fact long enough for me to go the bathroom, and put the kettle on. When it finished I renamed the book and it took less than a second.

Please explain

BR

IIRC calibre uses hardlinks as long as the underlying filesystem supports it.
Like copying only faster. And understandably enough not an option for importing.

[BetterRed]

Originally Posted by eschwartz

IIRC calibre uses hardlinks as long as the underlying filesystem supports it.

Which is what I wrote in post 22, and which theducks disputes. My 'Please explain' was directed to him. Those two words have a specific resonance down under relating to matters political

BR

[theducks]

Originally Posted by BetterRed

Which is what I wrote in post 22, and which theducks disputes. My 'Please explain' was directed to him. Those two words have a specific resonance down under relating to matters political

BR

That says, both our answers are correct. DEPENDS on ... !

[BetterRed]

Originally Posted by theducks

That says, both our answers are correct. DEPENDS on ... !

I say what calibre does on a rename, depends on the file system, what do you say it depends on?

I've found that if calibre encounters a locked file it always throws an error message - it only happens in Windows because Linux and OSX don't have file locking. If it happens in a rename the locked file is left as is, in its current location, which will show up as errors in Check Library. I'm very familiar with this because I sometimes use Word 2007 which locks the files it has open, so I have to be careful to close them before I do anything with them in calibre - convert, rename, embed meta etc.

BR