Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Reply
 
Thread Tools Search this Thread
Old 02-17-2014, 10:47 AM   #16
DoctorOhh
US Navy, Retired
DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.DoctorOhh ought to be getting tired of karma fortunes by now.
 
DoctorOhh's Avatar
 
Posts: 8,768
Karma: 12516053
Join Date: Feb 2009
Location: North Carolina
Device: Nexus 7
Quote:
Originally Posted by Rbneader View Post
Is the first item talking about synching to Dropbox/another online backup site, or about something else?
If the receiving library was in Dropbox and you did not pause syncing it is very possible that one or more books may not have made it to their destination. If you are going to do any bulk operations with a library in Dropbox (or other cloud) then pause syncing before doing anything.

Quote:
Originally Posted by Rbneader View Post
Until this is figured out I'm setting up a backup folder structure that will be my actual library.
Backups are always an excellent idea.

Quote:
Originally Posted by Rbneader View Post
It's less convenient, but a library that unpredictably deletes files is a library I can't rely on. I'm also setting up a whole new portable Calibre installation, in case it was just something about my old installation.
Calibre does not delete any files at anytime unless directed by the user. You are experiencing OS level issues or cloud sync software issues.

Good luck with your work around.
DoctorOhh is offline   Reply With Quote
Old 02-17-2014, 11:07 AM   #17
theducks
Grand Sorcerer
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 14,211
Karma: 5495470
Join Date: Aug 2009
Location: The (original) Silicon Valley, USA
Device: Galaxy Tab 2, Astak Pocket Pro, K4NT
Quote:
Originally Posted by Rbneader View Post
The last 3 don't apply here. Is the first item talking about synching to Dropbox/another online backup site, or about something else?
Dropbox or any other connection that can write (or block a calibre write) to the Library folder.

Write behind (delayed) caching is another culprit.

Antivirus scans can block a change. Exclude the Library folder from a 'realtime' scan.

IMHO another problem with using 'the cloud' : Everything in my library has been scanned BEFORE it arrived. With the cloud, it seems possible to inject a contaminated copy if real time scanning has been excluded for structure integrity purposes . (like preventing your problem?)

4+ years running Calibre. Every book loss case has been identified to: ME Something (stupid) I did.
theducks is offline   Reply With Quote
Old 02-17-2014, 02:20 PM   #18
Rbneader
Addict
Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.
 
Posts: 282
Karma: 1152061
Join Date: Mar 2012
Device: None
Quote:
Originally Posted by DoctorOhh View Post
If the receiving library was in Dropbox and you did not pause syncing it is very possible that one or more books may not have made it to their destination. If you are going to do any bulk operations with a library in Dropbox (or other cloud) then pause syncing before doing anything.

Good luck with your work around.
That's what I thought. I don't sync my library online anywhere, so that's not it.
Rbneader is offline   Reply With Quote
Old 02-17-2014, 02:27 PM   #19
Rbneader
Addict
Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.
 
Posts: 282
Karma: 1152061
Join Date: Mar 2012
Device: None
Quote:
Originally Posted by theducks View Post
Dropbox or any other connection that can write (or block a calibre write) to the Library folder.

Write behind (delayed) caching is another culprit.

Antivirus scans can block a change. Exclude the Library folder from a 'realtime' scan.

IMHO another problem with using 'the cloud' : Everything in my library has been scanned BEFORE it arrived. With the cloud, it seems possible to inject a contaminated copy if real time scanning has been excluded for structure integrity purposes . (like preventing your problem?)
I do have security software that might have decided to scan at exactly that moment. I'll tell it to not worry about my ebooks.

I don't sync my library anywhere, so it's not an issue with that.

If Calibre can't handle the standard Windows caching methods, than that is a problem with Calibre. I don't think that's the case, though. An OS feature that's been standard for years shouldn't be able to cause book loss except with egregiously inflexible coding. There would be many, many more reports of this issue if that was the case, so I don't think it's related to caching.

My workaround will at least keep books from being lost, so that's something. It's still incredibly frustrating. Hopefully I'll be able to reproduce it.

Could it be related to multiple libraries? I do have several different Calibre libraries and move books between them fairly often using Calibre's standard move-book functionality. If swapping books around isn't something that Calibre can handle well that's at least an explanation that can be worked around.
Rbneader is offline   Reply With Quote
Old 02-17-2014, 02:55 PM   #20
theducks
Grand Sorcerer
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 14,211
Karma: 5495470
Join Date: Aug 2009
Location: The (original) Silicon Valley, USA
Device: Galaxy Tab 2, Astak Pocket Pro, K4NT
Calibre does a set of transactions: folder(s) DB, cover, format(s)
Anything that seriously impacts their creation is the fault of that system as a normal Windows has no problem.
Millions of Users vs Your install.
theducks is offline   Reply With Quote
Old 02-17-2014, 04:23 PM   #21
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 2,830
Karma: 1682890
Join Date: Mar 2012
Location: NSW Australia
Device: none
Rbneader - If you have a book open in a 3rd party program when you move books between libraries things can go awry. Some 3rd party programs (eg Microsoft's Word and Trackers Xchange) will lock the files they have open and calibre will pop an error when you try to do the move, but many programs don't lock the files they have open and then...

BR
BetterRed is offline   Reply With Quote
Old 02-17-2014, 04:54 PM   #22
Rbneader
Addict
Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.Rbneader ought to be getting tired of karma fortunes by now.
 
Posts: 282
Karma: 1152061
Join Date: Mar 2012
Device: None
Quote:
Originally Posted by BetterRed View Post
Rbneader - If you have a book open in a 3rd party program when you move books between libraries things can go awry. Some 3rd party programs (eg Microsoft's Word and Trackers Xchange) will lock the files they have open and calibre will pop an error when you try to do the move, but many programs don't lock the files they have open and then...

BR
I know I didn't open the disappearing book in any other program. I might have had another book I was working on open, but that shouldn't cause a locking issue with the disappearing book. Unless maybe their IDs were the same behind the scenes or something like that. It would be strange if that was the case, but this whole thing is strange.
Rbneader is offline   Reply With Quote
Old 02-17-2014, 05:32 PM   #23
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 2,830
Karma: 1682890
Join Date: Mar 2012
Location: NSW Australia
Device: none
Quote:
Originally Posted by Rbneader View Post
I know I didn't open the disappearing book in any other program. I might have had another book I was working on open, but that shouldn't cause a locking issue with the disappearing book. Unless maybe their IDs were the same behind the scenes or something like that. It would be strange if that was the case, but this whole thing is strange.
Might you have changed an Author Name or Title when a format file was open that will result in a move, if a file was open in another program - eg Sigil IIRC it doesn't lock files - that can result in lost edits, but not lost books.

when you change a book title (which triggers a rename of the book folder) then IIRC calibre creates a folder with the new name, then it creates hardlinks to the files in the new folder, and then it trashes (shift/del) the old folder. But it then again it might actually be NTFS that does all that...

Do you have any gadgets/widgets/belles [sic] or whistles that might interfere with that process - I can't imagine what, but...

BR
BetterRed is offline   Reply With Quote
Old 02-17-2014, 07:55 PM   #24
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,016
Karma: 3679628
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Ubuntu Linux, Cybook Opus, Motorola Xoom with Mantano Premium
Calibre is very careful when moving files.

Copy first, then delete. If there was any problems with the copy, that would crash calibre or generate an error, before the delete. So I still say that it isn't calibre that caused your problems. If it was thousands would have reported problems a long time ago and it would have been fixed.

It is also unlikely to be a hardware error, since you seem to be able to use your computer and the harddrives.

Unfortunately that only(?) leaves user error. That you yourself somehow inadvertently deleted the books. At times like that it is good to have recent backups.

I don't trust myself or hardware, so I let my computer automatically backup my calibre library (and other stuff I don't want to loose) every hour to an extra harddrive on my computer, and daily to a NAS with redundancy. And to external USB hard drives now and then. And I have most of my library on my reading device. But I do trust calibre to not delete any books, unless I tell it to.
Adoby is offline   Reply With Quote
Old 02-17-2014, 08:59 PM   #25
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 2,830
Karma: 1682890
Join Date: Mar 2012
Location: NSW Australia
Device: none
@Rbneader - I just noticed that you started this thread back in July last year, I'm not sure if the problems you raised then were resolved. Following my post of 07-27-2013, 08:36 AM the thread was quiet until your post of 02-14-2014, 08:41 PM

I'm intrigued as to what happened in the intervening 6 months - was calibre working OK or did you stop using it or... whatever.

Around October last year there was an issue regarding temporary folders being created and not being removed when deleting (moving ?) books and formats. I can't recall precise details nor whether this created actual problems or if it was just a nuisance issue. And IIRC around August there was an issue about the truncation of file names being altered that caused some consternation, again I can't recall details or the ramifications.

So I'm wondering if your current problems are a continuation of the problems you were having in July, or whether they might be related to those couple of issues that occurred in Aug & Oct. And I'm hoping people with better memories than mine can throw in there 2 cents worth.

BR

Last edited by BetterRed; 02-17-2014 at 09:05 PM.
BetterRed is offline   Reply With Quote
Old 02-17-2014, 11:47 PM   #26
eschwartz
Irrational Optimist
eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.
 
Posts: 4,377
Karma: 5552846
Join Date: Nov 2012
Location: The Beaten Path, USA, Roundworld, This Side of Infinity
Device: Kindle Touch (Wifi only)
The temporary folders issue (when deleting a book) only happened on unusual filesystem mounts, like when I reported this using a school computer. It is a fat client running LTSP, so the filesystem is I believe mounted from the server computer. And it only kicked in after the book gets deleted, which if I understand correctly should be after the copy was made. Something to do with the new database backend having a new method of sending files to the trash.

The filename truncation was an issue involving a lack of proper support for long filenames (a Windows problem) and it meant that people expected to find their books under the usual {author}\{title} - {author}.{format} location, but didn't, since the unusually long field was given to the filesystem in shortened form. calibre kept perfect track of it, it just didn't play nice with people "looking under calibre's skirt" due to a Windows limitation.

As far as I am aware, neither should cause any problems with the books disappearing.
eschwartz is offline   Reply With Quote
Old 02-18-2014, 01:06 AM   #27
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 2,830
Karma: 1682890
Join Date: Mar 2012
Location: NSW Australia
Device: none
Quote:
Originally Posted by eschwartz View Post
The temporary folders issue (when deleting a book) only happened on unusual filesystem mounts, like when I reported this using a school computer.

It is a fat client running LTSP, so the filesystem is I believe mounted from the server computer. And it only kicked in after the book gets deleted, which if I understand correctly should be after the copy was made. Something to do with the new database backend having a new method of sending files to the trash.
I had them on a standard windows/ntfs rig, on a fixed internal drive. I also had a problem with hundreds of them turning up in the recycle bin

Both problems were repeatable but neither caused any loss of data or library corruption - Check Library saw them as extra author folders - but I first noticed them in my backup logs.

Both problems were fixed

Quote:
Originally Posted by eschwartz View Post
The filename truncation was an issue involving a lack of proper support for long filenames (a Windows problem) and it meant that people expected to find their books under the usual {author}\{title} - {author}.{format} location, but didn't, since the unusually long field was given to the filesystem in shortened form. calibre kept perfect track of it, it just didn't play nice with people "looking under calibre's skirt" due to a Windows limitation.

As far as I am aware, neither should cause any problems with the books disappearing.
I have a vague memory that the file name truncation length changed from from 36 to 34 or something like that - it seemed to disturb a few people - although I can't recall whether they were of a material or aesthetic kind.

I can't recall them leading to disappearing books either - but there's a first time for everything.

I don't have any formal reconciliation audit controls on my libraries so a 'book' or three could easily disappear from the two larger ones without me noticing.

BR
BetterRed is offline   Reply With Quote
Old 02-18-2014, 01:27 AM   #28
eschwartz
Irrational Optimist
eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.eschwartz ought to be getting tired of karma fortunes by now.
 
Posts: 4,377
Karma: 5552846
Join Date: Nov 2012
Location: The Beaten Path, USA, Roundworld, This Side of Infinity
Device: Kindle Touch (Wifi only)
Quote:
Originally Posted by BetterRed View Post
I had them on a standard windows/ntfs rig, on a fixed internal drive. I also had a problem with hundreds of them turning up in the recycle bin

Both problems were repeatable but neither caused any loss of data or library corruption - Check Library saw them as extra author folders - but I first noticed them in my backup logs.

Both problems were fixed
Well, the problem as Kovid explained it was:

Quote:
For some reason, your OS is raising permission denied (EPERM) errors
when calibre tries to delete a non-empty directory, instead of
ENOTEMPTY, which is causing the problem. Do you have your calibre
library on some kind of special filesystem?
and
Quote:
In any case, I have changed the code to not rely on the OS sending the
proper error code, it will now manually check if the directory is empty.
That should fix your problem.
Whatever the problem was, that is what caused it for me.

And I got them in the parent directory to my library folder (in "Calibre Libraries") -- also no loss of data, just a pain in the neck to clean up after. Either way, it shouldn't cause errors until after the book was copied, so...

Quote:
I have a vague memory that the file name truncation length changed from from 36 to 34 or something like that - it seemed to disturb a few people - although I can't recall whether they were of a material or aesthetic kind.
Aesthetic, definitely. The change was deliberate and calibre still does it.
eschwartz is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Calibre content-server doesn't seem to see new books offby1 Calibre 7 05-05-2013 10:58 AM
Calibre doesn't recognize .azw3 books Breed92014 Devices 6 04-15-2012 02:50 PM
Calibre 7.28 doesn't List Nook Books Kavon Calibre 0 11-15-2010 11:20 AM
K2 self-deletes 2 books at once BarryTX Amazon Kindle 14 04-09-2010 03:30 PM
Why doesn’t calibre let me store books in my own directory structure? darknessangel Calibre 28 04-08-2010 07:53 PM


All times are GMT -4. The time now is 04:11 PM.


MobileRead.com is a privately owned, operated and funded community.