Error when trying to load multiple books on my Kobo Aura HD

[Iskariot]

When trying to upload multiple books to my Kobo Aura HD I get the following error.
I never had these problems before.
The same books had already been on my reader, but after I installed the plugin that creates kepubs from epubs I wanted to reload them.
I do not get this error when I upload each and every book one by one.
Any idea what causes this?


calibre, version 1.26.0
ERROR: Unhandled exception: <b>OSError</b>:[Errno 13] Permission denied

calibre 1.26 [64bit] isfrozen: True is64bit: True
Windows-7-6.1.7601-SP1 Windows ('64bit', 'WindowsPE')
('Windows', '7', '6.1.7601')
Python 2.7.5
Windows: ('7', '6.1.7601', 'SP1', 'Multiprocessor Free')
Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 1234, in dispatch_sync_event
File "site-packages\calibre\gui2\device.py", line 1436, in sync_to_device
File "site-packages\calibre\gui2\library\models.py", line 584, in get_preferred_formats_from_ids
File "site-packages\calibre\db\legacy.py", line 340, in copy_format_to
File "site-packages\calibre\db\cache.py", line 57, in call_func_with_lock
File "site-packages\calibre\db\cache.py", line 667, in copy_format_to
File "site-packages\calibre\db\backend.py", line 1323, in copy_format_to
File "site-packages\calibre\startup.py", line 155, in local_open
OSError: [Errno 13] Permission denied

[kovidgoyal]

Something on your computer is preventing calibre from writing to its own temporary files. Since you say if happens only when sending multiple books, I would guess it is an antivirus program that gets triggerred when many temp files are created.

[Iskariot]

Originally Posted by kovidgoyal

Something on your computer is preventing calibre from writing to its own temporary files. Since you say if happens only when sending multiple books, I would guess it is an antivirus program that gets triggerred when many temp files are created.

O.k. I will look into that.
I have been running with the same set of security programs for a long time, and I know them quite well, but perhaps a recent update changed something in the behavior of one.

Thanks for the info. I would never have thought of this possibility otherwise.

[Iskariot]

The problem is not that I wanted to copy multiple files, but that some files in the selection were denied access.

At first I thought it might be Skydrive, but disabling skydrive doe not change anything.

I cannot even open certain books to read them. I cannot move them, copy them or delete them.

Restarting my PC does not change anything.

For example when I try to open The Alastor Omnibus by Jack Vance I get a slightly different error:

calibre, version 1.26.0
ERROR: Could not open ebook: [Errno 13] Permission denied: u'I:\\MICROSOFT SKYDRIVE\\SkyDrive\\CALIBRE LIBRARY\\Jack Vance\\[ALASTOR0] Alastor Omnibus (LIC) (14215)\\[ALASTOR0] Alastor Omnibus (LIC - Jack Vance.epub'

Traceback (most recent call last):
File "site-packages\calibre\gui2\viewer\main.py", line 40, in run
File "threading.py", line 761, in run
File "site-packages\calibre\ebooks\oeb\iterator\book.py", line 96, in __enter__
IOError: [Errno 13] Permission denied: u'I:\\MICROSOFT SKYDRIVE\\SkyDrive\\CALIBRE LIBRARY\\Jack Vance\\[ALASTOR0] Alastor Omnibus (LIC) (14215)\\[ALASTOR0] Alastor Omnibus (LIC - Jack Vance.epub'

[kovidgoyal]

http://manual.calibre-ebook.com/faq....n-denied-error

and

http://manual.calibre-ebook.com/faq....rked-drive-nas

[BetterRed]

@Iskariot - you can give calibre a 'dedicated' temporary location instead of having it share the common temporary location, see ==>> http://manual.calibre-ebook.com/cust...ment-variables

This would enable you to easily exclude calibres temporary location from your AV, whilst letting it continue to monitor the common temporary location used by everything else.

Also if your using 64bit calibre then try installing the 32bit version (they can be installed alongside one another). Its overcome some other Permissions problems, I've no idea why.

BR

[Iskariot]

@ kovidgoyal

As far as I can see none of the options in the links seem to apply.
As far as I know now only three of the books are blocked.
I wanted to load 80 or so books on my Kobo and only 3 of them gave me the same error.
I still suspect Microsoft's skydrive has something to do with it.
I will have to look into it some more.
I never encountered this problem before.

It is not difficult to find a workaround for the problem (I can use a backup, or I can boot with my other OS), so I will be able to get the books onto my Kobo for sure.

@BetterRed

Giving calibre a dedicated temp dir might be a good idea anyway. Perhaps it can keep calibre and skydrive out of each other's hair.
Skydrive has crashed on me several times lately when I was doing some work on my library with calibre.
At the moment I do not believe my AV has anything to do with the problem.

I do not believe using the 32bit version will help because I cannot access the files outside of calibre either.
I am quite sure calibre is not the problem here.

Thank you very much for your input.

[BetterRed]

Originally Posted by Iskariot

Giving calibre a dedicated temp dir might be a good idea anyway.

Perhaps it can keep calibre and skydrive out of each other's hair.
Skydrive has crashed on me several times lately when I was doing some work on my library with calibre.

@Iskariot - I wish more programs had the ability to set up dedicated locations for things like temp etc - I loathe having to go fishing in there for some log file or other.

Running calibre and a cloud synching service concurrently should be avoided.
See para 3 ==>> http://manual.calibre-ebook.com/faq....rked-drive-nas

Also https://www.mobileread.com/forums/sho...d.php?t=235125

BR

[Iskariot]

Originally Posted by BetterRed

@Iskariot
Running calibre and a cloud synching service concurrently should be avoided.
See para 3 ==>> http://manual.calibre-ebook.com/faq....rked-drive-nas

Also https://www.mobileread.com/forums/sho...d.php?t=235125

BR

Okay... slowly it is becoming clear.
Now I feel a little bit embarrassed.

I always thought that in principle a cloud synching service in general would be smart enough to detect whether files are used, or a directory is being worked in, and thus refrain from interfering.

Perhaps I overestimated the capabilities of the software.

Just to be sure. Google Drive is specifically mentioned as being incompatible, but the same goes for Microft's Skydrive?

[eschwartz]

Originally Posted by Iskariot

Okay... slowly it is becoming clear.
Now I feel a little bit embarrassed.

I always thought that in principle a cloud synching service in general would be smart enough to detect whether files are used, or a directory is being worked in, and thus refrain from interfering.

Perhaps I overestimated the capabilities of the software.

Just to be sure. Google Drive is specifically mentioned as being incompatible, but the same goes for Microft's Skydrive?

GDrive uses the number in parentheses at the end of book folders for version control, and thus is incompatible with calibre. No other syncing program does that. SkyDrive is perfectly safe, just don't try opening it in two programs at once. SkyDrive is a program opening the file(s).

[BetterRed]

Originally Posted by Iskariot

Okay... slowly it is becoming clear.
Now I feel a little bit embarrassed.

I always thought that in principle a cloud synching service in general would be smart enough to detect whether files are used, or a directory is being worked in, and thus refrain from interfering.

Perhaps I overestimated the capabilities of the software.

Just to be sure. Google Drive is specifically mentioned as being incompatible, but the same goes for Microft's Skydrive?

Pretty sure same goes for most cloud services - Dropbox, Skydrive, Copy.com, iCloud etc - disable synching when calibre is in use.

Google Drive creates specific problems for calibre, it renames the folders, which means they become 'detached' from their library database - metadata.db.

Many operating/file system combinations do not provide the facilities for something like a cloud synch program to detect if a file is in use by another program. Even when the OS/FS does provide the facilities then their usage is optional. Its a can of worms, happy to elaborate or find some links if you're interested.

BR

[Iskariot]

Originally Posted by BetterRed

Pretty sure same goes for most cloud services - Dropbox, Skydrive, Copy.com, iCloud etc - disable synching when calibre is in use.

Google Drive creates specific problems for calibre, it renames the folders, which means they become 'detached' from their library database - metadata.db.

That seems madness. Why not just copy to a temp dir before synching?


Oh well, from now on I will make sure Skydrive is disabled.
Still weird though that these three files are totally unusable even after rebooting.
Also, I have been using Skydrive for a few years now and never encountered problems like these before.

[BetterRed]

Originally Posted by Iskariot

That seems madness. Why not just copy to a temp dir before synching?

Because that would add both overhead, and further complications. The more one tries to work around what are fundamentally primitive operating and file systems the deeper becomes the hole in which you find yourself (I'm speaking for the people who write Skydrive, Dropbox etc)

Also see below - these services are designed to be used with one-at-a-time files.

Originally Posted by Iskariot

Oh well, from now on I will make sure Skydrive is disabled.

That's the best and only way to safely use calibre in conjunction with cloud synching.

Originally Posted by Iskariot

Still weird though that these three files are totally unusable even after rebooting.

Where & what are these three files - when you say they are unusable can you see them in Windows Explorer and view their properties?

Originally Posted by Iskariot

Also, I have been using Skydrive for a few years now and never encountered problems like these before.

You've probably been using it with one-at-a-time files like word processing files, images, spreadsheets etc.

The basic unit of processing in calibre is a book, which is a folder with more than one file - a cover, an opf file and one or more format files (epub, pdf etc). In addition to that, all the book files are tied together via the Catalogue database - metadata.db.

Consequently its the whole library that needs to be kept in synch, not just individual files within it. Products like Dropbox and Skydrive cannot cope with such scenarios, and IMO, nor should they be expected to do so.

BR

[Iskariot]

Originally Posted by BetterRed

Where & what are these three files - when you say they are unusable can you see them in Windows Explorer and view their properties?

They were 3 epub files.
I already used an unlocker tool to delete them. And replaced them from a backup. Problem solved.

I get the feeling it wasn't Skydrive that locked them after all, because I rebooted with skydrive deactivated and the system still gave me the same errors I mentioned before. I could not open, move or delete the epubs from within calibre, nor from outside calibre.

But all is well now.

You've probably been using it with one-at-a-time files like word processing files, images, spreadsheets etc.

No I have always used Skydrive with my library. Actually that was the main reason I started using SKydrive.

The basic unit of processing in calibre is a book, which is a folder with more than one file - a cover, an opf file and one or more format files (epub, pdf etc). In addition to that, all the book files are tied together via the Catalogue database - metadata.db.

Consequently its the whole library that needs to be kept in synch, not just individual files within it. Products like Dropbox and Skydrive cannot cope with such scenarios, and IMO, nor should they be expected to do so.

But does skydrive not just check which files have been used/changed/opened (timestamp?) and then sync them? So, if I change a book file would not all files involved, whether it be cover, opf or db etc. always get synched?

I always thought I understood the process reasonably well, but now I'm starting to get doubts.

[DoctorOhh]

Originally Posted by Iskariot

But does skydrive not just check which files have been used/changed/opened (timestamp?) and then sync them? So, if I change a book file would not all files involved, whether it be cover, opf or db etc. always get synched?

You understand exactly. As you change files they are synced. I've been using calibre with Dropbox for 4 years and have never had any files lock the way you described. Although I have had files lock like that before (not in dropbox). It is best with the cloud software to pause syncing while actively working on your library.

Originally Posted by Iskariot

I always thought I understood the process reasonably well, but now I'm starting to get doubts.

You understand fine, although it may have been OneDrive that locked up your files. Even if OneDrive wasn't running the file could have still been locked up by the process. Stuff happens, that is why that tool that allowed you to delete them exists.