TimeoutError with some books

[tomsem]

Cannot delete this post, maybe someone else can.

[gbm]

https://manual.calibre-ebook.com/faq...rked-drive-nas

bernie

Originally Posted by tomsem

I have gotten this error on a few books I added a while ago:

calibre, version 7.2.0
ERROR: Unhandled exception: <b>TimeoutError</b>:[Errno 60] Operation timed out: '/Users/thomas/Library/CloudStorage/OneDrive-Personal/calibre/Neil Druckmann/The Last of Us_ American Dreams (4406)/metadata.opf' -&gt; '/Users/thomas/Library/CloudStorage/OneDrive-Personal/calibre/.caltrash/b/4406/metadata.opf'

calibre 7.2  embedded-python: True
macOS-14.1.2-x86_64-i386-64bit Darwin ('64bit', '')
('Darwin', '23.1.0', 'Darwin Kernel Version 23.1.0: Mon Oct  9 21:27:27 PDT 2023; root:xnu-10002.41.9~6/RELEASE_X86_64')
Python 3.11.5
OSX: ('14.1.2', ('', '', ''), 'x86_64')
Interface language: None
Successfully initialized third party plugins: Gather KFX-ZIP (from KFX Input) (2, 7, 0) && DeDRM (10, 0, 3) && Package KFX (from KFX Input) (2, 7, 0) && BookFusion Plugin (0, 8, 1) && Ebook Translator (2, 2, 0) && EpubCheck (0, 2, 5) && EpubSplit (3, 6, 0) && Generate Cover (2, 3, 3) && Goodreads (1, 7, 9) && Goodreads Sync (1, 16, 4) && Hello World Plugin (1, 0, 0) && KFX metadata reader (from KFX Input) (2, 7, 0) && From KFX (2, 7, 0) && KFX Input (2, 7, 0) && Set KFX metadata (from KFX Output) (2, 5, 0) && KFX Output (2, 5, 0) && KindleUnpack - The Plugin (0, 83, 1) && Quality Check (1, 13, 6) && Reformat plugin (2, 4, 0)
Traceback (most recent call last):
  File "calibre/gui2/actions/delete.py", line 480, in delete_books
  File "calibre/gui2/actions/delete.py", line 457, in do_library_delete
  File "calibre/gui2/library/models.py", line 465, in delete_books_by_id
  File "calibre/db/cache.py", line 88, in call_func_with_lock
  File "calibre/db/cache.py", line 2221, in remove_books
  File "calibre/db/backend.py", line 2277, in remove_books
  File "calibre/db/backend.py", line 2164, in move_book_to_trash
  File "calibre/utils/copy_files.py", line 282, in copy_tree
  File "calibre/utils/copy_files.py", line 57, in copy_all
  File "shutil.py", line 436, in copy2
  File "shutil.py", line 262, in copyfile
  File "shutil.py", line 104, in _fastcopy_fcopyfile
  File "shutil.py", line 97, in _fastcopy_fcopyfile
TimeoutError: [Errno 60] Operation timed out: '/Users/thomas/Library/CloudStorage/OneDrive-Personal/calibre/Neil Druckmann/The Last of Us_ American Dreams (4406)/metadata.opf' -> '/Users/thomas/Library/CloudStorage/OneDrive-Personal/calibre/.caltrash/b/4406/metadata.opf'

I cannot edit metadata, view, convert. Delete does not work; I can delete only after removing the folder content.

So far I have gone back and reimported them, and everything is okay. Just wondering what might be the issue, and how many of these there are in my library that I need to fix...

[Quoth]

/Users/thomas/Library/CloudStorage/OneDrive-Personal/c

Never use a NAS, network share (Samba or real MS windows or other), Google, One Drive, Dropbox or any other Internet service for a running library. Those are only for sharing a copy of your local files (not direct access by an application).
Exclude all so called "cloud" services (some distant server that belongs to someone else) from any folders/directories used by programs like calibre. It will break many things. This has always been true, not just for calibre. Even if it has worked, it will break.

[tomsem]

The issue here was that the files were in cloud only. I just needed to download them and then everything works. An inconvenience, yes. A mystifying error message, yes.

But nothing is 'broken'.

[Quoth]

Broken by regular definition of broken, IT wise. It didn't work because you used remote files.
https://manual.calibre-ebook.com/faq...rked-drive-nas

That was the sort of error message I'd expect.

[tomsem]

The manual section you reference concerns issues with 'networked drives', in contrast to file syncing tool 'like DropBox' (OneDrive in this case). I'm not doing 'simultaneous access' scenario here, which calibre does not support. I use calibre on a single computer.

It's just to back up the files in a convenient way, and at some point local copy gets removed, which makes disk storage go further. And as such, sometimes the file I want has to be downloaded. There's no risk of file or database corruption associated with this. The filenames calibre uses are compatible with what can be stored in OneDrive.

I mostly work with recently added books, which have not yet been 'archived' and stubbed out (or whatever the term is). So it isn't that much of an inconvenience.

A more relevant error might be something like 'file is not present in local file system, needs to be downloaded from cloud'. But timeout error will do, I don't expect calibre to explicitly integrate with cloud services, or to support multi-user scenarios.

Sure: it would be nice if calibre would wait patiently for the file to download. I can, for example, select a file with Finder that is not locally stored, and tell it to open in calibre viewer, without explicitly downloading it first. And there's no error.

I did a little experiment and downloaded only the .opf, and not the content file. I was able to do everything without any errors.

So perhaps a fix would be for calibre to wait longer for .opf file to open. They're very small of course and as long as there's an internet connection, they will download very quickly.

[DNSB]

Perhaps calibre expects it's files to be available where they are supposed to be stored and not removed at the whim of a cloud storage solution?

As you may have noticed over the years from messages posted by others, using a one way sync to backup your library works quite well. What you are trying to use is a two way sync which is almost always a path to a corrupt library. If you really think there is no chance of corruption, you are kidding yourself.

[kovidgoyal]

There is no timeout in any file open function calibre uses. That timeout is coming from the kernel.

[tomsem]

Originally Posted by DNSB

Perhaps calibre expects it's files to be available where they are supposed to be stored and not removed at the whim of a cloud storage solution?

As you may have noticed over the years from messages posted by others, using a one way sync to backup your library works quite well. What you are trying to use is a two way sync which is almost always a path to a corrupt library. If you really think there is no chance of corruption, you are kidding yourself.

It is never actually a two way sync in this case, since the only place I add or delete or modify files in the calibre folder is on the computer where calibre is installed, using calibre only.

I (very) occasionally download to another device, but that is not going to corrupt anything.

File presence should be transparent to calibre: it's the only way something like OneDrive can be at all functional. As soon as an application asks system for file open it should start to download that file, and return success only after that's done (if offline or download fails obviously there will be an error). It works thus with most applications (I cannot claim for all of them, obviously).

As my experiment demonstrates, calibre is perfectly fine with operating on books if only the .opf is downloaded.

I get no error if I right click and open a not-downloaded .opf with a text editor.

[tomsem]

Originally Posted by kovidgoyal

There is no timeout in any file open function calibre uses. That timeout is coming from the kernel.

Thanks, we can just blame Apple and Microsoft.

[Quoth]

Originally Posted by tomsem

The manual section you reference concerns issues with 'networked drives', in contrast to file syncing tool 'like DropBox' (OneDrive in this case). I'm not doing 'simultaneous access' scenario here, which calibre does not support. I use calibre on a single computer.

1. OneDrive and other "cloud" solutions are a special (more dangerous/ less reliable than LAN shares) case of a networked drive.

2. It's not just about simultaneous access by two Calibre programs.

Calibre and many other programs only work reliably with "really" local drives (USB Mass Storage (but not MTP or USB Network) and SCSI and eSATA count as local drives).

File presence should be transparent to calibre: it's the only way something like OneDrive can be at all functional.

It's not actually really a local file. OneDrive is an even bigger illusion of a file than Windows Share mapped to a drive letter.

Only use "Cloud" storage explicitly with a file manager. Anything else is going to give grief eventually.

Originally Posted by DNSB

Perhaps calibre expects it's files to be available where they are supposed to be stored and not removed at the whim of a cloud storage solution?

As you may have noticed over the years from messages posted by others, using a one way sync to backup your library works quite well. What you are trying to use is a two way sync which is almost always a path to a corrupt library. If you really think there is no chance of corruption, you are kidding yourself.

All true. I use rsync to backup to my remote server, and also it needs to happen when calibre is closed.

OneDrive is using MS opaque i/o to a server of unknown location and security belonging to MS. It (and other services like Dropbox) are best suited to sharing a copy with another person when you don't have your own hosting. It's advantage and disadvantage compared to SFTP is the OS integration.

[tomsem]

OneDrive works great for my use cases, including maintaining backup of my calibre data, for the many years I have used it for that, with painless transition from one computer to another. I have not lost a single file, or had any unrecoverable issue, using any application.

The issue with calibre I related here is a minor inconvenience, not evidence that I'm skating on thin ice.

You offer a very technical argument and I am not qualified to evaluate it. You could be 100% correct, and yet it does not invalidate my experience with OneDrive, and in particular using it with calibre (of which it seems you have little).