Calibre-Server can't write to DB with Google Cloud Storage / GCSFuse

[rfilmyer]

As an experiment I tried hooking up Calibre-server to a Google Cloud Storage bucket using GCSFuse. Probably not the best idea to hook up Calibre to a high-latency storage bucket, but I wanted to see if it would work.
It seems fairly fine, except I can't add books. It returns a 500 error and I get this traceback message on the server pointing to issues writing to the sqlite db holding the library.
How do I trace and diagnose this issue to see whether it's a Calibre issue, an Sqlite library issue, a GCSFuse issue, or something else?

this is with calibre 3.21 and gcsfuse 0.23.0

Unhandled exception in state: State: _job_done Client: 127.0.0.1:52764 Request: POST /cdb/add-book/1/y/[book filename]/calibre HTTP/1.0
Traceback (most recent call last):
  File "/usr/lib/calibre/calibre/srv/loop.py", line 569, in tick
    conn.handle_event(event)
  File "/usr/lib/calibre/calibre/srv/loop.py", line 307, in _job_done
    self.job_done(*event)
  File "/usr/lib/calibre/calibre/srv/pool.py", line 35, in run
    result = func()
  File "/usr/lib/calibre/calibre/srv/http_response.py", line 441, in run_request_handler
    result = self.request_handler(data)
  File "/usr/lib/calibre/calibre/srv/routes.py", line 313, in dispatch
    ans = endpoint_(self.ctx, data, *args)
  File "/usr/lib/calibre/calibre/srv/cdb.py", line 92, in cdb_add_book
    ids, duplicates = db.add_books([(mi, {fmt: rd.request_body_file})], add_duplicates=add_duplicates)
  File "/usr/lib/calibre/calibre/db/cache.py", line 1596, in add_books
    book_id = self.create_book_entry(mi, add_duplicates=add_duplicates, apply_import_tags=apply_import_tags, preserve_uuid=preserve_uuid)
  File "/usr/lib/calibre/calibre/db/cache.py", line 63, in call_func_with_lock
    return func(*args, **kwargs)
  File "/usr/lib/calibre/calibre/db/cache.py", line 1566, in create_book_entry
    self._set_metadata(book_id, mi, ignore_errors=True)
  File "/usr/lib/calibre/calibre/db/cache.py", line 1276, in set_metadata
    set_field('title', mi.title)
  File "/usr/lib/calibre/calibre/db/cache.py", line 1271, in set_field
    dirtied.update(self._set_field(name, {book_id:val}, do_path_update=False, allow_case_change=allow_case_change))
  File "/usr/lib/calibre/calibre/db/cache.py", line 1101, in set_field
    self._mark_as_dirty(dirtied)
  File "/usr/lib/calibre/calibre/db/cache.py", line 1045, in mark_as_dirty
    ((x,) for x in new_dirtied))
  File "/usr/lib/calibre/calibre/db/backend.py", line 863, in executemany
    self.reopen(force=True)
  File "/usr/lib/calibre/calibre/db/backend.py", line 1082, in reopen
    self.conn
  File "/usr/lib/calibre/calibre/db/backend.py", line 836, in conn
    self._conn = Connection(self.dbpath)
  File "/usr/lib/calibre/calibre/db/backend.py", line 270, in __init__
    self.execute('pragma cache_size=-5000')
  File "/usr/lib/calibre/calibre/db/backend.py", line 314, in execute
    return cursor.execute(sql, bindings)
  File "src/cursor.c", line 1019, in APSWCursor_execute.sqlite3_prepare
  File "src/statementcache.c", line 386, in sqlite3_prepare
IOError: IOError: disk I/O error

[rfilmyer]

Also, if you're thinking of doing this; a dangerous and experimental idea that will probably end up losing data and corrupting your database (Google Cloud Storage is non-locking and probably non-consistent), I'll write a full-fledged guide at some point but these are a few pitfalls I ran into trying to set it up:

1. Make sure your instance has write access to your cloud storage bucket - calibre won't load and you will get an error relating to a capitalization test if you can't write to the bucket.
2. if you copy your calibre library to GCS using gsutil (or anything that's not gcsfuse), make sure to enable

--implicit-dirs

with gcsfuse or else you won't get cover art or metadata showing up. Not sure if you can run this only once or youhave to mount it like this every time.

[kovidgoyal]

I'O errors mean the sqlite tried to read/write to the db file on disk and the OS reported a failed operation. ALmost always a sign of an error in a layer below sqlite.

[rfilmyer]

(I also realized I posted this to the wrong category, but I can't move or delete a post so... apologies)

[rfilmyer]

Thanks for the advice Kovid!

So my next step will be to see if I can make changes against a copy of the database in sqlite and see if it returns more detailed errors, and try to see if sqlite is making an unsupported FS call to gcsfuse.

[kovidgoyal]

Its unlikely to be that simple, with filesystems the usual culprit is not an unsupported call, but a pattern of calls that has some side-effect that is not supposed to happen. So you do calls X Y Z and they work fine on filesystem's A B C but cause errors with filesystem M.

Filesystem semantics really are not designed for remote storage, all the various efforts to shoehorn neetworks into filesystem layers sooner or later trip up against this.

[rfilmyer]

Yep. After a bit of tinkering involving my calibre library being corrupted ~3 times, I decided that the smarter thing to do would be to attach an actual disk (they call them "persistent storage disks"), and magically all of the errors I get (previews not being able to display, the database not being written to) disappear.

So if anyone's reading this in the future, attach a damn disk, don't use GCS right now.