|
|
Thread Tools | Search this Thread |
11-02-2018, 12:40 PM | #1 |
Junior Member
Posts: 7
Karma: 10
Join Date: Oct 2018
Device: Android/Moon+ Reader
|
Calibre-Server can't write to DB with Google Cloud Storage / GCSFuse
As an experiment I tried hooking up Calibre-server to a Google Cloud Storage bucket using GCSFuse. Probably not the best idea to hook up Calibre to a high-latency storage bucket, but I wanted to see if it would work.
It seems fairly fine, except I can't add books. It returns a 500 error and I get this traceback message on the server pointing to issues writing to the sqlite db holding the library. How do I trace and diagnose this issue to see whether it's a Calibre issue, an Sqlite library issue, a GCSFuse issue, or something else? this is with calibre 3.21 and gcsfuse 0.23.0 Code:
Unhandled exception in state: State: _job_done Client: 127.0.0.1:52764 Request: POST /cdb/add-book/1/y/[book filename]/calibre HTTP/1.0 Traceback (most recent call last): File "/usr/lib/calibre/calibre/srv/loop.py", line 569, in tick conn.handle_event(event) File "/usr/lib/calibre/calibre/srv/loop.py", line 307, in _job_done self.job_done(*event) File "/usr/lib/calibre/calibre/srv/pool.py", line 35, in run result = func() File "/usr/lib/calibre/calibre/srv/http_response.py", line 441, in run_request_handler result = self.request_handler(data) File "/usr/lib/calibre/calibre/srv/routes.py", line 313, in dispatch ans = endpoint_(self.ctx, data, *args) File "/usr/lib/calibre/calibre/srv/cdb.py", line 92, in cdb_add_book ids, duplicates = db.add_books([(mi, {fmt: rd.request_body_file})], add_duplicates=add_duplicates) File "/usr/lib/calibre/calibre/db/cache.py", line 1596, in add_books book_id = self.create_book_entry(mi, add_duplicates=add_duplicates, apply_import_tags=apply_import_tags, preserve_uuid=preserve_uuid) File "/usr/lib/calibre/calibre/db/cache.py", line 63, in call_func_with_lock return func(*args, **kwargs) File "/usr/lib/calibre/calibre/db/cache.py", line 1566, in create_book_entry self._set_metadata(book_id, mi, ignore_errors=True) File "/usr/lib/calibre/calibre/db/cache.py", line 1276, in set_metadata set_field('title', mi.title) File "/usr/lib/calibre/calibre/db/cache.py", line 1271, in set_field dirtied.update(self._set_field(name, {book_id:val}, do_path_update=False, allow_case_change=allow_case_change)) File "/usr/lib/calibre/calibre/db/cache.py", line 1101, in set_field self._mark_as_dirty(dirtied) File "/usr/lib/calibre/calibre/db/cache.py", line 1045, in mark_as_dirty ((x,) for x in new_dirtied)) File "/usr/lib/calibre/calibre/db/backend.py", line 863, in executemany self.reopen(force=True) File "/usr/lib/calibre/calibre/db/backend.py", line 1082, in reopen self.conn File "/usr/lib/calibre/calibre/db/backend.py", line 836, in conn self._conn = Connection(self.dbpath) File "/usr/lib/calibre/calibre/db/backend.py", line 270, in __init__ self.execute('pragma cache_size=-5000') File "/usr/lib/calibre/calibre/db/backend.py", line 314, in execute return cursor.execute(sql, bindings) File "src/cursor.c", line 1019, in APSWCursor_execute.sqlite3_prepare File "src/statementcache.c", line 386, in sqlite3_prepare IOError: IOError: disk I/O error |
11-02-2018, 12:52 PM | #2 |
Junior Member
Posts: 7
Karma: 10
Join Date: Oct 2018
Device: Android/Moon+ Reader
|
Also, if you're thinking of doing this; a dangerous and experimental idea that will probably end up losing data and corrupting your database (Google Cloud Storage is non-locking and probably non-consistent), I'll write a full-fledged guide at some point but these are a few pitfalls I ran into trying to set it up:
1. Make sure your instance has write access to your cloud storage bucket - calibre won't load and you will get an error relating to a capitalization test if you can't write to the bucket. 2. if you copy your calibre library to GCS using gsutil (or anything that's not gcsfuse), make sure to enable Code:
--implicit-dirs |
11-02-2018, 01:09 PM | #3 |
creator of calibre
Posts: 43,860
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
I'O errors mean the sqlite tried to read/write to the db file on disk and the OS reported a failed operation. ALmost always a sign of an error in a layer below sqlite.
|
11-02-2018, 01:11 PM | #4 |
Junior Member
Posts: 7
Karma: 10
Join Date: Oct 2018
Device: Android/Moon+ Reader
|
(I also realized I posted this to the wrong category, but I can't move or delete a post so... apologies)
|
11-02-2018, 01:14 PM | #5 |
Junior Member
Posts: 7
Karma: 10
Join Date: Oct 2018
Device: Android/Moon+ Reader
|
Thanks for the advice Kovid!
So my next step will be to see if I can make changes against a copy of the database in sqlite and see if it returns more detailed errors, and try to see if sqlite is making an unsupported FS call to gcsfuse. |
11-02-2018, 01:20 PM | #6 |
creator of calibre
Posts: 43,860
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Its unlikely to be that simple, with filesystems the usual culprit is not an unsupported call, but a pattern of calls that has some side-effect that is not supposed to happen. So you do calls X Y Z and they work fine on filesystem's A B C but cause errors with filesystem M.
Filesystem semantics really are not designed for remote storage, all the various efforts to shoehorn neetworks into filesystem layers sooner or later trip up against this. |
11-05-2018, 05:25 PM | #7 |
Junior Member
Posts: 7
Karma: 10
Join Date: Oct 2018
Device: Android/Moon+ Reader
|
Yep. After a bit of tinkering involving my calibre library being corrupted ~3 times, I decided that the smarter thing to do would be to attach an actual disk (they call them "persistent storage disks"), and magically all of the errors I get (previews not being able to display, the database not being written to) disappear.
So if anyone's reading this in the future, attach a damn disk, don't use GCS right now. |
Tags |
calibre-server, gcsfuse, google cloud storage |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Calibre + cloud eBook storage plugin | geometer | Development | 4 | 11-27-2014 04:04 PM |
Google Cloud Storage vs. Dropbox | chilady1 | Marvin | 2 | 03-18-2014 01:00 PM |
Cloud Storage For Kindle Books and Calibre | jscarbo | Library Management | 3 | 12-10-2013 06:41 AM |
Calibre Library Cloud Storage Review: Copy.com | DoctorOhh | Related Tools | 5 | 07-13-2013 05:20 PM |
Calibre Portable and Cloud Storage | library.ian | Related Tools | 3 | 01-14-2013 08:02 PM |