[ShellShock]

I have been spelunking through the IOS reader plugin code and I am getting a better idea of what is going wrong. Generally, we get an error like this:

Job: 9 Send metadata to device finished
database disk image is malformed
	
	Traceback (most recent call last):
	  File "site-packages\calibre\gui2\device.py", line 87, in run
	  File "site-packages\calibre\gui2\device.py", line 532, in _sync_booklists
	  File "C:\Users\Adrian\AppData\Local\Temp\calibre_vzn1dh\c5a2ct_ios_local_db\Marvin_overlays.py", line 1009, in sync_booklists
	  File "C:\Users\Adrian\AppData\Local\Temp\calibre_vzn1dh\c5a2ct_ios_local_db\Marvin_overlays.py", line 2109, in _profile_db
	DatabaseError: database disk image is malformed

database disk image is malformed is actually a SQLLite error (e.g., http://techblog.dorogin.com/2011/05/...-image-is.html).

Here I show line 2109 in the Marvin_overlays.py in the IOS reader plugin:

        con = sqlite3.connect(self.local_db_path)
        with con:
            con.row_factory = sqlite3.Row
            cur = con.cursor()

            # Hash the titles and authors
            m = hashlib.md5()
            cur.execute('''SELECT Title, Author FROM Books''')
            rows = cur.fetchall()									# line 2109

So the IOS reader plugin is trying to execute the standard query "SELECT Title, Author FROM Books'' on the SQLLite database when the malformed error is thrown by SQLLite.

Note that the database file being used is self.local_db_path. Elsewhere in the plugin (in a function called _localize_database_path) this resolves to a temporary file on the local PC, e.g, C:\Users\Adrian\AppData\Local\Temp\calibre_e7c0ov\ rndqpy_ios_local_db\mainDb.sqlite. If you search the equivalent location C:\Users\User\AppData\Local\Temp on your PC for mainDB.sqlite you may get a surprise--personally I had two such files from previous Marvin+plugin sessions. You can use your favourite SQLLite client to open these files and examine the contents.

The _localize_database_path function in the IOS reader plugin copies the mainDB.sqlite from the idevice to the above temporary file on your PC. I suspect that this copy operation is buggy, e.g., it could be reading the file whilst it is still open in Marvin on the device, or it could still be writing the file to your Windows disk when the plugin tries to query it as a SQLLite database. The more books you have, the bigger the database and the more critical timings become.

Or it could be bug in the libimobiledevice code which reads the file down over a socket, and this is certainly something I am going to look into.

A relatively simple fix/kludge would be for the IOS reader plugin to catch the SQLLiteException and to retry the download of the database (this assumes the database is OK on the device, which seems to be Terisa's experience).

Some people have also reported timeout errors, which I also have some information on, but I will deal with that later I think.