Fixed (beta) libraries for iOS 8 + Windows + Marvin

[pwalker8]

Originally Posted by ShellShock

Thanks @stanhere; I will have another look at the malformed database error this weekend, but I am beginning to think it is outside of my remit, e.g., in a component over which I have no control. If it is an error in the SQLLite database used by Marvin on the device, then trying to fix it via the interface between the Calibre plugins and Marvin is coming at it from the wrong direction.

Hum, if it's a problem with the database, then it seems odd that the problem would only occur when the plugin is trying to connect.

[Dhardrian]

Seems to me that it's not a problem with the database on the iDevice, or the database in Calibre... since independently both operate just fine. They are simply unable to communicate, which leads me to believe that the plugin is simply not able to read the database properly, not that it is truly corrupt. Not knowing the architecture and code in general... I could be way off base, but it seems logical to me.

[Terisa de morgan]

Originally Posted by Dhardrian

Seems to me that it's not a problem with the database on the iDevice, or the database in Calibre... since independently both operate just fine. They are simply unable to communicate, which leads me to believe that the plugin is simply not able to read the database properly, not that it is truly corrupt. Not knowing the architecture and code in general... I could be way off base, but it seems logical to me.

It's not corrupt. As I've said before, if I copy with DiskAid the same database that the plugin says it's wrong, it's open and handled with the database manager. The copy the plugin send to the computer, however, it's not able to be read.

[ShellShock]

I have been spelunking through the IOS reader plugin code and I am getting a better idea of what is going wrong. Generally, we get an error like this:

Job: 9 Send metadata to device finished
database disk image is malformed
	
	Traceback (most recent call last):
	  File "site-packages\calibre\gui2\device.py", line 87, in run
	  File "site-packages\calibre\gui2\device.py", line 532, in _sync_booklists
	  File "C:\Users\Adrian\AppData\Local\Temp\calibre_vzn1dh\c5a2ct_ios_local_db\Marvin_overlays.py", line 1009, in sync_booklists
	  File "C:\Users\Adrian\AppData\Local\Temp\calibre_vzn1dh\c5a2ct_ios_local_db\Marvin_overlays.py", line 2109, in _profile_db
	DatabaseError: database disk image is malformed

database disk image is malformed is actually a SQLLite error (e.g., http://techblog.dorogin.com/2011/05/...-image-is.html).

Here I show line 2109 in the Marvin_overlays.py in the IOS reader plugin:

        con = sqlite3.connect(self.local_db_path)
        with con:
            con.row_factory = sqlite3.Row
            cur = con.cursor()

            # Hash the titles and authors
            m = hashlib.md5()
            cur.execute('''SELECT Title, Author FROM Books''')
            rows = cur.fetchall()									# line 2109

So the IOS reader plugin is trying to execute the standard query "SELECT Title, Author FROM Books'' on the SQLLite database when the malformed error is thrown by SQLLite.

Note that the database file being used is self.local_db_path. Elsewhere in the plugin (in a function called _localize_database_path) this resolves to a temporary file on the local PC, e.g, C:\Users\Adrian\AppData\Local\Temp\calibre_e7c0ov\ rndqpy_ios_local_db\mainDb.sqlite. If you search the equivalent location C:\Users\User\AppData\Local\Temp on your PC for mainDB.sqlite you may get a surprise--personally I had two such files from previous Marvin+plugin sessions. You can use your favourite SQLLite client to open these files and examine the contents.

The _localize_database_path function in the IOS reader plugin copies the mainDB.sqlite from the idevice to the above temporary file on your PC. I suspect that this copy operation is buggy, e.g., it could be reading the file whilst it is still open in Marvin on the device, or it could still be writing the file to your Windows disk when the plugin tries to query it as a SQLLite database. The more books you have, the bigger the database and the more critical timings become.

Or it could be bug in the libimobiledevice code which reads the file down over a socket, and this is certainly something I am going to look into.

A relatively simple fix/kludge would be for the IOS reader plugin to catch the SQLLiteException and to retry the download of the database (this assumes the database is OK on the device, which seems to be Terisa's experience).

Some people have also reported timeout errors, which I also have some information on, but I will deal with that later I think.

[ShellShock]

Further to my previous post, I have been studying the Calibre library function, copy_from_idevice (in libimobiledevice.py) that copies a file from the idevice to your PC. This function is used by the IOS reader application (IOSRA) plugin to copy Marvin's SQLLite database, mainDb.sqlite, to your PC, so the plugin can query it to find out what books you have on your device.

A potential problem with copy_from_idevice is that it uses a read lock to read the remote file (Marvins's mainDb.sqlite on the idevice). That means Marvin could still be updating the mainDb.sqlite whilst IOSRA is merrily copying it, and thus you end up with a partial copy on your PC, and you get the "database disk image is malformed" error.

I have now rewritten copy_from_idevice to use a write lock. If Marvin still has mainDb.sqlite open, then copy_from_idevice will fail to open it, wait for a second, and then retry, up to 60 times. I have also added some safety code to ensure file handles are released if there is some other error whilst copying.

The fix should work if the idevice correctly obeys file locking with a write lock, and that SQLLite is also using a write lock.

I don't want to submit this code change into the Calibre GitHub until I know it fixes the bug, and as I have never been able to repro the bug myself, I would be grateful if someone here could test the fix for me. I have packaged up the fix into a replacement for the entire Calibre python code, pylib.zip. Please install as follows:

1. Make sure you are on Calibre 2.8, preferably 64 bit (not sure if that matters). The fix may work with Calibre 2.9, but I have not tested it.
2. Take a back up copy of C:\Program Files\Calibre\pylib.zip.
3. Download my new version from my dropbox: https://www.dropbox.com/s/y03anlaxhh...pylib.zip?dl=0 (50 MB, too big to attach here).
4. Copy the new version to C:\Program Files\Calibre.
5. Preferably run Calibre with debugging on, and upload a tonne of books to Marvin!

If the fix does not work I have something else up my sleeve...

Many thanks

[AndrewG]

Thanks ShellShock. I am testing your new pylib.zip in conjunction with your beta dlls (not the single threaded alpha ones) and Calibre 2.8.0 64 bit. I will advise the outcome in a few hours.

[AndrewG]

Hi ShellShock,

I think we can call this a partial success. I used Calibre 2.8.0 and your beta dlls and new pylib.zip. To summarise:

The process of copying books to Marvin stuck frequently at an exact multiple of 100 books, just when IOSRA should have been staging the next hundred.

Then there was one case where the "On Device" column on Calibre disappeared when the connection failed.

There was a new TypeError message.

I got malformed db messages associated with failure to send metadata to the device, but these did not prevent me transferring more books.

Finally a TypeError message was followed, after restart, by a fatal Malformed Db message and all further transfers became impossible.

Now blow by blow with logs:

I started trying to transfer the 400 largest books. The transfer stuck at 200 books, showing 200 of 400 staged. Log is tmp4u73g5.

After restarting I successfully transferred 200 books then tried 1098. This stuck at 700 books. Log is tmptrtadb.

After restarting I transferred 398 books successfully but then the "on device" column disappeared from Calibre and it became unresponsive. Log is tmprlxru2.

A restart yielded the type error message:

calibre 2.8 [64bit] isfrozen: True is64bit: True
Windows-7-6.1.7601-SP1 Windows ('64bit', 'WindowsPE')
('Windows', '7', '6.1.7601')
Python 2.7.8
Windows: ('7', '6.1.7601', 'SP1', 'Multiprocessor Free')
Successfully initialized third party plugins: DeDRM && Marvin XD && iOS reader applications
Traceback (most recent call last):
File "calibre_plugins.marvin_manager.action", line 1842, in on_device_connection_changed
File "libimobiledevice.py", line 365, in exists
File "libimobiledevice.py", line 1061, in _afc_get_file_info
TypeError: byref() argument must be a ctypes instance, not 'NoneType'

but Calibre continued to function and I attempted to transfer another 1500 books. This time it stuck at 900 books. Log is tmp_hqd80.

Another restart was followed by the successful transfer of 600 books and then the program sticking again. Log is tmptzswi5.

Another restart, 700 books transferred out of 1500, another stuck program. Log is tmptmj5dw.

After restart 800 books transferred but then I received, at the sending metadata stage, a Malformed Db message:

database disk image is malformed


Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 87, in run
File "site-packages\calibre\gui2\device.py", line 532, in _sync_booklists
File "C:\Users\Andrew\AppData\Local\Temp\calibre_bpv2hl \fa0znv_ios_local_db\Marvin_overlays.py", line 1009, in sync_booklists
File "C:\Users\Andrew\AppData\Local\Temp\calibre_bpv2hl \fa0znv_ios_local_db\Marvin_overlays.py", line 2109, in _profile_db
DatabaseError: database disk image is malformed

Oddly enough the transfer process continued to work and I transferred first 1 book, then 499, each time getting the malformed db message at the sending metadata stage. I tried sending 1000 books and it stuck at 800. The log is tmpjuchga.

Another restart gave me the type error message again.

A final restart gave me this malformed db message:

calibre, version 2.8.0
ERROR: Error: Error communicating with device

database disk image is malformed

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 87, in run
File "site-packages\calibre\gui2\device.py", line 502, in _books
File "C:\Users\Andrew\AppData\Local\Temp\calibre_lpf4iv \8fnwgu_ios_local_db\Marvin_overlays.py", line 285, in books
File "C:\Users\Andrew\AppData\Local\Temp\calibre_lpf4iv \8fnwgu_ios_local_db\Marvin_overlays.py", line 2352, in _restore_from_snapshot
File "C:\Users\Andrew\AppData\Local\Temp\calibre_lpf4iv \8fnwgu_ios_local_db\Marvin_overlays.py", line 2328, in _validate_mainDb_profile
File "C:\Users\Andrew\AppData\Local\Temp\calibre_lpf4iv \8fnwgu_ios_local_db\Marvin_overlays.py", line 1579, in _compare_mainDb_profiles
File "C:\Users\Andrew\AppData\Local\Temp\calibre_lpf4iv \8fnwgu_ios_local_db\Marvin_overlays.py", line 2109, in _profile_db
DatabaseError: database disk image is malformed

The final log is tmpzzbsvx.

All the best.

[AndrewG]

ShellShock,

I have also tried your new pylib.zip with your alpha dlls and Calibre 2.8.0.

It went better than with the beta dlls but produced many, but not all, of the same error conditions.

i transferred 400 books, then 1098 but at that point the "On Device" column disappeared from Calibre. Log is tmpboj4fc.

After a restart I transferred 400 of 1500 books selected for transfer and then the program stuck. Log is tmpx159wo.

After a restart I transferred 1100 books and then tried 1500, which timed out after 500 had been copied. Log is tmpxapz0e.

Restarted, transferred 800 out of 1000 books before program stuck. Log is tmptexh12.

Another restart, then transferred 700 books ok, then received "Malformed db error" - see file attached. Log is tmp4us134.

A final restart gave "TypeError" and "Err comm with device - malformed db" reports - see files attached.

[ShellShock]

I think I will concentrate on the "database disk image is malformed" error for now - I haven't seen the other errors before. Before running a test, please power off your IPad/IPhone, and disconnect the usb cable. Then reconnect, power on, start Marvin etc.

I have uploaded a new pylib.zip to dropbox (https://www.dropbox.com/s/y03anlaxhh...pylib.zip?dl=0). This time I have used a different method of locking the remote mainDb.sqlite file on the idevice. Thanks for trying this.

PS, the alpha dlls can be ignored.

[pwalker8]

I don't know if this will help you or not, but I'm having the same problem on an iMac with Yosemite. One thing that I notice is that when I plug my iPad mini in, a copy of the mainDb.sqlite file is created in one of the /var directories. This file is removed when I remove the iPad. When I open this file with an sqlite browser, the file is a valid sqlite file, but it's incomplete, i.e. it only has a couple of books in the books table rather than the 2000+ that it ought to have. If you start calibre in debug mode, you can look through the log and it will tell you where the temporary copy of the mainDb.sqlite file is stored.

[ShellShock]

Thank you pwalker, that mirrors the problem on Windows, it seems the SQLite database is not being correctly downloaded from the device, and I am working on various fixes to the download code, which should also fix it for Linux.

[AndrewG]

Good morning ShellShock,

I have tried out the latest pylib.zip with the beta dlls.

I transferred smaller numbers of books. Batches of 100 until I had transferred 2000 books, then 200s, 300s and 400s to a total of 4000. You will recall that I have sorted my books in order of decreasing size, so by the 4000 mark they are all smaller than 0.4 Mb each.

I then transferred another 500 books and received this error message:

calibre, version 2.8.0
ERROR: Error: Error communicating with device

database disk image is malformed

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 87, in run
File "site-packages\calibre\gui2\device.py", line 532, in _sync_booklists
File "C:\Users\Andrew\AppData\Local\Temp\calibre_9bts6z \o4sw7o_ios_local_db\Marvin_overlays.py", line 1009, in sync_booklists
File "C:\Users\Andrew\AppData\Local\Temp\calibre_9bts6z \o4sw7o_ios_local_db\Marvin_overlays.py", line 2109, in _profile_db
DatabaseError: database disk image is malformed

I was still able to transfer books and transferred 500, 1, then 499 more. Each time the transfer completed, when Calibre was sending metadata to the device it gave me precisely the same error message.

I disconnected Marvin, (the log up to this point is tmpfaenef) shut down both Calibre and Marvin, powered down the iPhone reconnected it, restarted Marvin and then Calibre. This time I received the following message:

calibre, version 2.8.0
ERROR: Error: Error communicating with device

database disk image is malformed

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 87, in run
File "site-packages\calibre\gui2\device.py", line 502, in _books
File "C:\Users\Andrew\AppData\Local\Temp\calibre_xtxjvc \g_3p3b_ios_local_db\Marvin_overlays.py", line 285, in books
File "C:\Users\Andrew\AppData\Local\Temp\calibre_xtxjvc \g_3p3b_ios_local_db\Marvin_overlays.py", line 2352, in _restore_from_snapshot
File "C:\Users\Andrew\AppData\Local\Temp\calibre_xtxjvc \g_3p3b_ios_local_db\Marvin_overlays.py", line 2328, in _validate_mainDb_profile
File "C:\Users\Andrew\AppData\Local\Temp\calibre_xtxjvc \g_3p3b_ios_local_db\Marvin_overlays.py", line 1579, in _compare_mainDb_profiles
File "C:\Users\Andrew\AppData\Local\Temp\calibre_xtxjvc \g_3p3b_ios_local_db\Marvin_overlays.py", line 2109, in _profile_db
DatabaseError: database disk image is malformed

I tried disconnecting and reconnecting Marvin but could not proceed further. Log is tmpiwi5rd.

Important points to note are, I think:

Calibre did not "stick" at any point.
There were no time-out or "Type" error messages.
The "on device" column did not disappear at any stage.

[pwalker8]

Originally Posted by ShellShock

Thank you pwalker, that mirrors the problem on Windows, it seems the SQLite database is not being correctly downloaded from the device, and I am working on various fixes to the download code, which should also fix it for Linux.

Glad to hear. Thanks for your effort. I downloaded your pylib.zip file and was trying to figure out what the matching files were on the mac.

[ShellShock]

@AndrewG thanks for the latest logs; I had added extra logging to the latest pylib.zip which was very useful. I think I know what is causing the error now and how to fix it. More later!

[Dchiplin]

Running Calibre 2.8.0 64 bit and latest Marvin and iOS Reader Application plug-ins on Windows Vista 64 bit.

I had 2 versions of Calibre installed - 32 bit and 64 bit.

Removed 32 bit and got the iOS Reader running, yeah!

And then got the Malformed Disk issue - boo!!

database disk image is malformed

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 87, in run
File "site-packages\calibre\gui2\device.py", line 502, in _books
File "C:\Users\Owner\AppData\Local\Temp\calibre_cqjiga\ 5xyukd_ios_local_db\Marvin_overlays.py", line 285, in books
File "C:\Users\Owner\AppData\Local\Temp\calibre_cqjiga\ 5xyukd_ios_local_db\Marvin_overlays.py", line 2352, in _restore_from_snapshot
File "C:\Users\Owner\AppData\Local\Temp\calibre_cqjiga\ 5xyukd_ios_local_db\Marvin_overlays.py", line 2328, in _validate_mainDb_profile
File "C:\Users\Owner\AppData\Local\Temp\calibre_cqjiga\ 5xyukd_ios_local_db\Marvin_overlays.py", line 1579, in _compare_mainDb_profiles
File "C:\Users\Owner\AppData\Local\Temp\calibre_cqjiga\ 5xyukd_ios_local_db\Marvin_overlays.py", line 2109, in _profile_db
DatabaseError: database disk image is malformed