Fixed (beta) libraries for iOS 8 + Windows + Marvin

[Beamis]

Shellshock - first, thank you for your efforts in trying to get this connectivity back. Its very much appreciated, by those of us who don't know a compiler from an interpreter, but still like using computers.

I've followed your instructions re installing the 64 bit DLLs; I can connect to my ipad 3 under Windows 8.1 w/ Marvin, but now getting this issue:
DEBUG: 30.7 iOSReaderApp:free_space()
DeviceJob: 1 Get device information done, calling callback
DeviceJob: 1 Get device information callback returned
DEBUG: 30.8 iOSReaderApp:set_progress_reporter()
Job: 2 Set library information started
DeviceJob: 2 Set library information done, calling callback
DeviceJob: 2 Set library information callback returned
DEBUG: 30.8 iOSReaderApp:set_progress_reporter()
Job: 3 Get list of books on device started
DEBUG: 30.8 iOSReaderApp:books()
DEBUG: 30.8 iOSReaderApp:_localize_database_path()
DEBUG: 30.8 copying local_db from u'/Library/mainDb.sqlite'
Job: 1 Get device information finished
No details available.
Job: 2 Set library information finished
No details available.
DEBUG: 31.4 iOSReaderApp:_restore_from_snapshot()
DEBUG: 31.4 Analyzing local booklist
DeviceJob: 3 Get list of books on device done, calling callback
must be string or buffer, not None

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 87, in run
File "site-packages\calibre\gui2\device.py", line 502, in _books
File "C:\Users\JIMIRE~1\AppData\Local\Temp\calibre_hikb rb\_mkbmg_ios_local_db\Marvin_overlays.py", line 285, in books
File "C:\Users\JIMIRE~1\AppData\Local\Temp\calibre_hikb rb\_mkbmg_ios_local_db\Marvin_overlays.py", line 2352, in _restore_from_snapshot
File "C:\Users\JIMIRE~1\AppData\Local\Temp\calibre_hikb rb\_mkbmg_ios_local_db\Marvin_overlays.py", line 2328, in _validate_mainDb_profile
File "C:\Users\JIMIRE~1\AppData\Local\Temp\calibre_hikb rb\_mkbmg_ios_local_db\Marvin_overlays.py", line 1579, in _compare_mainDb_profiles
File "C:\Users\JIMIRE~1\AppData\Local\Temp\calibre_hikb rb\_mkbmg_ios_local_db\Marvin_overlays.py", line 2111, in _profile_db
TypeError: must be string or buffer, not None

DeviceJob: 3 Get list of books on device callback returned
Job: 3 Get list of books on device finished
must be string or buffer, not None

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 87, in run
File "site-packages\calibre\gui2\device.py", line 502, in _books
File "C:\Users\JIMIRE~1\AppData\Local\Temp\calibre_hikb rb\_mkbmg_ios_local_db\Marvin_overlays.py", line 285, in books
File "C:\Users\JIMIRE~1\AppData\Local\Temp\calibre_hikb rb\_mkbmg_ios_local_db\Marvin_overlays.py", line 2352, in _restore_from_snapshot
File "C:\Users\JIMIRE~1\AppData\Local\Temp\calibre_hikb rb\_mkbmg_ios_local_db\Marvin_overlays.py", line 2328, in _validate_mainDb_profile
File "C:\Users\JIMIRE~1\AppData\Local\Temp\calibre_hikb rb\_mkbmg_ios_local_db\Marvin_overlays.py", line 1579, in _compare_mainDb_profiles
DEBUG: 42.8 iOSReaderApp:shutdown()
DEBUG: 42.8 iOSReaderApp:eject()
_db\Marvin_overlays.py", line 2111, in _profile_db
TypeError: must be string or buffer, not None

[Beamis]

I completely uninstalled the IOS and Marvin XD plugins, and cleaned every temp file / cached info directory related to them I could spot.

Reinstalled them and tried reconnecting - now a new error message:

calibre, version 2.8.0
ERROR: Error: Error communicating with device

database disk image is malformed

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 87, in run
File "site-packages\calibre\gui2\device.py", line 502, in _books
File "C:\Users\JOHNMU~1\AppData\Local\Temp\calibre_lahh j5\lnd04d_ios_local_db\Marvin_overlays.py", line 310, in books
DatabaseError: database disk image is malformed

[ShellShock]

@Beamis, please read the other posts in this thread for some guidance on the database disk image error.

[ShellShock]

Originally Posted by AndrewG

Another run of Calibre in debug mode, after the malformed database message failed to connect to the iPhone and yielded the attached log.

Thanks for the logs Andrew, I need to analyse them. I have a few things to try which may take a while...

[ShellShock]

Attached are some alpha dlls which I would be grateful if someone could use to try and repro the malformed database error. These dlls are a bit of a stab in the dark--I have effectively made the interface single threaded, so if the bug is due to the lack of thread synchronisation, then this should avoid it altogether. Sledgehammer and nut, and worth a punt. PS: I have never got the malformed database error in my dev environment, so I can't test this out myself.

Many thanks!

[Terisa de morgan]

I've tried them and I've got the same error.

[Terisa de morgan]

I've checked yesterday the database that the plugin had transferred to the computer and it was definitively wrong. Today I've transferred the database with DiskAid to PC and it's OK.

[AndrewG]

Hi ShellShock - not good news - sorry!

I uninstalled and reinstalled Marvin on my IOS8 iPhone, put the new dlls in place in Calibre and started transferring books to Marvin. The first 400 went OK, then the next 1000 (except one where Calibre didn't like the ebook format), then I tried another 1600 - this gave a time out error after exactly 700 books had transferred and 800 had staged. The log for this is tmpyvjwux.

I then restarted Calibre. 899 books transferred OK. The I tried 2000 and the program stopped responding after transferring 500 of them. The log is tmprlhapd.

I restarted and tried 1500 books. 700 transferred and the program again stopped responding. The log is tmpyp1hoa.

At this stage I reset the iPhone, rebooted Windows and then restarted again. I tried 800 books. These all transferred and then the Calibre process display changed to showing Upload, 100 books added, 50% and the running time froze. I came back for another look 20 minutes later and found that there had been a "Send Metadata error", then an apparent attempt to restart concluding in a "Get list error", and the Database Malformed error. The log is tmpv6g91b.

Here is the Database Malformed message in its entirety:

calibre, version 2.8.0
ERROR: Error: Error communicating with device

database disk image is malformed

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 87, in run
File "site-packages\calibre\gui2\device.py", line 532, in _sync_booklists
File "C:\Users\Andrew\AppData\Local\Temp\calibre_9lyh18 \bfnmcq_ios_local_db\Marvin_overlays.py", line 1009, in sync_booklists
File "C:\Users\Andrew\AppData\Local\Temp\calibre_9lyh18 \bfnmcq_ios_local_db\Marvin_overlays.py", line 2109, in _profile_db
DatabaseError: database disk image is malformed

[Dhardrian]

Am receiving the exact same error as stanhere. iOS 8.2 on iPad3 (ME392LL/A) connecting to Windows 7 64 bit running Calibre 2.8 64 bit with iOSRA 1.4.4 and Marvin XD 1.2.4 with ShellShock's 64 bit DLL replacements dated 11/3. Running Library maintenance function "Check Library" results in no database errors found. I have completed a Library metadata backup, renamed the DB file and attempted to perform a restore DB. This locked up partway through, potentially due to the LARGE size of my library (11,121 books currently). I have restored the backup DB file. Calibre opens and operates fine against the Library, however attempting to connect with Marvin results in the traceback.

Error traceback:

calibre, version 2.8.0
ERROR: Error: Error communicating with device

database disk image is malformed

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 87, in run
File "site-packages\calibre\gui2\device.py", line 502, in _books
File "C:\Users\Rob\AppData\Local\Temp\calibre_l7evvl\9q nrr4_ios_local_db\Marvin_overlays.py", line 285, in books
File "C:\Users\Rob\AppData\Local\Temp\calibre_l7evvl\9q nrr4_ios_local_db\Marvin_overlays.py", line 2352, in _restore_from_snapshot
File "C:\Users\Rob\AppData\Local\Temp\calibre_l7evvl\9q nrr4_ios_local_db\Marvin_overlays.py", line 2328, in _validate_mainDb_profile
File "C:\Users\Rob\AppData\Local\Temp\calibre_l7evvl\9q nrr4_ios_local_db\Marvin_overlays.py", line 1579, in _compare_mainDb_profiles
File "C:\Users\Rob\AppData\Local\Temp\calibre_l7evvl\9q nrr4_ios_local_db\Marvin_overlays.py", line 2108, in _profile_db
DatabaseError: database disk image is malformed

[Dhardrian]

Repeating with the Alpha DLL's results in the following traceback:

calibre, version 2.8.0
ERROR: Error: Error communicating with device

database disk image is malformed

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 87, in run
File "site-packages\calibre\gui2\device.py", line 502, in _books
File "C:\Users\Rob\AppData\Local\Temp\calibre_lb4ovn\8b fkoj_ios_local_db\Marvin_overlays.py", line 285, in books
File "C:\Users\Rob\AppData\Local\Temp\calibre_lb4ovn\8b fkoj_ios_local_db\Marvin_overlays.py", line 2352, in _restore_from_snapshot
File "C:\Users\Rob\AppData\Local\Temp\calibre_lb4ovn\8b fkoj_ios_local_db\Marvin_overlays.py", line 2328, in _validate_mainDb_profile
File "C:\Users\Rob\AppData\Local\Temp\calibre_lb4ovn\8b fkoj_ios_local_db\Marvin_overlays.py", line 1579, in _compare_mainDb_profiles
File "C:\Users\Rob\AppData\Local\Temp\calibre_lb4ovn\8b fkoj_ios_local_db\Marvin_overlays.py", line 2108, in _profile_db
DatabaseError: database disk image is malformed

[Dhardrian]

Originally Posted by MSWallack

Given the number of books I have in Marvin, I'll hold off trying to test things...

For what its worth MSWallack, this malformed database error has not had any impact on the books I already have in Marvin on my iPad. I am simply unable to connect.

[Dhardrian]

Originally Posted by ShellShock

I would be grateful if someone who gets the malformed database error could be a guinea pig and generate a debug log for me when the error occurs:

Customise the IOS reader and Marvin plugins to enable debug logging (two check boxes each).
Right click the Perferences toolbar button in Calibre and choose Restart in debug mode.
Reproduce the problem, starting with a good database and send a bunch of books until the problem occurs.

Warning - the debug log may get very big, and we may need several goes at this (e.g., I may need to add more debug logging to the code).

Thanks

Shellshock, I have generated a debug log for you. I was not able to give you exactly what you want, as I can not start with an empty library in Marvin on the iPad as I am in need of whats on it currently.

So, the result is a debug log in which I attempt to connect Calibre to Marvin on an iPad with 11,147 books in the library (Calibre currently has 11,121, as I have added books to Marvin directly via Safari)

[chookie2]

Have read all instructions and posts but think i should wait for Calibre upgrade to fix it all ( if possible) i really do not understand dll etc stuff. Am managing Marvin with opds at the moment...at least it is working, still get msg to reinstall on Calibre but can put up with it a bit longer. Pretty sure i would mess things up if I try the fix. Thanks for all your work Shellshock, much appreciated.

[stanhere]

FWIW - Status Report for those also operating off W7 32&64 OS.................

Am still running fine after updating from Calibre 64 v2.8 to 2.9 with iOSRA 1.4.4 & Marvin XD 1.2.4 Plugs AND ShellShock's 64 bit DLL replacements connecting to iPad retina with iOS 8.2.

My updated book collection on both iPad & W7 64 Calibre v2.9 is now at 2,248 without a return of the "Malformed DB ....." 'monkey-wrench' exception. Hopefully this continues!

In the event the 'monkey-wrench returns', I expect to restore to "normal" and up-to-date operation within minutes per my prior post #75.

Regards to ShellShock and his continuing efforts!.

[ShellShock]

Thanks @stanhere; I will have another look at the malformed database error this weekend, but I am beginning to think it is outside of my remit, e.g., in a component over which I have no control. If it is an error in the SQLLite database used by Marvin on the device, then trying to fix it via the interface between the Calibre plugins and Marvin is coming at it from the wrong direction.