[Device Plugin] iOS reader applications

[WJCronon]

Thanks, davidfor. Again, I really appreciate your good work and your willingness to help with this. I'm pretty sure I had in fact previously deleted all the plugin components you list in your most recent post, but just to make absolutely sure, I deleted the *entire* calibre folder from the Mac OS X Library/Preferences folder, so that the entire folder had to be rebuilt from scratch when I reinstalled calibre 2.1.0. Once calibre had done this, I reinstalled the two plugins, and got the following error report, close to but not quite identical to the one I shared above.

I tried turning on the debugging routines in each plug-in in turn, but each time got a "Device already detected" message saying that "A device (IOS reader applications) is already detected by calibre. If you wish to debug the detection of another device, first disconnect this device." (This note continues below the error report.)

calibre, version 2.1.0
ERROR: Error: Error communicating with device

database disk image is malformed

Traceback (most recent call last):
File "site-packages/calibre/gui2/device.py", line 86, in run
File "site-packages/calibre/gui2/device.py", line 501, in _books
File "/var/folders/vz/zhdqchxj0xv1fb7x_f2_x7nr0000gn/T/calibre_2.1.0_tmp_wODM4A/vjigGL_ios_local_db/Marvin_overlays.py", line 310, in books
DatabaseError: database disk image is malformed

But here's one additional piece of information that may be suggestive. I just tried connecting my iPhone to calibre 2.1.0, since it also runs Marvin XD, albeit with a much smaller number of books. It connected with no problem and I was able to load a book with no difficulty. The difference may be that my iPhone has just 500 books on it, whereas my iPad has more like ten times that number. Is it possible that your new plugin is somehow capacity limited in a way that the old plugin under calibre 1.48 was not? I know that one *possible* solution may be to blow away Marvin XD on my iPad and try to rebuild its library, but given how risky and time-consuming that project will be--I'm going to be very unhappy if I do it and then discover that I can't rebuild the collection and can't go back to calibre 1.48 either--I'd really like to feel confident that that will address the problem.

I hope all this information is helpful to you. Thanks again.

P.S.: For what it's worth, I just deinstalled calibre 2.1.0, reinstalled 1.48, backloaded the old plug-ins, and the iPad is working fine once again.

[davidfor]

WJCronon: The changes I made shouldn't have affect this. But, it is possible that something else in the underlying framework has in the change to Qt5 is causing a problem.

I have to admit I haven't looked much at this section of the code. My suggestion so far is based on where the errors are occurring and a quick read of the code around it. The idea that it is a size problem could be it. But, it could be a database on the device. The code that is having trouble is about building a book list database. I'm not completely sure if it has copied a database from the device or is updating a database on the device. The full debug log I mentioned before would help.

Edit: I should have also said, that with one device working and one not, that points to a problem on the device. Whether its the Marvin database or something the plugin adds, I don't know.

Also, there is an option in the configuration called "Enable booklist caching on device". Do you have it turn on? If so, can you try turning it off.

[lamp]

Hi - When I posted the first message, I actually already followed your un-installation instruction, i.e. remove both the json file and resources folder of your plugins. I now just did it again removing the plugin. This time, I also removed Marvin plugin, its XD resources and json file as well.

I should point out when I have your plugin, booklist caching was not on, and I am running on Mac OS X 10.9.4.

After reinstalling IOS Reader app plugin, Marvin Plugin, starting Marvin on my iPad and clicking Calibre connector, I again got this error message:

calibre, version 2.1.0
ERROR: Error: Error communicating with device

database disk image is malformed

Traceback (most recent call last):
File "site-packages/calibre/gui2/device.py", line 86, in run
File "site-packages/calibre/gui2/device.py", line 501, in _books
File "/var/folders/x7/94fnxx_91nl3m6wtrll3_qnr0000gn/T/calibre_2.1.0_tmp_ZW2hrF/tmMVqx_ios_local_db/Marvin_overlays.py", line 301, in books
DatabaseError: database disk image is malformed

Then, I enable debug mode and then I got ERROR (I've attached the two debug files): Unhandled exception, AttributeError:'IOSReaderApp' object has no attribute 'cached_books'

calibre, version 2.1.0
ERROR: Unhandled exception: <b>AttributeError</b>:'iOSReaderApp' object has no attribute 'cached_books'

calibre 2.1 isfrozen: True is64bit: True
Darwin-13.3.0-x86_64-i386-64bit Darwin ('64bit', '')
('Darwin', '13.3.0', 'Darwin Kernel Version 13.3.0: Tue Jun 3 21:27:35 PDT 2014; root:xnu-2422.110.17~1/RELEASE_X86_64')
Python 2.7.8
OSX: ('10.9.4', ('', '', ''), 'x86_64')
Successfully initialized third party plugins: DeDRM && Reading List && Search The Internet && Quality Check && Hyphenate This! && Annotations && Marvin XD && iOS reader applications && Find Duplicates && Extract ISBN
Traceback (most recent call last):
File "calibre_plugins.marvin_manager.action", line 1267, in device_diagnostics
File "calibre_plugins.marvin_manager.action", line 944, in _add_device_book_count
AttributeError: 'iOSReaderApp' object has no attribute 'cached_books'

[AndrewG]

davidfor, after reading your post above I checked to see if the connection problem was restricted to my ipad, and found the same problem with trying to connect to Marvin on my iphone. The error report looks similar:calibre, version 2.1.0
ERROR: Error: Error communicating with device

database disk image is malformed

Traceback (most recent call last):
File "site-packages/calibre/gui2/device.py", line 86, in run
File "site-packages/calibre/gui2/device.py", line 501, in _books
File "/var/folders/g4/jj3y1bvs4h53f4tpsc7_d3x40000gn/T/calibre_2.1.0_tmp_CGRsHK/1Rtzo0_ios_local_db/Marvin_overlays.py", line 285, in books
File "/var/folders/g4/jj3y1bvs4h53f4tpsc7_d3x40000gn/T/calibre_2.1.0_tmp_CGRsHK/1Rtzo0_ios_local_db/Marvin_overlays.py", line 2352, in _restore_from_snapshot
File "/var/folders/g4/jj3y1bvs4h53f4tpsc7_d3x40000gn/T/calibre_2.1.0_tmp_CGRsHK/1Rtzo0_ios_local_db/Marvin_overlays.py", line 2328, in _validate_mainDb_profile
File "/var/folders/g4/jj3y1bvs4h53f4tpsc7_d3x40000gn/T/calibre_2.1.0_tmp_CGRsHK/1Rtzo0_ios_local_db/Marvin_overlays.py", line 1579, in _compare_mainDb_profiles
File "/var/folders/g4/jj3y1bvs4h53f4tpsc7_d3x40000gn/T/calibre_2.1.0_tmp_CGRsHK/1Rtzo0_ios_local_db/Marvin_overlays.py", line 2109, in _profile_db
DatabaseError: database disk image is malformed

[hokusbloke]

Here's the problem

calibre, version 2.1.0
ERROR: Unhandled exception: <b>ImportError</b>:No module named PyQt4

calibre 2.1 isfrozen: True is64bit: True
Darwin-13.3.0-x86_64-i386-64bit Darwin ('64bit', '')
('Darwin', '13.3.0', 'Darwin Kernel Version 13.3.0: Tue Jun 3 21:27:35 PDT 2014; root:xnu-2422.110.17~1/RELEASE_X86_64')
Python 2.7.8
OSX: ('10.9.4', ('', '', ''), 'x86_64')
Successfully initialized third party plugins: Marvin XD && iOS reader applications
Traceback (most recent call last):
File "site-packages/calibre/gui2/dialogs/plugin_updater.py", line 789, in _configure_clicked
File "site-packages/calibre/customize/__init__.py", line 152, in do_user_config
File "calibre_plugins.ios_reader_apps.__init__", line 493, in config_widget
File "site-packages/calibre/customize/zipplugin.py", line 179, in load_module
File "calibre_plugins.ios_reader_apps.config", line 26, in <module>
File "/Users/hokusbloke/Library/Preferences/calibre/plugins/iOS_reader_applications_resources/widgets/main_ui.py", line 10, in <module>
from PyQt4 import QtCore, QtGui
ImportError: No module named PyQt4

[hokusbloke]

I deleted both plugins and tried again...still no joy.
I even deleted them and rebooted and then reinstalled them...still does not work.

[hokusbloke]

Still doesn't work

[hokusbloke]

Solved this issue with the plugins...finallly....and now I'm getting this message [Mac OSX 10.9.4]

calibre, version 2.1.0
ERROR: Error: Error communicating with device

database disk image is malformed

Traceback (most recent call last):
File "site-packages/calibre/gui2/device.py", line 86, in run
File "site-packages/calibre/gui2/device.py", line 501, in _books
File "/var/folders/yk/mh7hw3_n4wvd_bp5pr8br0nc0000gn/T/calibre_2.1.0_tmp_T58jfa/TRzcUK_ios_local_db/Marvin_overlays.py", line 285, in books
File "/var/folders/yk/mh7hw3_n4wvd_bp5pr8br0nc0000gn/T/calibre_2.1.0_tmp_T58jfa/TRzcUK_ios_local_db/Marvin_overlays.py", line 2352, in _restore_from_snapshot
File "/var/folders/yk/mh7hw3_n4wvd_bp5pr8br0nc0000gn/T/calibre_2.1.0_tmp_T58jfa/TRzcUK_ios_local_db/Marvin_overlays.py", line 2328, in _validate_mainDb_profile
File "/var/folders/yk/mh7hw3_n4wvd_bp5pr8br0nc0000gn/T/calibre_2.1.0_tmp_T58jfa/TRzcUK_ios_local_db/Marvin_overlays.py", line 1579, in _compare_mainDb_profiles
File "/var/folders/yk/mh7hw3_n4wvd_bp5pr8br0nc0000gn/T/calibre_2.1.0_tmp_T58jfa/TRzcUK_ios_local_db/Marvin_overlays.py", line 2108, in _profile_db
DatabaseError: database disk image is malformed

[Philantrop]

First of all, I'm sorry for your troubles. Unfortunately, I can't reproduce either of them at all on either Linux or Mac OS X.

So, for the "database disk image is malformed" issue. The debug logs were very useful:

DEBUG:    4.3  src: u'/Library/mainDb.sqlite'
DEBUG:    4.3  dst: /var/folders/x7/94fnxx_91nl3m6wtrll3_qnr0000gn/T/calibre_2.1.0_tmp_arhgnW/T_eZpI_ios_local_db/mainDb.sqlite

The plugin copies Marvin's database to a local directory and then tries to read from it which fails:

File "/var/folders/x7/94fnxx_91nl3m6wtrll3_qnr0000gn/T/calibre_2.1.0_tmp_arhgnW/T_eZpI_ios_local_db/Marvin_overlays.py", line 301, in books

DatabaseError: database disk image is malformed

So this *might* be an issue with the database on the device since the copying itself shows no issues.

The other option would be that the copied file ("/var/folders/x7/94fnxx_91nl3m6wtrll3_qnr0000gn/T/calibre_2.1.0_tmp_arhgnW/T_eZpI_ios_local_db/mainDb.sqlite") somehow gets corrupted.

Thus, it would be helpful if you, lamp, could email me (wk@mailstation.de) the file "mainDb.sqlite" so that I can check it for corruption.

Some adventurous soul might want to try re-installing Marvin itself, populating Marvin's library only with the provided ebooks and trying again to use the plugin.

Again, sorry for your troubles but the code should work fine (and it does for me and it wasn't changed recently, nor should that part be in any way impacted by the switch to Qt5/PyQt5) and without a way to reproduce the issue, it's hard to see what's going on.


As for the PyQt4 error, exit Calibre, delete the directory "plugins/iOS_reader_applications_resources/widgets" and start Calibre again. This looks very much like an issue caused by switching between Calibre 1.x and 2.x.


Finally, you don't have to dig into your backups to get *any* version of the plugin - you can download them all from https://github.com/Philantrop/calibr...tions/releases

[lamp]

Hi Wulf -- I've sent you a Dropbox link of my file to your email address, as the file /var/folders/x7/94fnxx_91nl3m6wtrll3_qnr0000gn/T/calibre_2.1.0_tmp_arhgnW/T_eZpI_ios_local_db/mainDb.sqlite is bigger than 200MB.

Thanks for your help

[milmom2005]

Thanks to davidfor and the other smart folks who make these plugins for Marvin and enhance the great work of Kovid with calibre. The new updates worked for me and I am now back in business using Marvin as my ereader.

[Philantrop]

Originally Posted by lamp

Hi Wulf -- I've sent you a Dropbox link of my file to your email address, as the file /var/folders/x7/94fnxx_91nl3m6wtrll3_qnr0000gn/T/calibre_2.1.0_tmp_arhgnW/T_eZpI_ios_local_db/mainDb.sqlite is bigger than 200MB.
Thanks for your help

Thanks, Peter, got it and I could indeed verify that the database file is definitely corrupted.

What I'd like to check now is if it's corrupt on-device as well or if something happened to it on the way to your computer.

Could you please follow http://blog.marvinapp.com/post/62814894703 to get the mainDb.sqlite from Marvin itself and provide it via Dropbox again?

[lamp]

Hi Wulf — Thanks for your tremendous help, here is the link of the SQLite file I’ve extracted from Marvin App on my iPad (using iFunBox) as you mentioned from Marvin blog:
https://www.dropbox.com/s/5c3x8ov9i3...Db.sqlite?dl=0

[davidfor]

Wulf: As a test to see what is happening, you could add an database integrity check to the code. Running either "PRAGMA integrity_check" or "PRAGMA quick_check" and writing the output to the log will give you an idea of what is happening. It isn't something you would want to do all the time, but it might help debug things.

And as the plugin is copying the database from the device, it might be that Marvin still has it open. In that case, a file copy of the database file is likely to produce be corrupt database. I managed to do that with my Kobo Utilities plugin.

[ChristopherTD]

FWIW, I also get this corrupted database error. If the provided files don't supply enough information to determine the cause I can try to extract mine as well.
Thanks again for the extra work involved in sorting this out.