Fixed (beta) libraries for iOS 8 + Windows + Marvin

[rjh4242]

I agree that the malformed database problem is likely to be related to the number of books in the Marvin library. I am running calibre 2.8 64-bit on a Windows 8.1 computer, connecting to an iPhone 4s running ios8.1 and an iPad 3rd gen running ios 8.1.

With the new DLLs, the iPhone is fine, but I only keep about 200 books on it, and transfer only a few (say up to 5) at a time.

I had over 3000 books on my iPad, and when calibre was trying to detect it I immediately got the "database malformed" error. I uninstalled Marvin from my iPad and reinstalled it. No problem with calibre connecting when the library was empty. Transferring over 3000 books resulted in the job uploading the books saying it had finished successfully but the subsequent (related) job to update the metadata failing with the "database malformed" error message.

I uninstalled and reinstalled Marvin on the iPad again. Transferring 600 books worked fine, no errors, calibre could connect to the iPad fine. Transferring another 600 books worked fine. I carried on transferring 600 books at a time until I was on the last 650. Calibre told me the upload job finished successfully, but again the subsequent metadata update failed with the "database malformed" error. All the books are on the iPad, but I now get the "database malformed" error when I try to connect calibre to the device.

I uninstalled and reinstalled Marvin again on the iPad, and tried transferring 2500 books at once. That worked fine and calibre had no trouble connecting to the iPad after a disconnect/reconnect/restart calibre.

I then added books 100 at a time. All was fine until I was on the 3rd lot of 100, when again the metadata job after the upload errored out with the "database malformed" error.

So I'm thinking that somewhere above 2700 books in Marvin causes a problem.

[Zebedi]

I only got to 1218 when the malformed database error appeared, and in fact the message appeared while transferring the last 29, though they did transfer, but they were the last.

[Josieb1]

Originally Posted by ShellShock

One of two things need to happen first: either we confirm the "malformed database" error is not caused by the new dlls, or we confirm it is (and the old ones are OK) and I managed to fix it.

Thank you. Will you post something somewhere when it's all official so I know what to upload, either a new calibre version or an update to the plug in. Or do I just keep an eye on the updates within Calibre?

[ShellShock]

Best bet is to keep an idea on this thread. I have not thought yet about how to get any fixed dlls into Calibre.

[PeterT]

Not to be stupid, but how do the original DLLs get there? If they are installed during the calibre installation, then the obvious solution is for kovid to include them in the next release.

[ShellShock]

Yes, but Kovid may want to build them from source (https://github.com/Polyfun/libimobiledevice-windows)--I assume this is how the current dlls get into the Calibre build. Anyway, I would like to try and get the malformed database error fixed first, if possible.

[kovidgoyal]

I will definitely want to build them from source. Everything in the calibre windows build is built from source (except for the Microsoft C runtimes).

But I will wait to do so until ShellShock tells me he is ready.

[MSWallack]

Given the number of books I have in Marvin, I'll hold off trying to test things...

[stanhere]

For Zebedi, rjh4242 & other W7 64 Calibre + Marvin + iPad users:

After importing ShellShock's "magic" 'DLL trio' to my 64bit Calibre v2.7 & again to v2.8 .....successfully!! ....., I also suddenly received the "Malformed database disk image" exception \ complaint.

After futzing around, I finally "cleared" out my Marvin iPad book database by simply uninstalling, shutting down & re-installing Marvin v2.8 on my iPad, updated to OS 8.1.

After re-connecting the iPad with an "empty Marvin" to W7 \ Calibre ....finally voila!!!!!

Reloaded Marvin's database by transferring batches of books from my Calibre library, 150 to no more than 250 at a time' until I successfully topped off at 2,214. Plus in the last two days, I've incrementally added more to a present total of 2,248.

So far Marvin's iPad DB remains UN-malformed AND All software & All Devices continue to Play Well Together!

Again, Lots of Appreciation due from W7 based users to ShellShock

[ShellShock]

I would be grateful if someone who gets the malformed database error could be a guinea pig and generate a debug log for me when the error occurs:

Customise the IOS reader and Marvin plugins to enable debug logging (two check boxes each).
Right click the Perferences toolbar button in Calibre and choose Restart in debug mode.
Reproduce the problem, starting with a good database and send a bunch of books until the problem occurs.

Warning - the debug log may get very big, and we may need several goes at this (e.g., I may need to add more debug logging to the code).

Thanks

[Terisa de morgan]

I've the log but I don't think it's useful because the error is there, at the first try... and I only have 289 books.

[Terisa de morgan]

BTW, I've kept the database that has been copied to the local path... and it's totally corrupt (SqlLite Manager has problem reading some data).

[AndrewG]

I uninstalled Marvin from my iPhone (IOS8) then reinstalled. Customised Calibre plugins for debug, started Calibre in debug mode then gave it the task of transferring my 500 largest books to the iPhone. After 28 minutes it was reporting staging 400 of 500 books. No change after another 20 minutes so I disconnected the iPhone and closed Calibre. The log is attached as tmpxbjlbt.txt.zip
I then restarted Calibre in debug mode and discovered that 400 books had been successfully transferred and gave it the task of transferring the next 1100 books. 2 books were reported as not transferrable because of being in the incorrect format and then Calibre began to transfer the remaining 1098. After 16 minutes and 5 seconds Marvin gave a timeout error message, log attached as tmppe3ier.txt.zip.
I restarted Marvin in debug mode to find a further 1000 books had been transferred. Calibre would not transfer book 1401, it claimed no suitable formats found but reported the presence of an ePub. Calibre was next tasked with transferring the next 1599 books. No problem, so then the next 2000...and this time we have an error sending the metadata and the malformed database message:

calibre, version 2.8.0
ERROR: Error: Error communicating with device

database disk image is malformed

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 87, in run
File "site-packages\calibre\gui2\device.py", line 532, in _sync_booklists
File "C:\Users\Andrew\AppData\Local\Temp\calibre_vzn1dh \c5a2ct_ios_local_db\Marvin_overlays.py", line 1009, in sync_booklists
File "C:\Users\Andrew\AppData\Local\Temp\calibre_vzn1dh \c5a2ct_ios_local_db\Marvin_overlays.py", line 2109, in _profile_db
DatabaseError: database disk image is malformed

The debug log is attached as tempdr7hyn.txt.zip

[AndrewG]

Another run of Calibre in debug mode, after the malformed database message failed to connect to the iPhone and yielded the attached log.

[stanhere]

Given the many reports of MalFormed Marvin DB's here & other forums, I offer the following -----
For Windows 7 32/64 Calibre users out there wanting to BackUp your iPad's Marvin "Book Records" composed of two key parts: A= "mainDb.sqlite" , a single small file {4.2mb in my case for 2,246 books} and B= a matching set of individual book files {2,246 books (collection) in my case totaling 4.67 gigs} ........

There are a number of freeware (e.g. iFunBox) & shareware Windows progs available to access & transfer Files & Data, TO & FROM one's iPad vs LapTop \ DeskTop, quite quickly. **1

Besides routinely Backing Up my "Window's Calibre Book Library" every time I alter it, I am now also backing up my iPad's Marvin library with DiskAid sw so that within minutes, I can restore my iPad \ Marvin DB if & when I get another "communication error; database is malformed" complaint from Calibre upon iPad connection!

In my case, using DiskAid on two Marvin App paths, a BackUp to my PC takes under 5 minutes: Copy "mainDb.sqlite" database file from 'Apps\Marvin\Library' and the complementary books "collection" from 'Apps\Marvin\Library\Documents' (folder contents). **1

If & when a restoration is required, simply reverse [overwrite] per the prior paragraph. **1

**1 - Marvin must be closed!

Lastly, Use google for clarification & further detail and another Thank You to ShellShock!