Fixed (beta) libraries for iOS 8 + Windows + Marvin

[chaley]

Originally Posted by ShellShock

The _localize_database_path function in the IOS reader plugin copies the mainDB.sqlite from the idevice to the above temporary file on your PC. I suspect that this copy operation is buggy, e.g., it could be reading the file whilst it is still open in Marvin on the device, or it could still be writing the file to your Windows disk when the plugin tries to query it as a SQLLite database. The more books you have, the bigger the database and the more critical timings become.

Based on my experience on Windows and Android with SQLite, if you are actually copying the file from the iDevice's "disc" then there are several things that could go wrong if Marvin is running and the DB is open. First would be the possibility of an auto-reorganize, where the DB layer decides to compact the DB or update indices. Second could be the use of in-memory journals, meaning the db file doesn't contain the real db from time to time. SQLite can use write-behind updates. Usually SQLite supports read-only or read-write, and almost no app will switch between these because opening the DB is too expensive.

If at all possible you should get Marvin to make the copy for you, then upload that copy. That way Marvin could close the DB (which would apply all updates etc), copy the file, then re-open it. I don't know if you want to write the DB back to Marvin, but if you do then you should do the same thing in reverse: copy the file back to the iDevice then ask Marvin to replace its DB with the copy.

[ShellShock]

Thanks chaley, that is very helpful. I have 3 hypothesis:

1. Contention for the remote database file on the device; based on your experience this is now top of my list. I have previously tried allowing for this using two different methods of requesting an exclusive lock on the remote file via the interface library (libimobiledevice). However, I have no idea if this locking request is obeyed on the device; I suspect not because in testing the request always succeeds, whereas I would expect it to sometimes fail (or block) when the device is under load and writing to the file. As you suggest, we probably need Marvin to make a copy of the database, or maybe expose a mutex that both ends can use to synchronise access to the fie.
2. Contention for the local database file on the PC. Although there are several Calibre threads which appear in the logs, I can't see any real evidence that they are in contention for the file.
3. A bug in the libimobiledevice code that downloads the remote database file. No evidence so far.

I am going to pursue 1 for now.

[chaley]

@ShellShock: you might be able to "prove" #1 if you ask the testers to quit Marvin (really quit, not moving it to a background service or something like that) before connecting. I don't know if this is possible given how the plugins work, but if it is then you could remove Marvin having the DB open as one possible culprit.

[ShellShock]

Actually, I think I have just proven #3:

1. I used iMazing to copy a known good Marvin database from the device to my PC.
2. Then used SQLLite expert to add 10000 records to the Books table in the PC copy of the database.
3. Then used IMazing to copy the bulk database back to the device with a different file name. This database will not be touched by Marvin, as it does not have the normal Marvin database file name.
4. Then used IMazing to copy the bulk database from the device back to the PC.
5. Used WinMerge to compare 2 to 4. Files are identical. This proves I can rely on IMazing to copy the binary file.
6. I hacked the IOS reader plugin to download my bulk database as well as the normal Marvin database.
7. Compared 4 to 6 - files are different.
8. Opened 6 in Sql Lite expert and tried to execute select on Books table - database disk image is malformed error!

Conclusion: there is a bug in the download of the database file so that it is not producing an accurate copy. I suspect the libimobiledevice library which I am supporting. Now I can repro the bug at will, I have a much better chance of fixing it.

Edit: I can see from the log that libimobiledevice is only reading the first 4194304 bytes, which is 400000 in hex. What a nice round number that is! The rest of the file (up to the original size of the remote file) is null bytes. Ouch.

Edit2: I think I know exactly where the bug is now, and have a pretty good idea how to fix it. But this will have to wait until tomorrow...

[ChristopherTD]

@Shellshock thanks for all your work on these issues. It is much appreciated!

[stanhere]

Hey ShellShock!

Viewing your latest "Investigative Chapter" [#109] of "The Case of The 'Counterfeit' Mal-Formed Database'' reads like a good detective novel.

Really appears you're on the right criminal's trail!!

Thanks Again for your continued sleuthing!!!

[MSWallack]

Let me echo the others who are thanking ShellShock for the work the fix this problem.

[ShellShock]

Please note that the "database disk image is malformed" error should be fixed in tomorrow's Calibre release. The fix applies to all platforms Calibre supports. The bug occurred if your Marvin database on your IPad/IPhone exceeded 0x400000 bytes in size, e.g., when sending 1000s of books to the device. Thanks to AndrewG for testing the fix.

For Windows users, tomorrow's Calibre release does not include new libimobiledevice dlls with IOS 8 support (more time is required to integrate the Windows libimobiledevice library into the Calibre build). Please continue using the beta dlls in the first post of this thread. If you install tomorrow's Calibre release, and are already using the beta dlls, you will of course need to copy the beta dlls back into your Calibre installation, as explained in the first post.

[Zebedi]

Thank you ShellShock for all the time spent solving this problem, it is very much appreciated.

[pwalker8]

Originally Posted by ShellShock

Please note that the "database disk image is malformed" error should be fixed in tomorrow's Calibre release. The fix applies to all platforms Calibre supports. The bug occurred if your Marvin database on your IPad/IPhone exceeded 0x400000 bytes in size, e.g., when sending 1000s of books to the device. Thanks to AndrewG for testing the fix.

For Windows users, tomorrow's Calibre release does not include new libimobiledevice dlls with IOS 8 support (more time is required to integrate the Windows libimobiledevice library into the Calibre build). Please continue using the beta dlls in the first post of this thread. If you install tomorrow's Calibre release, and are already using the beta dlls, you will of course need to copy the beta dlls back into your Calibre installation, as explained in the first post.

Most excellent! Thanks for your hard work!

[Dhardrian]

I can confirm that the new Calibre 2.10.0 64 bit with ShellShock's 64 bit DLL's are working perfectly with my iPad 3 device with a Marvin library of over 11.3K books.

Many thanks to Shellshock for the effort that went into finding this bug!

[pwalker8]

Just downloaded 2.10.0 on an iMac (Yosemite) and connected my iPad mini and ... (insert favored mad scientist cackle here) it's working, it's working! (insert favored version of Handel's Hallelujah Chorus here). Many, many thanks ShellShock! I predict that your karma value will soon put you in nirvana.

[Beamis]

Marvin again working seamlessly with Calibre under Windows 8.1

Many thanks to ShellShock and Kovid for working through this!

[MSWallack]

Sadly, though, those of us stuck using Calibre 1.48 (because of the need to remain on Windows XP) are apparently stuck. Sigh.

[ShellShock]

Originally Posted by MSWallack

Sadly, though, those of us stuck using Calibre 1.48 (because of the need to remain on Windows XP) are apparently stuck. Sigh.

....thinks....is it the "database disk image is malformed" error that is causing a problem on XP?