Calibre Companion syncing experience: issues with a large ondevice library

[chaley]

Originally Posted by Sefiriot

Do you get logs submitted through normal Android error reports when the app crashes?

It depends. If the user allows the device to send a report then we get it. It also seems to be the case that we get them sometimes "just in case".

The CC crash that wiped the database, or seemed to at least had some rather interesting error info when I looked at it on the phone, which I didn't think to screenshot at the time. I'm not too sure now, but I believe that error message indicating empty metadata happened when CC was sending information for that book where calibre had previously timed out whilst in mid-transfer during session 1-- and the crash promptly wiped the database.

Wiping the database is not good (what a fatuous statement that is. ). It is very hard to explain why this could happen. Perhaps Kovid's pointer is very relevant, because records get re-created every time metadata comes from calibre to the device.

If CC's database is corrupt, is there anyway to force a rebuild? Would uninstalling, reinstalling and then letting CC match against calibre for books already on device do the trick?

The easiest way is to go to settings -> apps and "clear memory". That blows away the DB and all the settings but leaves the books. You would then redo the important settings, such as "use UUID" (which you seem to be using), ensure "scan" is on, and then connect. CC will extract metadata from epubs and send that up to calibre. Almost certainly the metadata will contain the UUID, greatly facilitating book matching. For other formats, CC sends the books themselves up to calibre for analysis.

As part of my testing I frequently nuke the DB and rescan. But then, my test machine has only 600 books on it...

[DoctorOhh]

I pop off to do some work finally get around to submitting my post and there have been a slew of posts in the last hour and my post seems out of place.

[chaley]

Originally Posted by DoctorOhh

I pop off to do some work finally get around to submitting my post and there have been a slew of posts in the last hour and my post seems out of place.

Not the least bit out of place. During this discussion, knowing that it works for someone is very valuable.

[Sefiriot]

Indeed. I'm interested to see that you can send books in lots of 1000 at a time-- CC invariably times out before book 500 or so, so I've not tried sending in large numbers.

I think that my WiFi connection and possibly the router is an issue, now that DoctorOhh mentions it: signal strength on my laptop is not a problem where it's located in my bedroom, but strength on my cellphone is another matter-- it's usually only at 50-75% and at certain areas in my room drops to a bar or less if I move around holding on to it. Might be time to check out signal boosting tricks on Lifehacker. The best place of course is close to the router, but that's in my stuffy box of a storeroom, and the kitchen has limited space for my laptop and my dad's at the same time, not to mention not enough power outlets. I'd prefer to use the LAN connection and eliminate WiFi interference on/from my laptop while leaving the phone downstairs to sync, but my router is one of those where wired and wireless connections seem to be on different subnets, so both have to be on WiFi. As it is, even when my laptop is upstairs with a good signal and my phone is downstairs close to the router, timeouts and errors do still occur.

[DoctorOhh]

Originally Posted by Sefiriot

Indeed. I'm interested to see that you can send books in lots of 1000 at a time-- CC invariably times out before book 500 or so, so I've not tried sending in large numbers.

The first time I experimented 500, 1000, 1500, 2000 and came to the conclusion that the overall time per book got longer the larger the batch. The next time loaded all of my books I queued 10 - 1000 book jobs and let it go.

[theducks]

Originally Posted by chaley

As DoctorOhh said, we thought we had already done this.

I just tried an experiment on my Galaxy Tab 10.1. I first set my screen timeout to 15 seconds. I connected, then set the device aside to see if it would ever sleep. Half-hour later, the screen was still on, no sleep. I then "ejected" the device from calibre. A few seconds later the screen went dark then the tab went to sleep.

Does your 10.2 do something different?

I have a 7" TAB2 (CC 3.0.5, which I have not recently tried to send the Library to) for ? less than a year and CC a little less than that.

Screen timeout is set for 2 min
I don't care if the screen times out as long as it does not sleep the network (and drop the connection )

VLC continues to play MP3's with a locked screen. which is why I wondered why I could not send 4K+ books in a single go (if I had enough battery)?

I will try

[chaley]

Originally Posted by theducks

Screen timeout is set for 2 min
I don't care if the screen times out as long as it does not sleep the network (and drop the connection )

Does the screen actually shut off while you are connected? On my 10.1 and my N7 it does not. Also, the network does not shut down, as evidenced by calibre sending "hi there" packets to CC every 10 seconds and getting a response.

One easy test you could run is to start calibre in debug mode, connect your device to calibre then put it down for a while. When you come back calibre should still be connected and you should see "NOOP" messages arriving in the debug log every 10 seconds.

[theducks]

Originally Posted by chaley

Does the screen actually shut off while you are connected? On my 10.1 and my N7 it does not. Also, the network does not shut down, as evidenced by calibre sending "hi there" packets to CC every 10 seconds and getting a response.

One easy test you could run is to start calibre in debug mode, connect your device to calibre then put it down for a while. When you come back calibre should still be connected and you should see "NOOP" messages arriving in the debug log every 10 seconds.

Seems this is now a non-issue.
I just ran a set of tests: 500 and 1K, which ran perfectly.

Notes:
I was connected to the charger during this test
The WiFi has always been in the same room
The desktop (Calibre host) is wired, reserved IP, static port and always on (no automatic hibernate)

Grr I hate it when system behavior changes.
I really did try 2 or 3 different times before and just figured it was: WAD

[chaley]

@Sefiriot: by any chance on Aug 25 did you allow your device to report a CC crash?

We received a crash report saying that CC's database was closed (by what?) in the middle of receiving books from calibre. I am trying to figure out how that can happen. At the moment, the only thing that begins to make sense is that the CC user:

1) Connected to calibre
2) Started a long transfer (sent a lot of books)
3) Put CC into the background and started doing other things.

It is conceivable that under these conditions, Android would close CC to recover memory, and therefore close the database. Unfortunately, CC has a background task running that would still be downloading books, and Android doesn't close that task. Result: this task would crash because its database is now gone.

This is at best a theory, but it is the only one I have. Does the scenario apply to any of your crashes?

Thanks.

[Sefiriot]

Aug 25-- yes, that would likely have been me. The above scenario sounds like the background to both crashes that wiped the database, now you mention it.

Backgrounding CC however... there could be several points where it could've happened because of something I did: it's on a phone, and I had several messages coming in that night I responded to while CC was working. Responding only takes a matter of seconds though, and CC came back and was still responding after each time I skipped out to WhatsApp.

I think the big one where manure finally hit the oscillating ventilation device would have been after calibre told me CC got disconnected, and CC was still stuck at the file transfer screen. I remember trying to get out to the home screen in hopes of restarting CC by force and failing: it was one of those really nasty hangs where hitting reset doesn't seem to work too well... IIRC it crashed soon after with a request to error report.

[chaley]

Originally Posted by Sefiriot

Aug 25-- yes, that would likely have been me. The above scenario sounds like the background to both crashes that wiped the database, now you mention it.

Backgrounding CC however... there could be several points where it could've happened because of something I did: it's on a phone, and I had several messages coming in that night I responded to while CC was working. Responding only takes a matter of seconds though, and CC came back and was still responding after each time I skipped out to WhatsApp.

I think the big one where manure finally hit the oscillating ventilation device would have been after calibre told me CC got disconnected, and CC was still stuck at the file transfer screen. I remember trying to get out to the home screen in hopes of restarting CC by force and failing: it was one of those really nasty hangs where hitting reset doesn't seem to work too well... IIRC it crashed soon after with a request to error report.

It might not be obvious that the db was closed. When you come back to CC, if necessary Android will restart it. The restart could mask the fact that the db went away while CC was in the background.

I have made a change to verify that the db is still open every time the db is used. This won't slow down the normal case because Android keeps a cache of open databases and gives the cached value back. If the database had been closed, then it will take a bit longer, but this is better than (sometimes silent) death. This change will be in the next CC release, something that should come near the end of the week.

I have also changed some things in when books being sent by calibre are stored into the DB. The existing version starts storing each book when the book arrives, something that works fine if the DB is running at full performance. However, if the DB has slowed down, then this method can make it even slower by having multiple "books" try to access the DB at the same time. It now uses a queue so only one book accesses the db at a time.

[Sefiriot]

Originally Posted by chaley

I have made a change to verify that the db is still open every time the db is used. This won't slow down the normal case because Android keeps a cache of open databases and gives the cached value back. If the database had been closed, then it will take a bit longer, but this is better than (sometimes silent) death. This change will be in the next CC release, something that should come near the end of the week.

I have also changed some things in when books being sent by calibre are stored into the DB. The existing version starts storing each book when the book arrives, something that works fine if the DB is running at full performance. However, if the DB has slowed down, then this method can make it even slower by having multiple "books" try to access the DB at the same time. It now uses a queue so only one book accesses the db at a time.

Thanks chaley. I do seem to notice an improvement in CC functioning since the last DB wipeout happened-- maybe it's my imagination, but rebuilding the DB from scratch seems to have fixed some of the slowness in metadata transfer. The rest of it I guess is likely my phone starting to show its age and RAM limitations with everything I throw at it, though since I did an app cull the poor thing's had more space to breathe. Meanwhile I'll keep logging my calibre sessions; if anything else interesting happens I'll let you know. I wonder if a (semi-)daily log of calibre activity when CC is connected would be of interest to you-- sorta like a daily patient diary?

Looking forward to the new release with upcoming improvements; the on-device library count will hit 10k when I sync it in the morning and probably be at 11k by the next CC release, so I'll be happy to see what the new improvements do then.