Android 4.4 (KitKat) breaks CC writing to removable SD cards

[chaley]

Originally Posted by kaufman

I then tried copying my whole folder from within calibre. It failed out after about 150 books with a disconnected message. I tried opening the files that had copied, and everything worked fine.

So as before, calibre complained that the device went away, but the device was still doing stuff? Do you have an estimate for how long it took to send those 150 books?

Could you send me a CC debug log? There might be something in there that helps understand what is happening. Send it to cc_debug_logs at charles dot haleys dot org.

If I do a couple of 100 files at a time, it works ok, but anything more then that fails after a while.

It should be noted that I have been able to copy my entire folder of several thousand books all at once under 4.3, so perhaps 4.4 is slowing things down.

Hmmm...

I hadn't thought that there might be a 4.4 change that affects this. If so, it is going to be hard to find.

Did the tablet do anything while the copy was happening, for example shutting off the screen? I have some memory that 4.4 does more aggressive power management than 4.3. I wonder if it might be choosing to put CC to "sleep" under some conditions, which would stop it from talking to calibre. More research needed.

Just read some bug reports that say that KitKat can spontaneously decide to switch from WiFi to mobile networking then switch back. An HTC bug report says that connections can be unstable if the access point name (SSID) contains spaces or special characters. (!) If these are true then it explains CC's problems.

By any chance did you try sending the books to main memory? I wonder if the problem is KitKat or the SD card.

Is increasing the timeout under Calibre a developer change or can I make this change in settings?

No, these timeouts are hard coded into calibre and CC.

[kaufman]

Originally Posted by chaley

So as before, calibre complained that the device went away, but the device was still doing stuff? Do you have an estimate for how long it took to send those 150 books?

Sometimes it complains that the device went away, sometimes it just stopped transfering and just canceled the job on the calibre side. The cc side is still waiting for more books. It takes about five minutes to get into this state.

Originally Posted by chaley

Could you send me a CC debug log? There might be something in there that helps understand what is happening. Send it to cc_debug_logs at charles dot haleys dot org.

I just sent you two logs. One from the last SD card transfer and one from after I tried to do it to internal memory.

Originally Posted by chaley

Did the tablet do anything while the copy was happening, for example shutting off the screen? I have some memory that 4.4 does more aggressive power management than 4.3. I wonder if it might be choosing to put CC to "sleep" under some conditions, which would stop it from talking to calibre. More research needed.

Just read some bug reports that say that KitKat can spontaneously decide to switch from WiFi to mobile networking then switch back. An HTC bug report says that connections can be unstable if the access point name (SSID) contains spaces or special characters. (!) If these are true then it explains CC's problems.

No screen turn off. No switch from Wifi because its a tablet without a cellular modem.

Originally Posted by chaley

By any chance did you try sending the books to main memory? I wonder if the problem is KitKat or the SD card.

I hope you appreciate this, because I spent an hour loading them onto the sd card.

Just deleted everything out of the library and cleared CC and tried it with internal memory and it failed after 80 books.

It doesn't look like its an SD issue.

[chaley]

Originally Posted by kaufman

I just sent you two logs. One from the last SD card transfer and one from after I tried to do it to internal memory.

Thanks.

I hope you appreciate this, because I spent an hour loading them onto the sd card.

In fact, I really do. It is close to impossible to debug this kind of problem without data from users.

Just deleted everything out of the library and cleared CC and tried it with internal memory and it failed after 80 books.

It doesn't look like its an SD issue.

Well {expletive deleted}. The only thing that changed was KitKat. None of us needs a buggy operating system.

[chaley]

@kaufman: those debug logs are very interesting. It appears that kitkat has changed some fundamental things about how tasks (threads) are scheduled. Because of that, CC is behaving differently internally, which might be exposing some problems that never came up before. Thanks again!

[kaufman]

Originally Posted by chaley

@kaufman: those debug logs are very interesting. It appears that kitkat has changed some fundamental things about how tasks (threads) are scheduled. Because of that, CC is behaving differently internally, which might be exposing some problems that never came up before. Thanks again!

Glad I could help. Your app gives me so much pleasure every day. Its very nice to give something back.

[chaley]

Originally Posted by kaufman

Just deleted everything out of the library and cleared CC and tried it with internal memory and it failed after 80 books.

It doesn't look like its an SD issue.

My original version N7 was upgraded to kitkat. It doesn't have an SD card so I can't test that. However, given that it is failing for you working with main memory, that I can test.

Three times I sent 1400 books, once with an empty database and twice overwriting. I could not make it fail. I conclude that there is something "special" either with your network setup or with LG's version of KitKat.

First, some questions:

• Calibre is running on what sort of machine? (OS & version). Mine is Win7 Pro.
• How is the machine running calibre connected to the network? Mine is by cable. If by wifi, what band & protocol is it using?
• What sort of network setup do you have? Mine uses a central fiber optic router connected to a gigabit switch. All devices go through the switch. My N7 is connected to a Netgear DG834G running in access point mode.
• What wifi band/protocol is your LG using?

What I am looking for is something that could explain what I think we are seeing; the traffic is being flow controlled somewhere, causing the connection to behave erratically.

Second, a request: given that you can make it fail reliably, could you do it while making a calibre (not CC) debug log? You shouldn't need to clean out the library again. Sending the books again should be sufficient. My goal is to see which error calibre's networking reports, which might give a clue about where the problem originates.

[kaufman]

Originally Posted by chaley

First, some questions:

• Calibre is running on what sort of machine? (OS & version). Mine is Win7 Pro.
• How is the machine running calibre connected to the network? Mine is by cable. If by wifi, what band & protocol is it using?
• What sort of network setup do you have? Mine uses a central fiber optic router connected to a gigabit switch. All devices go through the switch. My N7 is connected to a Netgear DG834G running in access point mode.
• What wifi band/protocol is your LG using?

• iMac 2013 running 10.9.2
• Wired Connection to Airport Extreme
• Airport extreme with one wired connection to iMac and wired connection to cable model. All other connections wireless.
• Probably 802.11 N

Originally Posted by chaley

Second, a request: given that you can make it fail reliably, could you do it while making a calibre (not CC) debug log? You shouldn't need to clean out the library again. Sending the books again should be sufficient. My goal is to see which error calibre's networking reports, which might give a clue about where the problem originates.

Good news (sort of). As a test, I just tried to transfer a bunch of books to my Galaxy Note 3. Same failure. So it appears that it is not a 4.4 issue after all. I have transferred thousands of files in the past without seeing this issue.

I just sent you the CC debug log.

I have to leave my house now, but I will send you the Calibre log later.

[kaufman]

In case anyone was following this, Chaley sent me a few test versions, and I did a few reboots, and now everything seems to be working correctly in 4.4 with data on the external SD card.

You have to put the library in the Android/.../files folder, but Moon+Reader picks them up fine from there.

[kaufman]

One last thing about this. My phone (which I use a lot more then my tablet) updated to KitKat yesterday. I had previously put my book files in the SD Card /Android/... directory.

I spent a while yesterday banging on the app. My results are as follows:

1) The first time I connected, it did that thing where it transfers all my metadata. It seemed like it took a longer then usual. I hadn't made any global changes since the previous connect, so I am not sure what triggered the update. Subsequent connects did not try to do this again.

2) Transferring files to and from the phone all worked perfectly.

3) Sending books to Moon+Reader to read worked perfectly. I installed Cool Reader and Aldiko reader to test and in my limited testing, both worked perfectly as well. Aldiko copies the books to a local directory when it opens them and it had no problem doing so.

4) Everything else worked perfectly.

I think its safe to say that CC supports KitKat with the caveat that if you want to put your books on the SD card, you have to use the application specific directory.

[chaley]

Originally Posted by kaufman

I think its safe to say that CC supports KitKat with the caveat that if you want to put your books on the SD card, you have to use the application specific directory.

Thanks for doing these tests. They gave me confidence that I could add kitkat private folder support to CC.

[kaufman]

Originally Posted by chaley

Thanks for doing these tests. They gave me confidence that I could add kitkat private folder support to CC.

My pleasure. Glad I could give a little back. I really looking forward to testing the auto-sort functionality.

[chaley]

CC V3.3.5, released 22/May, supports using the KitKat private folder on the SD card.

[dfrye]

Just a followup to V3.3.5 (Thanks chaley!!) for anyone with existing books on the sdcard. To start the conversion process you need to change the default dir for the books (app menu> Settings > "Book Formats and Folders") Note where the files are currently stored. You will need this info to manually clean up later. Tap on the default folder and get a dialog with a short summary of this thread. Click on the suggested folder name (not the Choose Folder button). The program will copy the existing books to the new folder. Make sure there is enough room on the sdcard for the copied books. It may take a while to copy. Once it is done you will get a error message for each of the books that it was not able to delete the old files. That's OK. At this point Calibre Companion will work normally. The old copy of the books are still on the sdcard. To free up the space you will need to delete the old books using other means. Either use a file manager that knows how to get past KitKat to the delete files on the sdcard or pull the sdcard and plug into a PC.

[chaley]

@dfrye: thanks for that. Do you mind if I put your message in the FAQ? I would normally "quote" you, but I can not do that if you object.

[dfrye]

That's fine with me. Glad I could help.