Android 4.4 (KitKat) breaks CC writing to removable SD cards

[chaley]

Google in their infinite wisdom has decided that starting in kitkat (android 4.4), apps cannot write to removable SD cards except in extremely limited ways. This means users will not be able to use CC to put their library on SD cards. There is nothing we can do about this.

For the moment a user can get around the restriction by rooting the device and changing a system file. This is not something that we can recommend.

EDIT: for the latest information, see this thread starting at post 42 and our FAQ answer How do I use an SD card with Android 4.4 (Kitkat).

[theducks]

This smells of collusion with device manufacturers to sell more of the premium priced devices that have more on board storage.

Charles, is there a possibility of a Calibre-companion compatible save to disk plugin where we could insert the card into a PC card-writer for the transfer and then sneaker it to the device and have it still be CC usable ?

[chaley]

Originally Posted by theducks

Charles, is there a possibility of a Calibre-companion compatible save to disk plugin where we could insert the card into a PC card-writer for the transfer and then sneaker it to the device and have it still be CC usable ?

Something we have wanted to do since the beginning but never managed to get it to the top of the pile is import books saved by calibre's connect-to-folder. Everything we need to set up metadata is in the .metadata.calibre file, and of course the books are there.

The challenge (not a hard one but work) is to map the storage naming system used when saving the books to the one used by CC. There aren't a zillion choices:

1. Move the books so that the file name matches what CC would have used if it stored it.
2. Use the existing book file and accept that it isn't where CC expects, dealing with the side effects of that.

I prefer option 1 because the rest of CC wouldn't have to know about it, but it is more work. For example, we would probably need to update the .metadata.calibre file so that calibre could find the books the next time the card is mounted on the PC. If we use option 2 then moving books when folders are changed would be "interesting".

Of course, we can do this today for epubs. If you save-to-disk to the SD card then calibre's metadata will be stored in the book (assuming no DRM). CC would "scan for new books" and find them, and very importantly find the calibre UUID so calibre's book matching would be accurate.

[kaufman]

Originally Posted by theducks

This smells of collusion with device manufacturers to sell more of the premium priced devices that have more on board storage.

That seems highly unlikely (at least in the US) because the US carriers generally only sell the smallest version of each phone. Every once in a while they add a larger one six months into the model year, but they certainly aren't following a strategy of pushing people to larger phones.

[samy2]

Does:
https://developer.android.com/refere...TERNAL_STORAGE

Not work?

[chaley]

Originally Posted by samy2

Does:
https://developer.android.com/refere...TERNAL_STORAGE

Not work?

No. KitKat explicitly ignores that permission on removable storage. There is no way to get write permission without rooting the phone.

[kaufman]

Chaley,

Are you considering adding support for a Calibre Companion private directory? I never go to the directory and my book reader would still be able to read the books.

[chaley]

Originally Posted by kaufman

Chaley,

Are you considering adding support for a Calibre Companion private directory? I never go to the directory and my book reader would still be able to read the books.

We are thinking about it. There are two problems that we see today:

1) Android deletes everything in that folder if CC is uninstalled or has its data cleared. This is a highly unfriendly act that we think will bring the Hus to our gate.

2) It isn't clear that reader apps will work. They can open the file for reading (we think) but they will not be able to create any temporary files in that folder. We don't know if any do this.

A third problem is that non of us have a kitkat machine with a removable SD card, so we can't test anything. That problem can probably be worked around with the help of someone who has such a device by sending them privately an APK to sideload.

Bottom line: we aren't yet convinced that the benefits of supporting that special folder outweigh the costs.

[kaufman]

Originally Posted by chaley

1) Android deletes everything in that folder if CC is uninstalled or has its data cleared. This is a highly unfriendly act that we think will bring the Huns to our gate.

I agree that would be annoying, but at least no data would ever be lost, since presumably, its all still in Calibre. Perhaps the Huns would only bang on your gate and not bust it down completely.

Originally Posted by chaley

2) It isn't clear that reader apps will work. They can open the file for reading (we think) but they will not be able to create any temporary files in that folder. We don't know if any do this.

I can't say that none of them do this, but the big ones (Moon Reader, Cool Reader, Aldiko) that I have tried do not. They all create their files in a folder local to themselves.

Originally Posted by chaley

A third problem is that non of us have a kitkat machine with a removable SD card, so we can't test anything. That problem can probably be worked around with the help of someone who has such a device by sending them privately an APK to sideload.

Do the same rules apply to flash drives plugged into a OTG adapter? If so, you could test that way. In any case, I'm happy to volunteer as soon as Verizon gets around to updating the GN3.

[jjmcgaffey]

Originally Posted by chaley

Bottom line: we aren't yet convinced that the benefits of supporting that special folder outweigh the costs.

Well - for me, it definitely would be worth it. I carry a lot of books with me - in fact, I was about to upgrade from a 32Gb card to a 64Gb (which contains, basically, books and music). I can read my existing books just fine, but I can't add to them - it's very disconcerting. I'd be very happy if CC gets access to the SD card. I have the latest version (issued 4/1/14), which says it does something about this, but I still can't add a book to my (SD-located) library.

I use FBReader as my reader, and ran into problems with it - it writes to a "cache" folder, which by default is the same place as its library (which is also CC's library). However, it's simple to change the cache folder to be elsewhere (it pops up to complain, and you can change the folder on that window) - so now it's writing to cache in internal memory, but can still see the library on the SD card. It doesn't, apparently, actually need to write anything to the books themselves.

The clearing data is problematic - less for uninstalling than for getting the data cleared, that sounds very unfriendly. I do use clearing data to deal with various freeze-ups for apps, though I don't think I've ever needed it for CC. If I were going to uninstall CC, I expect I'd sync first, or not be too surprised if the books were gone too.

I do have a Note 3 that I've upgraded to KitKat, so I'd be happy to be a tester if you need one. I've rooted other devices, but haven't this one (yet); I'm pretty familiar with fiddling around (sideloading, etc).

For right now, I guess I'll have to ignore CC (wahh! just after you added Date Read, too!) and copy stuff to the folder more directly (hmmm, wonder if WebDAV will work, or if I'll have to go USB or pull the card out?), then get it directly through FBReader. I'm not _yet_ going to root just to fix this, but if I find it's broken other stuff as well, I may have to.

[chaley]

Originally Posted by jjmcgaffey

Well - for me, it definitely would be worth it. I carry a lot of books with me - in fact, I was about to upgrade from a 32Gb card to a 64Gb (which contains, basically, books and music). I can read my existing books just fine, but I can't add to them - it's very disconcerting. I'd be very happy if CC gets access to the SD card. I have the latest version (issued 4/1/14), which says it does something about this, but I still can't add a book to my (SD-located) library.

The changes were to give an error message when writing to the SD card failed and to refuse to allow changing CC's folder to a read-only SD card folder. Nothing was added to allow using the single app-specific folder.

I use FBReader as my reader, and ran into problems with it - it writes to a "cache" folder, which by default is the same place as its library (which is also CC's library). However, it's simple to change the cache folder to be elsewhere (it pops up to complain, and you can change the folder on that window) - so now it's writing to cache in internal memory, but can still see the library on the SD card. It doesn't, apparently, actually need to write anything to the books themselves.

Good to know that one can work around the problem.

The clearing data is problematic - less for uninstalling than for getting the data cleared, that sounds very unfriendly. I do use clearing data to deal with various freeze-ups for apps, though I don't think I've ever needed it for CC. If I were going to uninstall CC, I expect I'd sync first, or not be too surprised if the books were gone too.

Fully agree with you -- nuking the folder on clear is extremely unfriendly. Unfortunately it is Android itself that deleted the folder. We app writers have zero control over it.

I do have a Note 3 that I've upgraded to KitKat, so I'd be happy to be a tester if you need one. I've rooted other devices, but haven't this one (yet); I'm pretty familiar with fiddling around (sideloading, etc).

There is one possibility that we are not able to test. It might be possible to manually create the app-specific folder, then use CC's change library dialog to move CC's library to that folder. It might even be possible for CC to create that folder. Who knows?

CC's app-specific library folder on the SD card should be
Android/data/com.multipie.calibreandroid/files
If you are running the latest version of CC then there is an example of this folder on the devices main memory. The debug log is stored there now instead of in the library (because we might not be able to write the library).

CC might be able to write to that folder if it exists, which means that CC's existing library folder dialog would permit changing the library to that folder. CC might be able to create the "files" folder if the parent folder (com.multi...) exists. Or perhaps nothing will work because the folder must be created by Android with specific permissions.

For right now, I guess I'll have to ignore CC (wahh! just after you added Date Read, too!) and copy stuff to the folder more directly (hmmm, wonder if WebDAV will work, or if I'll have to go USB or pull the card out?), then get it directly through FBReader. I'm not _yet_ going to root just to fix this, but if I find it's broken other stuff as well, I may have to.

If you can get books into a folder on the SD card then you can probably use CC's scan for books to find them and match them with your calibre library. CC does not write to that folder except when adding or deleting books.

[kaufman]

I don't have 4.4 yet, but I imagine that the update is coming soon. Is CC set up now to not write to an SD card under 4.4, or will it return an error if the write fails? Because I am happy to create the Android/data/... Folder by hand and then see if things work.

[EowynCarter]

Originally Posted by theducks

This smells of collusion with device manufacturers to sell more of the premium priced devices that have more on board storage.

Charles, is there a possibility of a Calibre-companion compatible save to disk plugin where we could insert the card into a PC card-writer for the transfer and then sneaker it to the device and have it still be CC usable ?

I sure see the reason behind Google's choice. But, changing that on phones that people bought expecting the SD to work in a certain way, that's hard.

[chaley]

Originally Posted by kaufman

I don't have 4.4 yet, but I imagine that the update is coming soon. Is CC set up now to not write to an SD card under 4.4, or will it return an error if the write fails? Because I am happy to create the Android/data/... Folder by hand and then see if things work.

CC 3.3.4 does two things:
1) It refuses to accept a non-writable folder as a library. It accepts read-only folders in the path to the end folder.
2) It produces an error popup if it cannot write in the library. If connected as a wireless device, because the error will break the protocol, CC will disconnect.

Because of 1), if a hand-created folder is writable then CC should be happy to use it. Please be sure to use the "files" subfolder in the app's private folder, because if/when we directly support the SD card private folder that is where the books will go.

Thanks for doing this.

[jbsavitz]

Hi Chaley,

I have a Nexus 7 2013 tablet running KitKat 4.4 and while the tablet does not have an SD card slot I use Nexus Media Importer with a cables to go adapter which allows me to use a USB flash drive with the tablet. The Nexus Media Importer has several different ways you could address/access the storage device connected to the USB port on the tablet. I'm currently using a 128GB USB flash drive with the adapter and it works fine for video play back at 1080p.

The Nexus Media Importer is an inexpensive app and can support flash card readers as well.

Would you consider having CC access the Nexus Media Importer?

Thanks in advance for your consideration.