[Android] slow transfer rate in my new Smartphone

[Pierre-Olivier]

Hi,

I'm facing the same problem of extremely slow transfer rate in my new Smartphone. It's a Galaxy note 4 using Android 6.0.1, with a 64 GB SD Card. I used the "connect as wireless device option"

Previously, I used a galaxy note 2 (Android 4.4.2) with the same card. I also use Galaxy note Pro 12.2 (Android 5.0.2), and a Lenovo Yoga pro 13" tablet. On these 3 devices I never had any problem to transfer my books. Galaxy note pro and Lenovo had also a 64 GB micro SD Card.

I read the article where you explained to have tried different templates, and experienced the same slowness as the number of books increased. After resetting completely CC, the first 11 books where transfered in 3 seconds. Then 193 Books where sent in 8 minutes and 8 seconds : 10 times slowly.

I would like to know what changed ? Is it due to Android 6 ???

I don't understand anything in the way to modify the template for the file name. I would like to know where writing this template you gave as example.

My Calibre is configured to send books with the following template :
{tags}/{#category}/{series}/{title}. It did the following things :

All of my books have a tag (and only one) , it creates a folder following the name of the tag (only 19 existing)

In this folder, if something is written in a personalized column named #category, it creates a folder following the name of this column

Each books which is part of a series is in a folder named according to the name of the serie,

and each books has a name like this : [Serie name-Number in the series] Title

I have always proceed the same way and never had any problem, I use the same template to transfer books on eReader, and it works fine.

I wish to transfer 27 000 eBooks.

I've tried to transfer books, but when reaching 7500 eBooks, Time to transfer one book reach more than 23 seconds, and it was going worst and worst.

So please can you help me ?

PS/ I only transfer ePub format.

[chaley]

Moderator Notice
Moved to new thread

[chaley]

The problem is almost certainly in the never-to-be-damned-enough Android Storage Protection Framework that CC must use to access an SD card in Android version 5 and later. The SPF can be very very slow.

I assume that you verified that the folder structure on the SD card matches your template. It is easy in calibre to think you are using template A when you are in fact using template B.

In your case, my guess is that you are running into exactly the problem described in the FAQ you referenced. The SPF gets more and more slow as you add more book files to a single folder. The way to avoid the slowness is to minimize the number of files in a folder. Using your template, all books without a category and without a series are stored in the {tag} folder. That can be a lot of files.

Solution 1 in the FAQ How do I get the same file names in CC no matter which connection I use? tells you how to enter a CC template.

If you really want to have the {tags}/{#category}/{series}/ in the file name (when using CC, file names aren't very important) then you could try this CC template:

{tags}/{#category}/{series}/{first_author:%1.1s}/{first_author}/{title}

This template avoids the problem described above (no category and no series) by adding folders based on the author name.

[Pierre-Olivier]

Merci Mille fois !

Thank you so much. I finally found where to write the template ...

Transfers are now light speed : 200 books in 147s ! ! !

It's faster than my previous transfers with Android 4 and 5 ...



You speak about Android Storage Protection for Android 5 and later, but my tablet, running Android 5 has no such problem ... Something new in Android 6 ???

[chaley]

Originally Posted by Pierre-Olivier

Merci Mille fois !

Thank you so much. I finally found where to write the template ...

Transfers are now light speed : 200 books in 147s ! ! !

It's faster than my previous transfers with Android 4 and 5 ...

Good. Et je vous en prie.

You speak about Android Storage Protection for Android 5 and later, but my tablet, running Android 5 has no such problem ... Something new in Android 6 ???

The SPF appeared in Android 5. However, we find that the performance of the SPF changes a lot between tablet manufacturers. Also, if you use the defaults (don't change the book storage folder) then CC doesn't use SPF because it knows it can directly read and write the default folder.

Finally, if your tablet was upgraded to Android 5 and if before upgrading you told CC to use the "special" folder on the SD card (Android/data/com.multipie.calibreandroid) then performance might not be affected. CC no longer uses that folder because we had reliability problems.

[Pierre-Olivier]

I was too much enthusiast...

As the number of books increase, the transfer rate is going lower and lower.
52minutes needed to transfer 683 books.

I'm afraid of the future, since I've only transferred 4559 books ...

May I suggest one thing ? Since I faced the problem because I changed my smartphone, but keep the same SD card : Wouldn't it be possible for you to add a saving capability for the Data base (different from the Library) ? For example, someone, like me, want to change his phone but continuing using the same SD card . On the old phone, CC will save the data base on the SD card, then install CC on the new phone (with old SD Card), and import the data base and that's all, Books are in the same place ...

[chaley]

Originally Posted by Pierre-Olivier

I was too much enthusiast...

As the number of books increase, the transfer rate is going lower and lower.
52minutes needed to transfer 683 books.

I'm afraid of the future, since I've only transferred 4559 books ...

What this means is that you have a lot of books where the authors have the same first letter, or even worse that you have a lot of books by the same author. The SPF begins to slow down if there are more than around 300 files/folders in a given folder. There is nothing that CC can do to prevent this.

What is needed is to find some metadata that ensures that the books are distributed through folders so that no folder has more than a few hundred folders/files in it. It appears that in your case using the first author doesn't achieve that goal. I am guessing with my suggestions because only you know what is in your library. That said, changing the template to

{uuid:%2.2s}/{title}

will introduce a lot of randomness. The uuid is (pseudo-)random so that should split the top level into 256 folders (the uuid is in hex, base 16). Assuming a perfect distribution (perhaps a bad assumption) this template will create 256 folders each containing around 100 books. Of course, using the uuid means that the path names are useless for browsing with a file manager, but that really shouldn't matter.

If you want even more separation then perhaps use

{uuid:%2.2s}/{first_author:%1.1s}/{title}

May I suggest one thing ? Since I faced the problem because I changed my smartphone, but keep the same SD card : Wouldn't it be possible for you to add a saving capability for the Data base (different from the Library) ? For example, someone, like me, want to change his phone but continuing using the same SD card . On the old phone, CC will save the data base on the SD card, then install CC on the new phone (with old SD Card), and import the data base and that's all, Books are in the same place ...

How does this differ from the existing backup & recovery mechanism? You can save the database and settings then reload them on the new device.

You are assuming that the new version of android would permit reading the old SD card folders, something that is not guaranteed. It is common for the new version to require re-initializing the SD card.

[Pierre-Olivier]

Thank you.

If i try the new template, am I obliged to erase all the books previously loaded ?

[chaley]

Originally Posted by Pierre-Olivier

Thank you.

If i try the new template, am I obliged to erase all the books previously loaded ?

No need to resend the books. New and resent books will use the new template.

[Pierre-Olivier]

Can you tell me what tools I can use in order to verify where the books are stored ? On my previous smartphones, I found them easily with any file manager, but now, I can't find them ... ?

[chaley]

I use the file manager Amaze. It can access the SD card. It might need to ask for permission. In Amaze, each storage area is a "screen" that you access by swiping left and right.

I am surprised that your current file manager can't open the SD card. There is probably a storage selection option somewhere.

[Pierre-Olivier]

Hi',

That's finished ! I've finally achieved to transfer all my books on my Galaxy Note 4 using the template you gave previously.

At the beginning, the transfer rate lowered a little as the number of books increased. But, after 5-7000 books, it was stabilized, between 5-6 seconds / book. Strangely, the last 4000 books were transferred quickly : 4 seconds / book.

I would like to thank you for your precious help.

But, Android 5 and 6 are really painful ... It's a shame to see such a regression in the functionalities ... I really hope to see improvements in future releases, because I think (and hope) that the developper's community is protesting against this ...

I realize, a bit too late that I could have, perhaps, avoid this long, very long operation with using the Backup function*, in order to copy the settings (and database ?) on my new phone, and then inserted the micro SD card of my old phone, in the new one. You told me previously, that perhaps the phone should need to format the micro SD ... but if not, I would have save a lot of time. But I've learned a lot about Android functionalities ...

* I did't notice that this option was available (shame on me ...)

I've also found why I could'nt find my books : I was using, in the template, some columns name that were not understood by CC, so that the books were stored in folder named [invalid column],