Uploading Large Library to CC Taking Forever

[dschnit1]

I recently lost my CC library (I told a cleaning app to delete "junk" files without reading which apps were being targeted) and am in the process of reloading my approx. 9100 book library. I am using wireless connection and batches of 400 each time. After loading 3000 books or so, the process slows down to where it literally takes hours to upload each batch. Is this a known phenomenon or is it my system? I am going from a Windows 10 PC to a Samsung Galaxy Tab S running Android 5, storing the library on a SD card.

[Sydney's Mom]

I attached my tablet to a library, and simply downloaded the books. Why do you need to do it wirelessly?

[theducks]

WiFi work just as fast as USB for me.
You may be pushing the Indexer (speed) portion of CC with 9100 books.

[dschnit1]

Originally Posted by Sydney's Mom

I attached my tablet to a library, and simply downloaded the books. Why do you need to do it wirelessly?

The Calibre Companion database is not the same as the Calibre library. It has to be imported in some way.

[theducks]

Something burped and I just reloaded CC on my TAB2
I sent 985 (EPUB) books via wireless device connection. The whole JOB took 13 minutes (total of 1034 books n the device)

Remember, the metadata in the book is refreshed as part of the task

[chaley]

Originally Posted by dschnit1

I recently lost my CC library (I told a cleaning app to delete "junk" files without reading which apps were being targeted) and am in the process of reloading my approx. 9100 book library. I am using wireless connection and batches of 400 each time. After loading 3000 books or so, the process slows down to where it literally takes hours to upload each batch. Is this a known phenomenon or is it my system? I am going from a Windows 10 PC to a Samsung Galaxy Tab S running Android 5, storing the library on a SD card.

Starting with version 5, CC uses Android's "Storage Access Framework" (SAF). This is required by Android to permit you (a user) to put books on an SD card anywhere other than the private app-specific folder.

One problem with the SAF is that it requires CC to talk to another application/system service to read and write files. The performance of these operations is entirely controlled by that other service. Unfortunately, it appears that this "other service" can be very slow on some devices.

I suggest you check the following things:

• Be sure calibre has enough memory for the library. Run calibre 64 if you can.
• Check that there aren't other apps running on the device, especially ones that consume network resources.
• Verify that you have a good wifi connection from your device, and from the computer running calibre if it uses wifi.

Hmmm ... something just occurred to me. Using a book path that specifies a folder hierarchy might improve things dramatically. The reason: SD cards usually run a FAT32 file system. Directories in FAT32 are not indexed, so the operating system (Android) must do a linear search (1, 2, 3, 4 ...) whenever CC creates a file in order to see if the file is already there. The file is never there, so the time required is the worst case: a search of all the elements. Clearly the more files it must look at the slower it will run. For example, in your case with 3000 books, doing the check runs 3,000 times slower than when the library was empty. The open question is "3,000 time what?" If that "what" is a millisecond then each search is now taking 3 seconds. Several searches are required to store a book. Another open question is how the OS caches the information. A bigger cache will give better performance until the number of files exceeds the size of the cache.

To test this I ran an experiment on my Fire 7/rooted running CM12. I cleared CC's library then sent 600 books to the sd card using three different file name templates. The first template stores all the books in the root folder, using no sub-folders. The second template creates a folder per first-author then stores the book. The third creates a folder per first letter of the first author, then a sub-folder per author (all beginning with the same letter), then the book. The results are quite surprising:

Results
In all cases the average time for the first 10 books is .65 seconds.

Template :: average time per book :: avg last 10 books
{title} - {authors}                                     :: 2.00 secs :: 3.2 secs
{first_author}/{title} - {authors}                      :: 1.15 secs :: 1.2 secs
{first_author:%1.1s}/{first_author}/{title} - {authors} :: 0.65 secs :: 0.65 secs

We see that with the flat template, at the 600th book the time per book is approximately 6 times worse than for the 1st book. For the second template the average time for the last books is approximately 2 times worse. For the third, the average time for last ten books is the same as the first ten.

At 3,000 books and using the first template, the time to store a book should be approximately 16 seconds, or approximately 4 books per minute. For the last template, that time would probably still be very close to 0.65 seconds which is approximately 90 books/minute.

For info: Android internal memory uses a file system that does not have this problem. It is near constant time regardless of the number of files. This is why we don't see these performance problems when using internal memory.

[chaley]

I added the above information to the CC FAQ: I have added a lot of books on an SD card and now downloading books takes forever. What do I do?

[Sydney's Mom]

Originally Posted by dschnit1

The Calibre Companion database is not the same as the Calibre library. It has to be imported in some way.

I attached CC to Calibre via Content Server, downloaded a book, looked at where it was downloading (not a separate file called CC), and then attached my K5 and downloaded books from 2 separate computers. Probably took 5 minutes for all 9000 books. Now Calibre won't recognize my K5 anymore, but the new books I get can be uploaded via Content Server.

When I open CC, every book is there. In the same order, with the same tags and metadata, as on Calibre. But like I said, I had to fiddle around for awhile, to find out where CC was looking for books. Not intuitive, IMHO.

[chaley]

Originally Posted by Sydney's Mom

I attached my tablet to a library, and simply downloaded the books. Why do you need to do it wirelessly?

Originally Posted by Sydney's Mom

I attached CC to Calibre via Content Server, downloaded a book, looked at where it was downloading (not a separate file called CC), and then attached my K5 and downloaded books from 2 separate computers. Probably took 5 minutes for all 9000 books. Now Calibre won't recognize my K5 anymore, but the new books I get can be uploaded via Content Server.

I am not sure what process you are describing.

CC does not see metadata or books downloaded over USB. CC only sees metadata and books downloaded over one of its connections (wireless device, content server, or cloud). If you download over USB then subsequently connect as a wireless device then CC will try to match the books to calibre's library. This may (or may not) work for EPUB and usually doesn't work for the other formats.

If you download a book with CC then later replace the book file with something new downloaded via USB then CC will "see" the old metadata but the book reader app will see the new book file.

When I open CC, every book is there. In the same order, with the same tags and metadata, as on Calibre. But like I said, I had to fiddle around for awhile, to find out where CC was looking for books. Not intuitive, IMHO.

Again, I am not sure what you are doing and what isn't intuitive.

If you are referring to where CC stores book files then the answer is "where you told it to in CC's settings, with the default being the folder "Calibre_Companion" in main memory unless you are using an Amazon device in which case the default is "Books".

If you are referring to something else then could you tell me what it is?

[gsparks2]

I am continuing to download my 24k books into CC. I am up to the G's (title) now.

I am running Android 6 on Samsung S7 Edge Exnos version. I have the Calibre Library on the SD Card. I have the CC folder on the SD card. I connect CC to the local cloud caliber library. The SD card is a 64 gb card that boasts 90 mbs read AND write times. The SD card was formatted by Android.

CC is then reading the book file from SD calibre library then writing the book file to the CC folder on the sd card.

The download times are very very slow. It takes minutes for each book file.

Before I started this I experimented. I put both the Calibre Library and the CC folder in internal memory. It took ~0.2 seconds per book.

Clearly the slowdown is in Android's process for reading/writing to/from the SD card.

I connected to Calibre as a wireless device. Download times are minutes per book when downloading to SD card. Very fast when the CC folder was in internal memory.

[chaley]

Originally Posted by gsparks2

I am continuing to download my 24k books into CC. I am up to the G's (title) now.

I am running Android 6 on Samsung S7 Edge Exnos version. I have the Calibre Library on the SD Card. I have the CC folder on the SD card. I connect CC to the local cloud caliber library. The SD card is a 64 gb card that boasts 90 mbs read AND write times. The SD card was formatted by Android.

CC is then reading the book file from SD calibre library then writing the book file to the CC folder on the sd card.

The download times are very very slow. It takes minutes for each book file.

Before I started this I experimented. I put both the Calibre Library and the CC folder in internal memory. It took ~0.2 seconds per book.

Clearly the slowdown is in Android's process for reading/writing to/from the SD card.

I connected to Calibre as a wireless device. Download times are minutes per book when downloading to SD card. Very fast when the CC folder was in internal memory.

What file template are you using? I ask because the experiment described in post 6 and in the FAQ answer I added show that things can get *really slow* when there are lots of files in a FAT32 directory.

[gsparks2]

Originally Posted by chaley

What file template are you using? I ask because the experiment described in post 6 and in the FAQ answer I added show that things can get *really slow* when there are lots of files in a FAT32 directory.

How do I find what the file template is? Is that in CC or Calibre?

Edit: in Calibre>Preferences>Sending books to devices the Save Template is {author_sort}/{title} - {authors}

Edit: Windows 10 properties of the SD card installed in the phone is "Generic Hierarchical

[chaley]

Originally Posted by gsparks2

How do I find what the file template is? Is that in CC or Calibre?

Edit: in Calibre>Preferences>Sending books to devices the Save Template is {author_sort}/{title} - {authors}

Edit: Windows 10 properties of the SD card installed in the phone is "Generic Hierarchical

This is complicated.

The answer depends on what you have chosen in CC's Settings / File names. To quote the FAQ on this subject

If none of the CC file name settings are chosen, CC's wireless device connection will use the file name provided by calibre, and CC's Cloud and Content Server connections will use the default CC template.

My guess is that you haven't checked any of the boxes. Because you are using the cloud connection, the default template is

{title} - {authors}

which does not use subdirectories. You can easily verify this by looking in the CC folder on your SD card.

I further guess is that you want to use something like the third template from the experiment described in the FAQ answer. That template creates a 3-level hierarchy. The first level is the first letter from the first author's name. I used that because all my authors are LN, FN. If your authors are FN LN then you might choose first_author_sort instead of first_author. The second level is the first author. As before, you might choose first_author_sort. The third is the title - author.

The goal of the exercise is to keep the number of file/directory entries in any given directory as small as possible. To that end, depending on the books in your library, you might choose to use the first letter of title_sort instead of the author for the first level. You would do this if the distribution of books-by-author isn't good (lots of clustering) but the distribution by title is better. You can get an idea of the clustering using the tag browser in calibre: group authors and title by first letter. You could also choose to use 2 letters in the first level, but that runs the risk of creating hundreds of directories at the top level, defeating the purpose of the exercise. Finally, you could choose to add a fourth level: perhaps the first letter of the title (or title_sort) or perhaps the two-letter author just below the top level containing the first letter of the author.

[theducks]

remember that EACH devices remembers the DEFAULT templates ONCE USED. (changing the default in Preferences will not affect those)

With CC connected (as a wireless device): right click the device icon (only there when connected): Configure

[gsparks2]

Thanks!