Uploading Large Library to CC Taking Forever

[gsparks2]

Originally Posted by chaley

This is complicated.

The answer depends on what you have chosen in CC's Settings / File names. To quote the FAQ on this subject
My guess is that you haven't checked any of the boxes. Because you are using the cloud connection, the default template is which does not use subdirectories. You can easily verify this by looking in the CC folder on your SD card.

I further guess is that you want to use something like the third template from the experiment described in the FAQ answer. That template creates a 3-level hierarchy. The first level is the first letter from the first author's name. I used that because all my authors are LN, FN. If your authors are FN LN then you might choose first_author_sort instead of first_author. The second level is the first author. As before, you might choose first_author_sort. The third is the title - author.

The goal of the exercise is to keep the number of file/directory entries in any given directory as small as possible. To that end, depending on the books in your library, you might choose to use the first letter of title_sort instead of the author for the first level. You would do this if the distribution of books-by-author isn't good (lots of clustering) but the distribution by title is better. You can get an idea of the clustering using the tag browser in calibre: group authors and title by first letter. You could also choose to use 2 letters in the first level, but that runs the risk of creating hundreds of directories at the top level, defeating the purpose of the exercise. Finally, you could choose to add a fourth level: perhaps the first letter of the title (or title_sort) or perhaps the two-letter author just below the top level containing the first letter of the author.

This is working great! Thank you!
Edit:
I added a fourth level to the template:
{first_author:%1.1s}/{first_author}/{title:%1.1s}/{title} - {authors}

Then I ran into the path length limit for some books with long titles. I manually saved those to the CC book folder and the next wireless sync they got added to the CC data base.

[chaley]

Originally Posted by gsparks2

This is working great! Thank you!
Edit:
I added a fourth level to the template:
{first_author:%1.1s}/{first_author}/{title:%1.1s}/{title} - {authors}

Then I ran into the path length limit for some books with long titles. I manually saved those to the CC book folder and the next wireless sync they got added to the CC data base.

Thanks for the feedback. Do you have any numbers on performance? I would like to add your experiences to the FAQ.

What the problem with path lengths? Are you saying that CC or calibre choked on a path? CC is supposed to deal with that by shortening the path components, and calibre is very careful to create proper paths. Perhaps it did that and you didn't like the result?

[gsparks2]

Originally Posted by chaley

Thanks for the feedback. Do you have any numbers on performance? I would like to add your experiences to the FAQ.

What the problem with path lengths? Are you saying that CC or calibre choked on a path? CC is supposed to deal with that by shortening the path components, and calibre is very careful to create proper paths. Perhaps it did that and you didn't like the result?

The download times were less than a second at the start. After 15k or so I noticed a slowdown to about 1 sec per book.

I {believe} the issue with the path length limit is the ~256 character limit for the complete path stored in the FAT32 table. I think that Android may have not liked the path for some reason.

Example: /storage/6144-AD4B/CC Books/{first letter of author's first name}/{Author's full name}/{first letter of the first word in the title}/{title} - {author}. 6144-AD4B is the label for the SD card.
For books with long titles and long author names I got the error message "unable to save {title} - {author}" or something of that sort. This appeared in a gray box and went away by itself. There were only 145 books out of the 24k that this message popped up.

All of the above was while downloading from local cloud Calibre library on the SD card. I tried connecting as a wireless device and downloading the same books from Calibre on the PC. The first book caused CC to disconnect from Calibre without error message.

Subsequently, from Calibre, I saved them as epub in a single directory to root of CC Books with the template ({series}) - {title} by {author}. I then connected CC as a wireless device. CC found the books and downloaded the metadata from Calibre.

EDIT: I tried to recreate the errors but the books download just fine from local cloud calibre library. Some of the 145 had duplicates in the Calibre Library. Some of the 145 had author first name starting with V. CC created folders with both upper case v and lowercase v. Multiple folders V were created V(1), V(2). these disappeared along with their book files. This may need to go into the mystery file unless it pops up again.

[chaley]

@gsparks2: thank you. If I understand you correctly, the download time went from minutes per book to a second or less. Excellent.

Your comments triggered some questions, so I (finally) did enough research to better understand path lengths. CC gets it wrong.

As far as I can tell, no file system used in Android has a draconian limit on path length. However, other operating systems such as Windows that might read an SD card do. For example, Windows has a limit of 255 characters. CC gets this right.

On the other hand, several file systems used in Android such as ext3 have a limit on a single component of a file path. The ext3 limit is 255 bytes, which if using unicode is 127 characters. The FAT32 limit is 255 characters.

CC doesn't get the second limit right. I will fix this.

[gsparks2]

Originally Posted by chaley

@gsparks2: thank you. If I understand you correctly, the download time went from minutes per book to a second or less. Excellent.

Your comments triggered some questions, so I (finally) did enough research to better understand path lengths. CC gets it wrong.

As far as I can tell, no file system used in Android has a draconian limit on path length. However, other operating systems such as Windows that might read an SD card do. For example, Windows has a limit of 255 characters. CC gets this right.

On the other hand, several file systems used in Android such as ext3 have a limit on a single component of a file path. The ext3 limit is 255 bytes, which if using unicode is 127 characters. The FAT32 limit is 255 characters.

CC doesn't get the second limit right. I will fix this.

Thank you! I am not sure now that I ran into any path length issue, but glad I could trigger questions that led to improvement.

[Steven Karp]

ETA: Oh, wait, I just read the FAQ more carefully. I see I'll need to re-download.

Am I correct in believing that if I use the "Replace" option described in the FAQ, the thousands of files in the base directory will actually be moved to their new locations (as opposed to deleted and re-downloaded)?

---

If I change the template in CC as described, what will happen to the thousands of books currently in the library? I'm guessing that CC won't reorganize them automatically, right?

If it won't, can I force a reorganization by having CC move the files to a new directory?

Or, if I pop the SD card into my computer and rearrange manually, will CC find them in the new locations?

Or am I just going to have to bite the bullet, wipe the directory, and redownload the entire collection?

[chaley]

Originally Posted by Steven Karp

ETA: Oh, wait, I just read the FAQ more carefully. I see I'll need to re-download.

Am I correct in believing that if I use the "Replace" option described in the FAQ, the thousands of files in the base directory will actually be moved to their new locations (as opposed to deleted and re-downloaded)?

No, you are not correct. The old book files will be deleted when the new book file is is downloaded. The effect is the same as overwriting the existing file.

Or, if I pop the SD card into my computer and rearrange manually, will CC find them in the new locations?

This won't work because a) CC won't know you did it, and B) the chances of getting the names exactly as CC produces isn't high.

Two thoughts:

• It seems that the OS (Android) keeps some caches of directories. If this is true then sending new books should be much faster even if the library has not yet been "cleaned".
• You can download books in segments rather easily by deleting the existing book files with a file manager then using the Wireless Device/Sync Books connection type. CC assumes that missing book files need to be re-downloaded.

EDIT: if you are going to do a massive re-download be sure to use the CC V5.1.3.3 released today. It has fixes for some strange Android behavior when file names differ only in letter case.

[Steven Karp]

Originally Posted by chaley

EDIT: if you are going to do a massive re-download be sure to use the CC V5.1.3.3 released today. It has fixes for some strange Android behavior when file names differ only in letter case.

Now he tells me...

Unfortunately, I'm on an Amazon Fire, where CC doesn't seem to have updated past 5.1.2.4 yet. (Update: the tablet notified me about the update as I was typing this message. *sigh*)

It went smoothly enough (albeit slowly, for obvious reasons). Ran into what was probably the issue you're referring to with L. Sprague de Camp, when CC created a bunch of folders ("d (1)", "d (2)", etc.)

I edited Calibre to sort as DeCamp and that solved the problem well enough for my purposes.

[Sydney's Mom]

Originally Posted by chaley

I am not sure what process you are describing.

CC does not see metadata or books downloaded over USB. CC only sees metadata and books downloaded over one of its connections (wireless device, content server, or cloud). If you download over USB then subsequently connect as a wireless device then CC will try to match the books to calibre's library. This may (or may not) work for EPUB and usually doesn't work for the other formats.

If you download a book with CC then later replace the book file with something new downloaded via USB then CC will "see" the old metadata but the book reader app will see the new book file.
Again, I am not sure what you are doing and what isn't intuitive.

If you are referring to where CC stores book files then the answer is "where you told it to in CC's settings, with the default being the folder "Calibre_Companion" in main memory unless you are using an Amazon device in which case the default is "Books".

If you are referring to something else then could you tell me what it is?

Not a problem. What I meant was if I use a USB and try and have Calibre manage my device, after about 5 minutes, it just stops. Since I never had that problem when I had a normal amount of books on my device, I assume it is the sheer number, which is over 10,000. And the fact that it hangs during the "reading books on device" stage. I use Calibre, not the Amazon cloud, to manage my books, and prefer to select a book right on CC and hit read. They get on CC during the once a month I connect via content server, It is working like a dream, and sorry for saying it wasn't intuitive. I don't remember the problem that lead to that snarky comment.