Extremely slow book import into companion

[rmzg]

The background:
I have relatively new phone (samsung note5) with the latest (paid) version of Calibre Companion installed.

I have a moderately large ebook library (7000+, epubs) stored in Calibre on a fast PC. The wifi connecting them is AC+ level meaning 200+mbits.

The short issue is that no matter what I try any type of 'book import' into the companion app is insanely slow.

At first I tried connecting the phone as a 'wireless device' and doing the transfer as a push from calibre. That mostly seemed to work as long as I didn't select more than a few hundred at a time to copy but it took on the order of 60+ minutes to transfer 200 books, which is pretty insane.

I gave up on that and manually copied the calibre folder from the PC to the phone via USB (took about 50 minutes total) and then selected that as a 'cloud server' via the connect menu and told it to download everything (because I have to redownload files already on the phone??)

It's currently running and according to its progress meter it's completing maybe one book per minute. This sucks. How do I make it faster?

[DoctorOhh]

This FAQ entry may help. http://cc_faq.multipie.co.uk/index.php?solution_id=1094

[rmzg]

I'm actually attempting to transfer directly to the main memory of the device which I assume isn't using fat32??

[rmzg]

Nope, changing the file name template doesn't appear to change much (although why on earth it defaults to all files in a single directory..)

[DoctorOhh]

Originally Posted by rmzg

Nope, changing the file name template doesn't appear to change much (although why on earth it defaults to all files in a single directory..)

You might want to send Charles a debug log so he can see exactly what is happening in your Calibre Companion.

[chaley]

Originally Posted by rmzg

Nope, changing the file name template doesn't appear to change much (although why on earth it defaults to all files in a single directory..)

It defaults to that because that is what most users want. The slowdown (normally) doesn't happen until one get hundreds of books. Sixty percent of CC users have fewer than 500 books in their library, and it is rare for the problem to appear for libraries in main memory.

As for why your device is slow, I cannot say. If you changed the template, did you reload the books? If not then you might still be seeing the slowdown because the directories share the same "level" as the already-there books. Up to now we have not encountered a case where enabling the two options in Settings / File names, changing the template as discussed in the FAQ, and reloading the books has not fixed the problem.

Assuming that adding the first book was quick but it slowed down as more books were added, if changing the template does not fix the problem then I am out of ideas. If we cannot solve your problems you are welcome to ask for a refund. If you want to do that then use "Contact Us" on the www.multipie.co.uk website.

Technical note: The local library cloud connection simply copies the file using Android system interfaces then adds a record to its database. It does almost nothing else. If this is slow then either reading a file from the SD card is very slow, creating/writing a file in main memory (using the SAF) is very slow, or both. It could be that writing the database might be the cause of the slowdown, but given that some people have in excess of 20,000 books and aren't seeing a DB slowdown this possibility is unlikely.

Hmmmm ... random thought. If you have a lot of apps that run all the time then CC could be dramatically affected. Every time CC asks Android to copy a bit of a file Android will consider swapping to some other application. If there is always such an app available then CC will lose. I consider this possibility very unlikely, but it is worth asking.

[rmzg]

Hey,

As I tried to explain before, there's no actual SD card involved, both Companion and the original copied library of books exist on the 'main memory' of the device. It's a note 5 so I couldn't even add a SD card if I wanted to!

I'm going to try again with a clean re-install of CC and try the template change again and see if I can get slightly more precise data. For reference copying the entire folder to the device over usb in the first place took about 60 minutes.

I have roughly 7.5k books in the library right now, can anyone give me some vague ballpark numbers as to how long that should take to import into CC as an on-disk 'cloud library' connection?

Also, I'm sure this gets asked all the time, but if I have a complete copy of a Calibre library including metadata.db why does CC need to duplicate it file by file?

[rmzg]

Update:

This time it's going much faster so far, so I guess we can blame my incompetence for the previous result. On the other hand, I am getting a ton of messages about "Can't save {book title} to main memory", specifically when the title contains a '#' character. This seems like a minor issue somewhere..

Update the second:

I left the device alone for 10 or 15 minutes and it made it down to something like "2771" books remaining and refused to move no matter how long I watched it so I disconnected, reconnected and hit 'download all' again. It told me it was going to download all 7500 books again but when I hit 'yes' it manage to only queue up the roughly 2700 remaining.

Still whining about titles with # though. No idea why.

[chaley]

Originally Posted by rmzg

Update:

This time it's going much faster so far, so I guess we can blame my incompetence for the previous result. On the other hand, I am getting a ton of messages about "Can't save {book title} to main memory", specifically when the title contains a '#' character. This seems like a minor issue somewhere..

What connection are you using? I ask because CC doesn't contain a message like the one you describe. Is calibre producing the message?

Update the second:

I left the device alone for 10 or 15 minutes and it made it down to something like "2771" books remaining and refused to move no matter how long I watched it so I disconnected, reconnected and hit 'download all' again. It told me it was going to download all 7500 books again but when I hit 'yes' it manage to only queue up the roughly 2700 remaining.

Still whining about titles with # though. No idea why.

This implies you are using the content server. Is that the case?

Queuing all the books is normal. CC determines that it already has the book during the queue processing phase.

As for the error, it could be something in calibre or in CC. If you send me debug logs as described in our FAQ How do I get help? I might be able to work out what is going wrong. Please erase the multipie.co.uk address.

EDIT: I just realized that you might have been "connecting" to the local cloud library and using its download-all function. That would mean that calibre isn't directly involved. If this is what you are doing then I am confused because I can't find any trace the message you describe. Is this the message you are seeing: "Cannot save book {title} to device memory"?

[rmzg]

I have a folder on the phone that I specified as my 'cloud connection' source that I then selected 'download all' from.

The exact error is "Cannot save book {title} to device memory"

Where {title} is the actual title of the book e.g. "Joe's Mystery Case #21"

[chaley]

Originally Posted by rmzg

I have a folder on the phone that I specified as my 'cloud connection' source that I then selected 'download all' from.

The exact error is "Cannot save book {title} to device memory"

Where {title} is the actual title of the book e.g. "Joe's Mystery Case #21"

Thanks for the clarification.

Without a debug log I can't say exactly what is happening, but I can make an educated guess. The calibre database contains the text of the file path from the root of the library to a book. The stored files and folders must exactly match that path or CC will not be able to find ("download") the book. Is it possible that the method you used to transfer the library to your device "helped" you out and changed the actual paths, perhaps by changing the # character to something else? You can check this by looking at the folders/files for some failed books. If there is a # in the title but that character is missing/changed in the title folder name or the actual book file name then CC will fail.

As an experiment I copied a small library from a Windows 10 machine to one of my devices using Windows MTP. One of the books has a # in the title. The folders and files were created correctly, and CC could "download" that book.

[rmzg]

Good guess as to the issue but confusion deepens over the cause.
Some relevant sections of the debug log below.

2016-09-28 23:50:28.318: checkLpathComponents started L/Laurence E. Dahners/Wanted (An Ell Donsaii Story #10) - Laurence E. Dahners w/URI=false
2016-09-28 23:50:28.319: Cloud: enqueing book already on device: 5125 Thief's Covenant
2016-09-28 23:50:28.323: Cloud: enqueing book already on device: 5124 The Wrong Sword
2016-09-28 23:50:28.325: checkLpathComponents finished L/Laurence E. Dahners/Wanted (An Ell Donsaii Story #10) - Laurence E. Dahners
2016-09-28 23:50:28.326: Cloud downloadFile: url=Laurence E. Dahners/Wanted (An Ell Donsaii Story #10) (6572)/Wanted (An Ell Donsaii Story #1 - Laurence E. Dahners.epub, localLpath=L/Laurence E. Dahners/Wanted (An Ell Donsaii Story #10) - Laurence E. Dahners.epub
2016-09-28 23:50:28.326: Cloud: enqueing book already on device: 5123 The Huntsman's Amulet
2016-09-28 23:50:28.327: cloud downloading
java.io.FileNotFoundException: /sdcard/Download/Calibre Library/Laurence E. Dahners/Wanted (An Ell Donsaii Story #10) (6572)/Wanted (An Ell Donsaii Story #1 - Laurence E. Dahners.epub: open failed: ENOENT (No such file or directory)
at libcore.io.IoBridge.open(IoBridge.java:452)
at java.io.FileInputStream.<init>(FileInputStream.jav a:76)
at com.multipie.cclibrary.Cloud.a.a.getFile(Source:94 )
at com.multipie.cclibrary.Cloud.b.a(Source:39)
at com.multipie.cclibrary.Cloud.af.run(Source:123)
Caused by: android.system.ErrnoException: open failed: ENOENT (No such file or directory)

When I check the folder on the PHONE I find the following path:

This PC\SAMSUNG-SM-N920A\Phone\Download\Calibre Library\Laurence E. Dahners\Wanted (An Ell Donsaii Story #10) (6572)\Wanted (An Ell Donsaii Story _1 - Laurence E. Dahners.epub

The book is saved on my LAPTOP's calibre library as:

\Documents\Calibre Library\Laurence E. Dahners\Wanted (An Ell Donsaii Story #10) (6572)\Wanted (An Ell Donsaii Story #1 - Laurence E. Dahners.epub

I have no idea as to why none of these individual bits match up: Why does the PHONE version have _1? Why do neither of the filenames contain '10' like the folder does? And so on.

I added the folder to my PHONE originally by simply opening it in the explorer window via "This PC" (Windows 10), navigating to the folder, then using ctrl-c and ctrl-v to paste the calibre library folder.

When I try this again with just the individual book folder I end up with an epub file named

Wanted (An Ell Donsaii Story #1 - Laurence E. Dahners.epub

on the phone.

I've tried copying the folder and file to the phone again in various combinations and I can't get it to reproduce the file name change.

I'll note that in my initial attempt to copy the Calibre Library over, something was *seriously* unhappy with the file named 'metadata.db', any attempt to copy files that included a file named that generated an error message along the lines of "The device has stopped responding or disconnected", so, uh, that's weird. I eventually changed the name of the file to something innocuous, copied it to the phone, then used the phone's file manager to rename it.

[chaley]

Originally Posted by rmzg

I have no idea as to why none of these individual bits match up: Why does the PHONE version have _1? Why do neither of the filenames contain '10' like the folder does? And so on.

The # was likely changed to _ by some component involved in the copy. Some people think that # is an invalid character so their apps replace it. Exactly which component I cannot say. It did not happen for me when I copied my library using copy/paste as you did.

It is worth noting that MTP is really flaky. I have personally seen it lose files (not show files that are on the device) and create files (show files that are not on the device). I avoid using it like the plague. Usually when I need to do a transfer I use one of the cloud tools, usually dropbox and dropsync. There are tools available like AirDroid that I haven't tried.

The 10 becoming 1 could be calibre shrinking the path component. It does that when it thinks there is a possibility that the name is too long.

I've tried copying the folder and file to the phone again in various combinations and I can't get it to reproduce the file name change.

I'll note that in my initial attempt to copy the Calibre Library over, something was *seriously* unhappy with the file named 'metadata.db', any attempt to copy files that included a file named that generated an error message along the lines of "The device has stopped responding or disconnected", so, uh, that's weird. I eventually changed the name of the file to something innocuous, copied it to the phone, then used the phone's file manager to rename it.

Given that the "long transfer" problem is apparently solved, use the wireless device connection. It handles all file operations on the appropriate machine so is not subject to these arbitrary renaming hassles.

[rmzg]

Originally Posted by chaley

The 10 becoming 1 could be calibre shrinking the path component. It does that when it thinks there is a possibility that the name is too long.
Given that the "long transfer" problem is apparently solved, use the wireless device connection. It handles all file operations on the appropriate machine so is not subject to these arbitrary renaming hassles.

Yeah, I hooked it up as a wireless device and synced over the remaining books and it seems to be working pretty well although there's a slight discrepancy between the number of books calibre thinks it has and what calibre companion says it has-- about 50 or so. I'll figure it out some other day.


By the way, is there a short cut for 'clearing' a search? Like, after I searched and it displayed a subset what do I press to go back to just the default 'all items' display?

[theducks]

Originally Posted by rmzg

Yeah, I hooked it up as a wireless device and synced over the remaining books and it seems to be working pretty well although there's a slight discrepancy between the number of books calibre thinks it has and what calibre companion says it has-- about 50 or so. I'll figure it out some other day.


By the way, is there a short cut for 'clearing' a search? Like, after I searched and it displayed a subset what do I press to go back to just the default 'all items' display?

Start another Search
There is a Clear button on the dialog