[Android App] Calibre Companion - experiences syncing a 2000+ book library

[ellett]

An elegantly designed app that works well most of the time. I'm glad I bought it.

I especially love the abiliity to sort by author and series. I'll be using CC as my library manager and just using Aldiko Premium (or whatever may work better with CC) as my reader.

Some observations:
So far I have about 1000 books synced to my phone. When I start CC it takes over a minute to open. I'm assuming the initial open time will increase as I continue to add books. Is CC scanning the collection when it starts or does it maintain a metadata file to provide the quickest startup?

With 2200 books in calibre, when I touch connect it takes over 4 minutes before calibre sees the device, and when I send books to device I only get about two books per minute transferred. Much, much, much slower than syncing my old phone via USB using a drive letter.

Given the slow sync speed, I've been selecting a couple of hundred books and sending to device. That runs fine for a while but occasionally I get a connection problem with calibre (it appears to be a standard timeout, see debug at bottom of post). In that case, CC continues to show the syncing with calibre screen but there is no progress. The problem is that I cannot return out of the application; it's stuck and apparently doesn't realize calibre is no longer communicating. I've found the easiest thing is just to reboot my phone.

I recommend that at the bottom of all of the CC screens there be a cancel button. Cancel could either terminate the operation immediately, attempting to notify calibre of a "problem", or cancel could wait a few seconds to attempt to let the current transfer finish and then terminate.

I don't know why I experience occasional time outs. I have the phone within a few feet of the wifi router, showing max signal strength and I have nothing else running on the PC except a low-bandwidth internet radio station I use to verify that there is no signal interruption. The phone is sitting on my desk not being used throughout the transfer.

calibre debug:

calibre, version 0.8.64
ERROR: Error: Error communicating with device

There was a timeout while communicating with the device in function: Device did not respond in reasonable time

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 83, in run
File "site-packages\calibre\gui2\device.py", line 478, in _upload_books
File "site-packages\calibre\devices\smart_device_app\driver.p y", line 43, in _synchronizer
File "site-packages\calibre\devices\smart_device_app\driver.p y", line 749, in upload_books
File "site-packages\calibre\devices\smart_device_app\driver.p y", line 396, in _put_file
File "site-packages\calibre\devices\smart_device_app\driver.p y", line 372, in _call_client
TimeoutError: There was a timeout while communicating with the device in function: Device did not respond in reasonable time

[ellett]

I can't imagine I'll ever want any other view and it's a pain to have to reset to group author sort series every time I come back into CC. It also takes over 30 seconds to do the grouping.

Thanks.

[ellett]

I get this message on the phone at random times, most recently when I touched return to exit the program?! I also get this intermittently in the middle of syncing with calibre, after pressing connect etc. I always press wait and CC continues operation.

[chaley]

Some quick notes:

1) to be honest, we never considered that people would transfer a thousand or more books. The app is made to be very responsive to sorts and groups at the 50-200 book level.

2) This weekend we are going to redo how memory is used. My suspicion is that we will improve response time (time to screen doing something) but reduce throughput (time to scroll, etc). Unfortunately there is no magic wand. Sorting and grouping requires *something* to look at all the metadata, and this looking takes time.

3) When CC connects to calibre, it must send the metadata for *every* book back to calibre. This is not an option -- it is what calibre demands. Each network transaction (each book's metadata) requires around 1/3 second (wifi turnaround -- we haven't found a way to improve this), so 1000 books takes 333 seconds, or around 4 minutes total. We might be able to improve this somehow, perhaps by sending more than one book at a time, but such a fix is not likely to be in the next few days.

4) We should be able to improve the time to send a book. That is in the todo queue.

5) Calibre does not support the notion of cancel from a device. It thinks that a device is a passive disk, and disks that "cancel" are broken disks. We could disconnect, but that is a very large hammer for a potentially small nail.

6) Sorts are already saved. I agree about saving the grouping. That said, saving the and group sort won't save any time. Android can kill the app when it is shoved to background. When it is restarted, it must rebuild the displays.

7) Timeouts. Calibre waits for 60 seconds for the device to respond. Very large libraries might prevent responding for longer than that. It seems that your large number of books are causing CC to taje longer than that to say "hello" back to calibre. We will see if we can do better at "responding" even if CC is busy.

We understand the frustration of people who want to manipulate large numbers of books on their phone. It doesn't work well today, primarily because grouping and sorting operations are not consistent with large data sets on a tiny device. Our problem: it isn't clear if we can make it work well in both the "normal" and "large" library cases. We will try, but I can't promise anything.

[ellett]

I only have about a quarter of my epubs loaded into calibre so far, right now that's 2200 books. So as time goes on I'll be swinging a bigger and bigger mallet at CC. If anyone's going to break CC from sheer scale, it's probably going to be me

It's understood that short of writing some sort of plug-in that caches what's on an Android device, the initial sync is going to have to exchange metadata for all books on the phone. That's just reality. And trying to cache the contents of possible multiple Android devices that get connected to a single calibre instance would open up a whole 'nother can of worms.

The speed during the initial sync at connect time and especially during transfer aren't optimal (and the transfer speed seems to be slowing down even further as the number of books on the phone increases?!?!?!), but once I get my collection synced, the number of books to sync at a given time will drop to the level of whatever I add in an hour or so's work during the evening. I can set the phone by my bedside and let the sync run overnight, so long as there's no timeout that stops the process. So I'd say that if you can ensure that CC does an ack with calibre often enough that timeouts don't occur, that's by far the most important improvement.

Sorting and grouping taking 30 seconds doesn't bother me much; it's just that CC forgets that that's how I want my display to look and I have to go through the process again, and again.

I was thinking that once you built a display grouped, sorted etc. to a user's satisfaction, there would be some sort of staged data that reflected the display that could be referred to so the screen could be reloaded very fast. So if CC comes out of background it would just refer to the staged data to repaint the display rather than having to go back to the books to re-retrieve the metadata and re-group and re-sort it.

So far, it's when calibre has already broken the connection that I'd like to have a cancel function. Right now CC just hangs, and pressing return repeatedly does nothing. That's what I'd like to see fixed in one way or another.

One thing that has me nervous is that it's at the very end of the transfer process, after all books are transferred, that I see a message about updating metadata. Does that leave the risk of having several hours of transfers not being reflected on the screen if something happens 98% through a transfer? Would it be better to checkpoint the metadata after each n books, or does everything heal itself the next time a connect is done?

Thanks again. CC is a really cool product.

[ellett]

One personal note from a different viewpoint. If I only had 50 books I wouldn't be using software at all, I'd be dragging the books over to the phone with Windows Explorer. Software becomes valuable when a computerized solution is able to manage something that's become unwieldy or downright impossible to do manually due to complexity or scale or some other factor.

In CC's case, when someone hits a few hundred books, that's when you hit the knee of the value curve and CC comes into its own. At that point there's no longer any way to keep a book inventory on the phone, updated and accurately grouped by author and series, without a software solution.

[DoctorOhh]

Originally Posted by chaley

6) Sorts are already saved. I agree about saving the grouping.

I see sorting saved for the full list when starting up, but when I go to groups by author and select an author the sort is always by title. I then choose to sort by series yet when I go to a second author, the sort is back to title again.

[chaley]

Originally Posted by ellett

One personal note from a different viewpoint. If I only had 50 books I wouldn't be using software at all, I'd be dragging the books over to the phone with Windows Explorer. Software becomes valuable when a computerized solution is able to manage something that's become unwieldy or downright impossible to do manually due to complexity or scale or some other factor.

In CC's case, when someone hits a few hundred books, that's when you hit the knee of the value curve and CC comes into its own. At that point there's no longer any way to keep a book inventory on the phone, updated and accurately grouped by author and series, without a software solution.

It was our thought that someone would manage books with calibre, and then use calibre companion to put some books on the device to be read in the next while. In my two years of doing development work on calibre and its device drivers, I have never been able to understand why people want to put more than a month or two of reading material on a device.

My lack of comprehension notwithstanding, people *do* want to put hundreds if not thousands of books on a device. We will do what we can to support that use case.

[chaley]

Originally Posted by ellett

I was thinking that once you built a display grouped, sorted etc. to a user's satisfaction, there would be some sort of staged data that reflected the display that could be referred to so the screen could be reloaded very fast. So if CC comes out of background it would just refer to the staged data to repaint the display rather than having to go back to the books to re-retrieve the metadata and re-group and re-sort it.

The problem here is that the android OS frequently kills apps when they "go to sleep". When they are awakened, they are in fact restarted from scratch. If CC is actually reawakened, which seems almost never to happen, then it reuses what it computed before.

So far, it's when calibre has already broken the connection that I'd like to have a cancel function. Right now CC just hangs, and pressing return repeatedly does nothing. That's what I'd like to see fixed in one way or another.

What "return" are you pressing?

One thing that has me nervous is that it's at the very end of the transfer process, after all books are transferred, that I see a message about updating metadata. Does that leave the risk of having several hours of transfers not being reflected on the screen if something happens 98% through a transfer? Would it be better to checkpoint the metadata after each n books, or does everything heal itself the next time a connect is done?

In calibre, most devices must separate the metadata update process from the send book process. There aren't any "smarts" in the device to grab the metadata as it goes by when sending a book. However, CC does have those smarts and does grab it then. Another thing that can happen when in automatic metadata management mode (which I think all CC users should be using) is that calibre can actually make updates to metadata after a send, but this is rare.

To avoid a lot of useless work in both cases, the device driver (in calibre) compares the metadata that is to be sent with what the CC already has. Because CC grabs metadata on the fly, it is rare that significant transfers are actually done here. A similar process is done during startup, where the device driver compares the metadata CC sent back to the metadata that automatic management wants to send. New metadata is sent to CC only if there are changes, for example if you changed tags, series, or some such.

Thanks again. CC is a really cool product.

Thank you for your comments. They are helpful.

[chaley]

Originally Posted by dwanthny

I see sorting saved for the full list when starting up, but when I go to groups by author and select an author the sort is always by title. I then choose to sort by series yet when I go to a second author, the sort is back to title again.

Yea, there was some leftover code to force sorting according to the grouping. I took it out.

[Adoby]

Slow sync speed is mainly a problem during the initial sync, if you have a large ebook library. After that you sync and transfer only changed and added books. Since CC is so easy and quick to use that can be done very often. You sit at your computer, add books to Calibre, or fix metadata, and have your phone or tablet with you and sync right away.

My tablet has 64 GB of memory. Assume that an average novel in ePub format is around 500 kB. That means that there really is no reason for me not to keep ALL my ePubs available on my tablet. Actually I have room for many more books on my tablet, than what I've actually got. Why wouldn't I want to have them all available on my tablet? One obvious reason is as a backup. Another is that I don't know in advance what I would like to read next. And I might like to reread or check stuff in some books. My "good" set of books (books that I like to have available) is around 2000 books. Mostly ePubs, but also some PDFs. It would be fantastic to be able to update that library using Wifi, instead of removing and syncing to a removable external memory card.

Hopefully CC will in the near future allow me to do this. (Fingers crossed.) Currently CC crash due to memory issues after around 580 books.

I use Calibre to add new books or to fiddle with metadata and series info and so on. And I read using my tablet. I currently use an elaborate scheme with folders for genres to allow me to easily create and update collections in Mantano. Works fine, but WiFi with CC would be much better!

[ellett]

This morning CC connected after one false "cannot find" attempt, and when I tried to push some books to my Galaxy Nexus, wow what a difference! Most books are transferring in a couple of seconds. Whatever y'all did @Chaley, it made a huge improvement. Thanks.

[ellett]

Woo hoo! I now have 2200 titles synced to my Galaxy Nexus! That was fast.

[Penfist]

I cannot sync at all. The reader connects to the library, but then I get a blank screen and nothing happens. I've waited two or three hours...

[DoctorOhh]

Originally Posted by Penfist

I cannot sync at all. The reader connects to the library, but then I get a blank screen and nothing happens. I've waited two or three hours...

CC does not automatically sync all of your books to your device. After you connect to calibre you have to go into calibre and choose the books you want sent to your device and click the Send to device icon? Nothing will happen without your input.