[Android App] Calibre Companion - experiences syncing a 2000+ book library

[ellett]

So syncing with my Galaxy Nexus is fast and errors or few. I'm always looking for trouble so I started trying to use CC to sync with my rooted Nook Color, running CM7.

I get somewhere between a dozen and three dozen books transferred and calibre gives me an "error communicating with device" message and disconnects the NC. Usually doing another connect from CC is successful and I resume transfers, but occasionally I get persistent "cannot find calibre..." messages on CC and have to shut down and restart calibre to allow CC to find calibre.

I don't think there's any black magic in the NC, but it's a severely memory constrained device compared to the Galaxy Nexus. With CC's drastically improved speed, could it be blocking up too much in a single operation and blowing out the NC memory?

Calibre diagnostics are:

calibre, version 0.8.65
ERROR: Error: Error communicating with device

Device responded with incorrect information

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 83, in run
File "site-packages\calibre\gui2\device.py", line 478, in _upload_books
File "site-packages\calibre\devices\smart_device_app\driver.p y", line 44, in _synchronizer
File "site-packages\calibre\devices\smart_device_app\driver.p y", line 799, in upload_books
File "site-packages\calibre\devices\smart_device_app\driver.p y", line 458, in _put_file
File "site-packages\calibre\devices\smart_device_app\driver.p y", line 425, in _call_client
ControlError: Device responded with incorrect information

Thanks.

[chaley]

Originally Posted by ellett

So syncing with my Galaxy Nexus is fast and errors or few. I'm always looking for trouble so I started trying to use CC to sync with my rooted Nook Color, running CM7.

The nook color does indeed present challenges. It is at the top of the list for problem reports around lack of resources.

What is happening is that it is failing to allocate memory for a cover thumbnail. The NC isn't out of memory. It is complaining that we have exceeded our budget for bitmaps. Our problem is that we do nothing with the thumbnails except hand them off to the Android OS, so we are left with the probability that something inside android is leaking thumbnail memory. The oracle of google agrees, saying that before Honeycomb, releasing bitmaps was "unpredictable" and could lead to spurious out of memory issues. Eventually it catches up, which is why reconnecting will sometimes work. Of course, sometimes it doesn't catch up, which is why CC must be closed and reopened.

What we will try to do is catch and ignore the error, replacing the cover with an empty box. Unfortunately this might not be possible if the error is considered "fatal". We will see...

[chaley]

Originally Posted by ellett

So syncing with my Galaxy Nexus is fast and errors or few. I'm always looking for trouble so I started trying to use CC to sync with my rooted Nook Color, running CM7.

Today's release 1.8 has several changes that should (fingers crossed) help with this problem. In particular we no longer repaint covers or books during reception from calibre, saving the updates for when the transfer finishes. If/when you try the update, let us know if you see any improvements.

Thanks in advance.

[nremi]

I am having trouble transferring PDFs using CC. I get a timeout and disconnect... and I cannot tell what the source of the problem is. It appears that, after metadata is sent, and I use Calibre to Send to Device, it stops after a few seconds, CC still claims to be connected, but Calibre loses the device, and I have to cycle it on my Android device, while nothing is ever sent.

Otherwise, CC seems to work well for me, though I would like to raise another matter, regarding Calibre's news feeds: I have a few subscriptions that run daily, but on my mobile device, I have absolutely no interest in keeping anything but the latest day's publication. At the moment, I have five or so copies of each feed, which is more clutter than I can deal with... it is already overwhelming. I have a cap set in Calibre itself so that I can refer to a week's worth of things, but they do self-delete there. How might this syncing situation be improved to a manageable one?

[DoctorOhh]

Originally Posted by nremi

I am having trouble transferring PDFs using CC. I get a timeout and disconnect... and I cannot tell what the source of the problem is. It appears that, after metadata is sent, and I use Calibre to Send to Device, it stops after a few seconds, CC still claims to be connected, but Calibre loses the device, and I have to cycle it on my Android device, while nothing is ever sent.

In order to send PDF files, within CC you have to go to Options menu - Book format locations and check PDF. Before I checked PDF in the CC menu on my Nexus when I tried to send a PDF calibre converted it to epub prior to transfer to my device. Once I checked PDF on CC the PDF transferred directly.

[chaley]

Originally Posted by nremi

I am having trouble transferring PDFs using CC. I get a timeout and disconnect... and I cannot tell what the source of the problem is. It appears that, after metadata is sent, and I use Calibre to Send to Device, it stops after a few seconds, CC still claims to be connected, but Calibre loses the device, and I have to cycle it on my Android device, while nothing is ever sent.

Hmmm...

It isn't clear whether it is calibre or CC that is stopping. Questions:
1) Did you enable PDF in CC? If you did not, then calibre might be auto-converting and failing. It would say something about auto-converting, so I suspect that this isn't the problem.
2) Is it always the same PDF, or any PDF?
3) Can you send any other kind of format, such as epub?

It would be helpful if you got a calibre debug log. See the FAQ in the first post of this thread for instructions on how to do this. The debug log will tell us what calibre thinks is happening.

My suspicion is that things are going south during auto-convert (if you haven't enabled PDF) or during PDF metadata update (just before send). I have seen *many* reports of PDF metadata update problems. The usual answer is to disable the PDF metadata writer (preferences -> plugins -> Metadata writer plugins -> PDF metadata). You could try that to see if it helps you.

Otherwise, CC seems to work well for me, though I would like to raise another matter, regarding Calibre's news feeds: I have a few subscriptions that run daily, but on my mobile device, I have absolutely no interest in keeping anything but the latest day's publication. At the moment, I have five or so copies of each feed, which is more clutter than I can deal with... it is already overwhelming. I have a cap set in Calibre itself so that I can refer to a week's worth of things, but they do self-delete there. How might this syncing situation be improved to a manageable one?

I am not a news user, but my understanding is that calibre never auto-deletes news from the device. The articles just accumulate if you turn on auto-sync. If this is true, then there isn't anything CC can do about it. Perhaps someone who uses news can correct me if I am wrong.

You can delete the older news articles (ones calibre has auto-deleted from the library) fairly easily. Connect the device, then go to calibre's device view. Sort on the In-Library column. The old news articles will be unchecked. Look at the list (there might be other things in there), select the ones to delete, and press the Del key. It could also be that you can sort by title or author to get these articles grouped together.

We intend to add the ability to delete books directly in CC. Once this happens, things might become a bit easier for you.

[DoctorOhh]

Originally Posted by chaley

1) Did you enable PDF in CC? If you did not, then calibre might be auto-converting and failing. It would say something about auto-converting, so I suspect that this isn't the problem.

It isn't clear that you have to enable formats prior to being able to send certain books. I would suggest you also make PDF a default since so many folks seem to use this format.

It is not intuitive that Book format locations menu would be where you choose to select formats the device will allow. You might want to change the menu name to Book format list. This would suggest more than just locations.

Also I'm aware that in the Book format locations menu the help says you need to select a format for CC to send that format. Unless this information is elsewhere the location of this advice is down in the weeds. I saw two feedbacks of one star because they had no clue how to enable other formats.

Just my input.

[chaley]

Dwanthny, thanks for the comments and for the assistance.

Originally Posted by dwanthny

It isn't clear that you have to enable formats prior to being able to send certain books. I would suggest you also make PDF a default since so many folks seem to use this format.

You are right.

It costs nothing for me to add defaults. Is [epub, mobi, pdf] the right set? Should others be there?

It is not intuitive that Book format locations menu would be where you choose to select formats the device will allow. You might want to change the menu name to Book format list. This would suggest more than just locations.

Again, good suggestion. We debated over the name of that menu, but it was after a 12-hour day.

One concern is that without "locations" people won't know to go there for that. I am half tempted to have both of "Choose book folder locations" and "Select book formats" menu lines. They would both go to the same dialog.

Also I'm aware that in the Book format locations menu the help says you need to select a format for CC to send that format. Unless this information is elsewhere the location of this advice is down in the weeds. I saw two feedbacks of one star because they had no clue how to enable other formats.

What you said here was one trigger for using two menu lines. Of course, some people will puzzle over how they are different. I can live with that, but I am not sure Steven can. Thank goodness he is very picky about such things, because I am not.

Question (hijacking this conversation a bit): as far as I can tell we have emptied your feature request queue, possibly with the exception of requesting transfer of books from the device. Have I lost track of anything? Let me know, here or perhaps in a different thread.

Thanks again for your help. Steven and I very much appreciate it.

[kovidgoyal]

You'll want azw3 as well in the defaults.

[DoctorOhh]

Originally Posted by chaley

It costs nothing for me to add defaults. Is [epub, mobi, pdf] the right set? Should others be there?

If it truly doesn't cost anything why not have them all checked by default?

Originally Posted by chaley

I am half tempted to have both of "Choose book folder locations" and "Select book formats" menu lines. They would both go to the same dialog.

Seems reasonable, I like two specific items. An alternative could be a single more vague Book format options or Ebook options. Whichever way you go will be an improvement.

Originally Posted by chaley

Question (hijacking this conversation a bit): as far as I can tell we have emptied your feature request queue, possibly with the exception of requesting transfer of books from the device. Have I lost track of anything?

Looking through it seems you have met my short term requests.

One suggestion, if it is under your control it would be nice if the %All by author category was sorted by Author, by Series, by Title.

Longer term

I think giving folks font size control is always a plus for folks getting up in age. The current font size works well, it is just my personal preference to have control.

Adding the ability to view a book jacket representation of the cover and metadata would be nice.

Being able to access calibre content server or any OPDS catalog you wanted to add.

[Steve_Haley]

Originally Posted by dwanthny

If it truly doesn't cost anything why not have them all checked by default?

I think his reasoning had to do with which formats were supported by the most common book readers on Android. However, this may indeed have been a case of trying to save the user from making a mistake and instead creating more problems.

I've just enabled PDF and AZW3 as the defaults as well - before adding more, I'll check with Charles again. Unfortunately, this will only affect new users.

Originally Posted by cbhaley

Of course, some people will puzzle over how they are different. I can live with that, but I am not sure Steven can.

Originally Posted by dwanthny

Seems reasonable, I like two specific items. An alternative could be a single more vague Book format options or Ebook options. Whichever way you go will be an improvement.

Ha I haven't been happy with that name either. But yes, I would not be happy with two menu options which take you to the same place. I've opted for "Book format options".

[chaley]

Originally Posted by dwanthny

If it truly doesn't cost anything why not have them all checked by default?

To follow on from what Steven said, I didn't want to be in a position where calibre sent files that no reader on the device can handle. Discovering that the problem exists can be problematic because no messages are produced. On the other hand, calibre pops up an "auto-convert" dialog if the formats are not supported, thus telling the user that a problem exists and (one hopes) giving some motivation for finding a solution.

Looking through it seems you have met my short term requests.

One suggestion, if it is under your control it would be nice if the %All by author category was sorted by Author, by Series, by Title.

If by "under my control" you mean under CC's control, then no, it isn't possible. Calibre does not provide the sorting criteria for collections.

I could change calibre's collection builder to make the default sort for aba to be what you suggest, but the idea scares me a bit. I don't know how many people depend on the current order.

You could change it by specifying sort criteria when you build the collection. See the Sony collections tweaks for that. I suspect you would need to build a composite custom column containing the sort string you want to use.

Longer term

I think giving folks font size control is always a plus for folks getting up in age. The current font size works well, it is just my personal preference to have control.

Steven and I haven't discussed this one, but I think you are right. I just added it to our todo list.

Adding the ability to view a book jacket representation of the cover and metadata would be nice.

This has percolated to near the top of our list.

Being able to access calibre content server or any OPDS catalog you wanted to add.

An OPDS client is a good idea. We will look at that when we start looking at how to use CC to get books from calibre instead of being only a passive client.

[DoctorOhh]

Originally Posted by chaley

You could change it by specifying sort criteria when you build the collection. See the Sony collections tweaks for that. I suspect you would need to build a composite custom column containing the sort string you want to use.

I think I had this set up previously but removed the custom columns when I stopped using it. I know where to find the discussion on this so I'll take care of it myself.

So the Sony tweaks apply to the Smart device?

I did create the custom column, changed the sort tweak as required and it has no affect on the All by authors on my Nexus. I even removed all books and resent all the books. This isn't a big thing, just trying to get it straight in my head.

[nremi]

I must say thanks for your rapid response.

That pdfs were not permitted, as it were, was the problem. On the other hand, the set-up is confusing to me, because I did get the conversion request, but I kept on saying 'no' to it, because I knew that I wanted, and could read, pdfs on my device--so that could use some tweaking. Personally, I'd rather have the error pop up on my Android thing, because then I know what is going on ("No reader available"), as opposed to the messages I got, which led me nowhere, despite my prodding around on both ends.

As for the news building up, Calibre definitely has the option of deleting things after x days. ("Schedule news download", at the bottom of that option dialog box, when you click the menu under "Fetch news".) It seems beneficial for this process to be automated, given its transient nature.

As for wanting more than one month's reading, a possible rationale would be reference books, such as cookbooks, or repair guides. I know, "The internet does this," but these things still serve a purpose.

Speaking of conversion... I mistakenly sent a converted file over to my device, and then the (original) format I wanted, and now I have two otherwise identical objects in the CC list with no ready way to differentiate the two. I would like to request some way to tell these files apart within CC, at least until I can delete them--but which one is the original, and which is the evil twin?

[chaley]

Originally Posted by nremi

As for the news building up, Calibre definitely has the option of deleting things after x days. ("Schedule news download", at the bottom of that option dialog box, when you click the menu under "Fetch news".) It seems beneficial for this process to be automated, given its transient nature.

That option causes news to be deleted from the library, not the device. The device (in our case Calibre Companion) is never told to delete anything, so we don't.

Calibre itself would need to change to make auto-delete off the device possible. My guess is that such a change would not be easy, given the option to delete news from the library on send that clearly could not delete the news from the device, but that isn't an area I have ever worked in.