Calibre Companion prerelease 3.2.0a

[chaley]

CC prerelease 3.2.0a has been published and will available soon to people who are members of the CC prerelease program.

This prerelease has the following changes:

• Uses a book cache on the calibre side to avoid sending metadata from CC to calibre during wireless device connections, reducing initial connection time by up to a factor of 10. This change requires calibre 1.5 released 26/9/2013. Note that you will not see the improvement until after a successful connection. CC will continue to work with older versions of calibre, offering no performance improvement.
• Provides a "Download all" menu item in the content server, permitting you to download all the books in a category such as all books by an author or a series. This change requires calibre 1.5 released 26/9/2013. If calibre has not been updated, this function should do nothing.
• [BUG FIX] Don't incorrectly erase the current grouping specification from the title bar.

[jackie_w]

I tried items 2 & 3 (with calibre 1.5). Both seem fine on my phone. I don't really have enough books on the phone to notice a performance improvement with item 1.

P.S. My red cover thumbnails are now looking much better, so thanks for that

[Dhevi]

I also don't have a large enough library on my devices to make notice of item 1. Haven't as of yet run into any issues, but will keep you up-to-date.

[Purple Lady]

Wow! The connect time for wireless is great now! I only have 430 books, but I am not a very patient person so I had changed to use the content server for my news feeds. I will go back to the wireless connection now and set to auto download in Calibre.

The grouping name looks good now too - I thought something was wrong before, but figured I messed up something myself.

[DoctorOhh]

Amazing, previously it was around 5 minutes to get the book list. I connected 5 separate times with 10,000 books and the connect times are attached.

3 from above is fixed on my device.

[DoctorOhh]

FYI the below error happens running calibre portable without Dropbox even running on the computer. Also I have not been able to repeat the problem with Calibre 64. Both instances run on the same box utilizing the same library (not at the same time).

I'm having some problems with the smartdevice cache when using calibre portable. After closing and restarting calibre portable I get the following error and the cache needs to do a complete metadata rebuild from CC to calibre.

calibre, version 1.5.0
ERROR: Problem starting the wireless device: The wireless device driver did not start. It said "start smartdevice unknown exception"

When I look, it shows it as being started.

I decided to start fresh and, with calibre closed, I deleted the device_drivers_SMART_DEVICE_APP_metadata_cache.pic kle file from calibre portable.

I then started up calibre portable w/o error (run from local C: drive). I connected CC as a wireless device and let it rebuild the cache. I disconnected and reconnected several times with connect times as expected.

I exited calibre portable then re-entered and everything was fine. I connected and disconnected a few more times without a hitch.

I exited calibre portable then re-entered and got the same error

calibre, version 1.5.0
ERROR: Problem starting the wireless device: The wireless device driver did not start. It said "start smartdevice unknown exception"

When I look it shows it as being started.

After receiving this error, when I connected, the cache has to be sent in total. Once the cache is rebuilt I can disconnect and reconnect as often as I like without error, but restart calibre and the error and rebuilding the cache happens every time until I delete the pickle and rebuild fresh.

I started fresh and limited the connection to once per calibre portable start and was able to go 6 straight restarts without an error. The error seems to pop up only after multiple connects and disconnects followed by restarting calibre.

Additionally when I delete the pickle it is either 163M or 336M in size, but the one for the calibre 64 version is 506M and the 506M version works fine with calibre portable.

It is always possible that it may be my machine. I'll try calibre portable on my work machine tonight and see if the same thing occurs.

[chaley]

@DoctorOhh: my guess is calibre is hitting an out-of-memory problem, because the portable edition is 32 bit. This would be consistent with the cache file sizes you report if the error is happening while the cache is being written.

I need to rethink how to write the cache. The current system makes a copy of the cache then writes it to the disk. I probably need to find a way to write it book-by-book.

Any fixes for this will be in calibre, so unless you are running from source (and IIRC you are not), experiments will be tied to calibre releases.

[DoctorOhh]

Running out of memory seems like a reasonable guess. I work 12 hour days through Wednesday, I don't run from source, but maybe I'll check into setting that up later on in the week.

[chaley]

Originally Posted by DoctorOhh

Running out of memory seems like a reasonable guess. I work 12 hour days through Wednesday, I don't run from source, but maybe I'll check into setting that up later on in the week.

No hurry. Recovering after such long days certainly takes priority.

[Sefiriot]

chaley, I've upgraded my phone to Android Jelly Bean 4.1.2 with the latest prerelease of CC. Having decided to make a fresh start with CC, I deleted all books on the phone and reinstalled CC.

I'm currently confused about what folders I should be pointing CC to. As far as I can tell, Jelly Bean now has the USB storage (internal storage) as /storage/sdcard0, and my external SD card as /storage/extSdCard. The confusing part is that in ES Explorer I can still see /sdcard0 as /sdcard, which also has a folder /storage/sdcard0/external_sd.

I first set CC's default folder to /storage/extSdCard/Books and further set it up so my epubs go into /storage/extSdCard/Books/ePub. I then connected CC to calibre, only to get this error:

calibre, version 1.5.0
ERROR: Error: Error communicating with device

u'free_space_on_device'

Traceback (most recent call last):
  File "site-packages\calibre\gui2\device.py", line 85, in run
  File "site-packages\calibre\gui2\device.py", line 511, in _sync_booklists
  File "site-packages\calibre\devices\smart_device_app\driver.py", line 50, in _synchronizer
  File "site-packages\calibre\devices\smart_device_app\driver.py", line 1064, in free_space
KeyError: u'free_space_on_device'

Where should I be pointing the folders to? It looks like a lack of free space issue, but that can't be it, because the external card is totally empty. USB storage also has around 9GB free.

I've also tried /storage/sdcard0/external_sd/Books as default folder and /storage/sdcard0/external_sd/Books/ePub for the ePubs, and get the same error as above. calibre side log included here, let me know if you need to see the CC log as well.

ETA: I did some reading on the Android system, figured out what was going on with the symlinks and where to point CC to to get my books on the external card. Since /mnt/sdcard and /storage/sdcard0/ are apparently now the same place, I made sure CC's default folder was set to /storage/extSdCard/Books, with the ePub folder in it as ../Books/ePub.

Connecting as wireless device with no books in CC triggers the error I reported earlier, but downloading a single test book through the content server put the book right where it's supposed to be, and no errors occur.

I then connected again as Wireless Device, and everything's working again as normal, no errors. Sent 200 books over to the device with no problems via Reading List plugin. Only change was whether or not there's a book in CC's folders that was already in calibre's library. I deleted all my books off the device and repeated the experiment, with the same result. I'll send you the debug logs.

[chaley]

@Sefiriot: thanks. There was a protocol problem in the case where no books were available to send to calibre. It will be fixed in the next CC prerelease.

[Sefiriot]

@chaley: Ah thanks. I was starting to wonder if I'd screwed something up during installation and setup. I doubt you still want my logs though, since the issue's located? Meanwhile I'm seeing some very nice speeds on metadata transfers. Excellent job there.

For anyone else encountering the same issue right now, the error in those cases doesn't prevent CC from sending books to the device, it just stops Calibre from sending the metadata after the books transfer. In any case setting the "scan folders" option will get you the proper metadata when you next connect, and then CC and Calibre will work as usual.