Calibre Companion prerelease 3.3.0f

[Purple Lady]

Originally Posted by Purple Lady

I created a new column in Calibre then connected wirelessly with my Nexus 7 and got an error while Calibre was sending metadata to the device. I sent the CC debug log.

calibre, version 1.28.0
ERROR: Error: Error communicating with device

Device closed the network connection

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 86, in run
File "site-packages\calibre\gui2\device.py", line 520, in _sync_booklists
File "site-packages\calibre\devices\smart_device_app\driver.p y", line 50, in _synchronizer
File "site-packages\calibre\devices\smart_device_app\driver.p y", line 1326, in sync_booklists
File "site-packages\calibre\devices\smart_device_app\driver.p y", line 50, in _synchronizer
File "site-packages\calibre\devices\smart_device_app\driver.p y", line 607, in _call_client
ControlError: Device closed the network connection

ETA: I connected again and it was successful.

I disconnected after all metadata was received by CC and then updated the read status on CC for a few books. When I connected wirelessly this time it sent all the metadata again.

Never mind, I had updated the data in my new column so of course it had to send the metadata for all the books again.

[Purple Lady]

There is a problem with the save template when connecting as a wireless device. I have a few anthologies and I have all the authors in the authors column. I tried the save template in both the save to disk and when connected by usb in Calibre and it worked fine. When connecting wirelessly, CC is still using all authors instead of only the first - see screen print. Here's the save template I am using:

{#genre}/{author_sort:list_item(0,&)}/{title} - {author:list_item(0,&)}

[Purple Lady]

Here's the screen print for the previous post. Tapatalk wouldn't let me edit the post and add it, but it let me attach to a new post.

[eschwartz]

Originally Posted by Purple Lady

There is a problem with the save template when connecting as a wireless device. I have a few anthologies and I have all the authors in the authors column. I tried the save template in both the save to disk and when connected by usb in Calibre and it worked fine. When connecting wirelessly, CC is still using all authors instead of only the first - see screen print. Here's the save template I am using:

{#genre}/{author_sort:list_item(0,&)}/{title} - {author:list_item(0,&)}

list_sort returns a case-insensitive sorted complete list. I think what you want is

{author_sort:sublist(0,1,&)}

using the sublist function.

[chaley]

Originally Posted by Purple Lady

I created a new column in Calibre then connected wirelessly with my Nexus 7 and got an error while Calibre was sending metadata to the device. I sent the CC debug log.

calibre, version 1.28.0
ERROR: Error: Error communicating with device

Device closed the network connection

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 86, in run
File "site-packages\calibre\gui2\device.py", line 520, in _sync_booklists
File "site-packages\calibre\devices\smart_device_app\driver.p y", line 50, in _synchronizer
File "site-packages\calibre\devices\smart_device_app\driver.p y", line 1326, in sync_booklists
File "site-packages\calibre\devices\smart_device_app\driver.p y", line 50, in _synchronizer
File "site-packages\calibre\devices\smart_device_app\driver.p y", line 607, in _call_client
ControlError: Device closed the network connection

ETA: I connected again and it was successful.

From what I can see from the logs, the original problem was caused by WiFi disconnecting then reconnecting. Is it possible that you had a "network event" where this might have happened?

[chaley]

Originally Posted by Purple Lady

There is a problem with the save template when connecting as a wireless device. I have a few anthologies and I have all the authors in the authors column. I tried the save template in both the save to disk and when connected by usb in Calibre and it worked fine. When connecting wirelessly, CC is still using all authors instead of only the first - see screen print. Here's the save template I am using:

{#genre}/{author_sort:list_item(0,&)}/{title} - {author:list_item(0,&)}

Be sure that the template is really changed in the smart device driver. To see what the driver is using, while connected, right-click on the device toolbar icon and choose "Configure this device".

@eschwartz: Purple Lady is using list_item, not list_sort. List_item returns the nth item.

[Purple Lady]

Originally Posted by chaley

Be sure that the template is really changed in the smart device driver. To see what the driver is using, while connected, right-click on the device toolbar icon and choose "Configure this device".

@eschwartz: Purple Lady is using list_item, not list_sort. List_item returns the nth item.

It's there:
{#genre}/{author_sort:list_item(0,&)}/{title} - {author:list_item(0,&)}

[chaley]

Originally Posted by Purple Lady

It's there:
{#genre}/{author_sort:list_item(0,&)}/{title} - {author:list_item(0,&)}

Did you change the template? If you do, the old value will be used for all existing books so that we don't get duplicates on the device. Try sending a book that has never been on the device and see what values are used. Alternatively, delete a book off the device and resend it.

[Purple Lady]

Originally Posted by chaley

Did you change the template? If you do, the old value will be used for all existing books so that we don't get duplicates on the device. Try sending a book that has never been on the device and see what values are used. Alternatively, delete a book off the device and resend it.

I changed the template and did a delete and resend. When that didn't work I restarted CC and did a delete and resend again.

I had actually deleted all books from my device using a file manager and then did a check for missing books in CC and then deleted missing books. When I saw the template didn't work, I connected by usb and sent all the books that way. After that was complete I deleted one of the books and sent it again.

[Purple Lady]

Originally Posted by chaley

From what I can see from the logs, the original problem was caused by WiFi disconnecting then reconnecting. Is it possible that you had a "network event" where this might have happened?

I guess that is possible - how would I check? I was sitting at the computer when it happened, but didn't notice anything.

[chaley]

Originally Posted by Purple Lady

I guess that is possible - how would I check? I was sitting at the computer when it happened, but didn't notice anything.

If you didn't notice it at the time, there is no way to know.

The CC debug log says that calibre asked for metadata but then never offered to send metadata back. CC waited almost 5 minutes, then disconnected. I am not sure whether that was a manual disconnect or if something timed out. The calibre exception says that calibre wanted to send metadata back to CC but the network connection to CC was gone. Without the rest of the calibre debug log (which of course you don't have), I can't tell more than that.

One thing I do is always run calibre in debug mode so that if something strange happens I have a full calibre debug log. I start calibre with a .bat file with the command "calibre-debug -g > c:\calibre_debug_log.txt" (no quotes). If something happens then I grab the debug log file before I restart calibre again. A bit of a bother, but it has helped me find rare bugs.

[chaley]

Originally Posted by Purple Lady

I changed the template and did a delete and resend. When that didn't work I restarted CC and did a delete and resend again.

I had actually deleted all books from my device using a file manager and then did a check for missing books in CC and then deleted missing books. When I saw the template didn't work, I connected by usb and sent all the books that way. After that was complete I deleted one of the books and sent it again.

OK, I know what is going on. Metadata caching is too aggressive, keeping information for books that are no longer on the device. Deleting them then sending them again finds the old cache entry and thus uses the old file path. The only books that would get the new path are brand new books that have never before been sent to CC. The problem is not in CC but in calibre, and has been there since December. I have a fix and am testing it.

You can work around the problem by:

1. Deleting all books off the device. What you did (file manager delete followed by find missing books) is a very efficient way of doing that.
2. Make sure calibre is not running.
3. Delete the calibre metadata cache files. They should be in c:\users\your-user-name\AppData\Local\calibre-cache, named wireless_device_long-string-of-nonsense. Delete all of them
4. Connect CC to calibre
5. Send the books. Perhaps test with a few to be sure that something else isn't going on.

Thanks for your patience!

[Purple Lady]

Originally Posted by chaley

OK, I know what is going on. Metadata caching is too aggressive, keeping information for books that are no longer on the device. Deleting them then sending them again finds the old cache entry and thus uses the old file path. The only books that would get the new path are brand new books that have never before been sent to CC. The problem is not in CC but in calibre, and has been there since December. I have a fix and am testing it.

You can work around the problem by:

1. Deleting all books off the device. What you did (file manager delete followed by find missing books) is a very efficient way of doing that.
2. Make sure calibre is not running.
3. Delete the calibre metadata cache files. They should be in c:\users\your-user-name\AppData\Local\calibre-cache, named wireless_device_long-string-of-nonsense. Delete all of them
4. Connect CC to calibre
5. Send the books. Perhaps test with a few to be sure that something else isn't going on.

Thanks for your patience!

Since I have all my books in CC now I didn't want to delete again. So I

1. deleted Calibre cache files after closing Calibre.
2. deleted a book from my device using CC.
3. started Calibre
4. connected to Calibre wirelessly - all metadata was set to Calibre again
5. sent the book again
6. profit!

All went well. After I deleted all my books last night, it seemed that sending 20 books thru the wireless connection took a while. I sent them by connecting by usb instead and it was a lot quicker. When I connected wirelessly after that, CC did need to send the metadata again of course, but overall it seemed much quicker. Is it my imagination or is usb a lot quicker? I keep about 800 books on my tablet.

[eschwartz]

Originally Posted by chaley

@eschwartz: Purple Lady is using list_item, not list_sort. List_item returns the nth item.

Whoops! The peril of doing anything at 43 minutes past midnight. I don't think it helped that it was Purim yesterday, either.

I saw "author_sort:list_item" and mentally switched it around.

[chaley]

Originally Posted by Purple Lady

Is it my imagination or is usb a lot quicker? I keep about 800 books on my tablet.

USB transfers data much faster than wifi, usually around 8 times faster. If your wifi is congested (neighbors, distance, etc.) or stuck at an older max speed (11 mbit) than USB can win even more. If you are using USB 1 (I would be very surprised if you were) then WiFi wins.

In addition to base wire speeds, when calibre transfers files by USB, that is all that it does. It does not maintain a database on the device. CC does maintain a DB, and writing that DB is much slower than the network. CC can write up to 7 or 8 books per second on my galaxy nexus, but that drops to 1 to 3 once the DB grows.

Finally, I expect that the total time required to use USB to transfer the books then have CC scan and detect them is longer than it would have been simply to transfer them with CC. First you must transfer the books. Next, start CC and let it scan. The scan process must open every book to get the metadata, write this intermediate metadata to the DB, send that metadata to calibre, receive back the correct metadata, then write that to the DB. This is much more work than when you send a book; that requires one write to the DB, writing the book file, and no opening of the book file. The network time to transfer and write a book is overlaid with the time to write the DB, so in many cases the network time becomes negligible.