Calibre Companion prerelease 3.3.0f

[chaley]

CC prerelease 3.3.0f has been published and will available soon to people who are members of the CC prerelease program.

This prerelease contains the following enhancements and bug fixes over CC prerelease 3.3.0e:

• Add a "Read" mark on the right side of the book list. It shows only if the option "Show Read information" is checked and if the Read info is not empty. This is an experiment. I might remove it for the production release.
• Attempt to handle read-only folders such as those on external SD cards on devices running kitkat. In the folder selector, tell the user if s/he has chosen a read-only folder.
• Add some spacing to the "Groups" item in the grouping drawer
• Refuse to sync Read info if calibre is not at V1.28 or higher
• [BUG FIX] Several changes to Read info syncing to prevent data loss when sending new books from calibre or scanning for new books
• [BUG FIX] Always display Title over Author in the book list when sorting by dates
• [BUG FIX] Honor the calibre date display formats when showing pubdate or calibre's date field in the book list.

If you have already installed CC 3.3.0e then this prerelease does not require a database upgrade.

As noted above, syncing Read info requires calibre 1.28 or better.

The changes to handing the read-only folders were rather invasive, touching basic code to write books and metadata. Although I have tested the changes, I would appreciate it if testers would put books onto their device using both the content server and wireless device to give further assurance. If someone is adventurous, then testing changing CC's book storage to a read-only folder would be nice. Especially nice would be if someone running kitkat and having an external SD card would try to put CC's storage on the external card. But all this aside, simply using CC will be very helpful.

[PatNY]

I've finally had a chance to fully test the new "Read" feature and everything is working perfectly. I made a roundtrip sync, changing values at each stop, from the Nook HD (KitKat) to Calibre and then to my Nook HD+ (ICS) and then back to Calibre, and finally back to the Nook HD. All the new values registered correctly at each stop.

I just have a question regarding the sorting of books in CC. I usually choose to sort by "Date Updated (CC)" with the most recently changed books at the top. I've noticed in this recent round of syncs using Wireless, every book on both devices was given the same new "Updated" date of today. So now they are all randomly mixed up. I gather this was necessary due to the new "Read" feature and database structure? Is there a way to keep the original "Update" dates in the future for books that don't change even when you have to add a new database feature?

--Pat

[jackie_w]

I've just tried this new version and here are my comments:

The 'Read' mark in the Book List was unexpected for 3.3 but it looks great and seems to work OK, I'm not sure why you'd want to remove it

I'm afraid I can't help with KitKat and I'm not sure how I'd create a read-only directory in Android 4.3.

I'm glad to see all my Published Dates are Year only, as per calibre prefs. I wonder how many people are interested in day/month details for pubdate?

Re: Book Details page. I don't know whether this behaviour was introduced in this version, or whether I just didn't notice before. If I accidentally tap the screen in white space to the right of one of the hyperlink fields (author, publisher, series etc) then it's as if I actually tapped the hyperlink itself. So, on a phone with almost non-existent side bezels it's quite easy to be whisked off to a subset selection of books by accident. The book I was currently reading the Details of is guaranteed to be one the subset so it wasn't too difficult to get back again, but it may be worse for those with a lot more CC books than I have.

[chaley]

Originally Posted by jackie_w

I've just tried this new version and here are my comments:

The 'Read' mark in the Book List was unexpected for 3.3 but it looks great and seems to work OK, I'm not sure why you'd want to remove it

Two reasons: performance and "look" on 10 inch devices (which I don't have one of). The first comes from the fact that I had to add a touch of time to the DB lookup when generating the book list in order to get the read status. Someone who has a huge library might see this. The second is one of those "artistic" issues that I am not good at.

I'm afraid I can't help with KitKat and I'm not sure how I'd create a read-only directory in Android 4.3.

I'm glad to see all my Published Dates are Year only, as per calibre prefs. I wonder how many people are interested in day/month details for pubdate?

A fair amount of people using calibre want month. Nobody wants day or time of day.

Re: Book Details page. I don't know whether this behaviour was introduced in this version, or whether I just didn't notice before. If I accidentally tap the screen in white space to the right of one of the hyperlink fields (author, publisher, series etc) then it's as if I actually tapped the hyperlink itself. So, on a phone with almost non-existent side bezels it's quite easy to be whisked off to a subset selection of books by accident. The book I was currently reading the Details of is guaranteed to be one the subset so it wasn't too difficult to get back again, but it may be worse for those with a lot more CC books than I have.

I didn't change anything related to tapping, so it must have been there. I can repeat this behavior. It seems to happen only for the last line, not word-wrapped lines. My guess is that the "bounding box" of the last text link is being extended to the edge of the box, and I have an idea for preventing that.

EDIT: My theory seems to be correct. I have fixed it.

[chaley]

Originally Posted by PatNY

I've finally had a chance to fully test the new "Read" feature and everything is working perfectly. I made a roundtrip sync, changing values at each stop, from the Nook HD (KitKat) to Calibre and then to my Nook HD+ (ICS) and then back to Calibre, and finally back to the Nook HD. All the new values registered correctly at each stop.

Thanks for this, especially the multi-device tests.

I just have a question regarding the sorting of books in CC. I usually choose to sort by "Date Updated (CC)" with the most recently changed books at the top. I've noticed in this recent round of syncs using Wireless, every book on both devices was given the same new "Updated" date of today. So now they are all randomly mixed up. I gather this was necessary due to the new "Read" feature and database structure? Is there a way to keep the original "Update" dates in the future for books that don't change even when you have to add a new database feature?

--Pat

This is an excellent point, one I am surprised has never come up before. As it is, any time the DB is rebuilt (upgrades because of new versions or changing the custom columns shown in the grouping drawer), the CC changed date gets updated. As you note, this is simply wrong. The fix is easy and risk-free, so I did it. Now the date is changed only when new metadata arrives from calibre.

[PatNY]

Originally Posted by chaley

The fix is easy and risk-free, so I did it. Now the date is changed only when new metadata arrives from calibre.

Great. Thanks for this. I did notice it before, but was never sure of the cause and thought it might have been something to do with how I was working with and changing the data in Calibre on the PC.

--Pat

[kaufman]

Just wanted to add that the grouping drawer looks perfect now. Thanks for spreading things out.

[kaufman]

This is not new to 3.3.0f, but I have noticed an interesting issue that might be a bug. I have a column that uses the values in other columns to determine it's value. If I change it, the next time I connect, it sends over the metadata for every book. This is expected because even though my change might not change every value, Calibre marks them all as updated. The issue is that it also sends it all over on the following connection as well.

[chaley]

Originally Posted by kaufman

This is not new to 3.3.0f, but I have noticed an interesting issue that might be a bug. I have a column that uses the values in other columns to determine it's value. If I change it, the next time I connect, it sends over the metadata for every book. This is expected because even though my change might not change every value, Calibre marks them all as updated. The issue is that it also sends it all over on the following connection as well.

I don't see the second-send behavior.

My experiment:

1. Connect as wireless device to ensure that all metadata is up to date. Repeat until no metadata is sent (usually zero times)
2. Disconnect
3. Update a composite custom column. I simply opened the editor (clicked on the composite column and hit F2) and pressed OK.
4. Connect. Metadata for all books is sent.
5. Disconnect
6. Connect. No metadata is sent.

Are you doing something different? Do you have any plugins installed that might touch the last_mod date?


One thought: If you are editing the template of the composite column in preferences/add your own columns then the last_mod doesn't get updated until calibre is restarted. The metadata will therefore all be resent after the restart. Is that consistent with what you are doing and seeing?

[kaufman]

Originally Posted by chaley

I don't see the second-send behavior.

My experiment:

1. ...

Are you doing something different? Do you have any plugins installed that might touch the last_mod date?

Nope, that's pretty much exactly what I am doing.

Originally Posted by chaley

One thought: If you are editing the template of the composite column in preferences/add your own columns then the last_mod doesn't get updated until calibre is restarted. The metadata will therefore all be resent after the restart. Is that consistent with what you are doing and seeing?

I always restart after change the definition of the column.

I don't know if this is relevant, but the last time I connected, I added six books to my library. I then quit calibre and CC. When I just reconnected, it sent the metadata for 6 books. I hadn't thought about it before, but it seems like each time I connect, it is resending the books from the last connection. I just updated the metadata for one book and resent it over. I then closed both apps and reopened and it sent over one book's metadata.

I never really noticed it before, because its only noticeable when I have to send everything.

[chaley]

Originally Posted by kaufman

I don't know if this is relevant, but the last time I connected, I added six books to my library. I then quit calibre and CC. When I just reconnected, it sent the metadata for 6 books. I hadn't thought about it before, but it seems like each time I connect, it is resending the books from the last connection. I just updated the metadata for one book and resent it over. I then closed both apps and reopened and it sent over one book's metadata.

I never really noticed it before, because its only noticeable when I have to send everything.

Aha! I think I know what you are doing, and it does trigger a bug. Are you closing calibre while CC is connected? If so, then the metadata cache file is not written back to the disk, meaning that the calibre driver is not up-to-date with what is on the device. On the next connect, the device will upload the missing/changed metadata, which might trigger metadata downloading.

I will fix this in calibre, making it write the cache if the disconnect is triggered by a shutdown.

[kaufman]

Originally Posted by chaley

Aha! I think I know what you are doing, and it does trigger a bug. Are you closing calibre while CC is connected?

Bingo. I just updated and transferred one book. Then I did a real Disconnect from CC and then exited both programs. I then restarted both and reconnected. No information was resent.

Thanks for figured out what I was doing wrong. That was really annoying. I'll start closing the right away until the fix is in.

[chaley]

Originally Posted by kaufman

Bingo. I just updated and transferred one book. Then I did a real Disconnect from CC and then exited both programs. I then restarted both and reconnected. No information was resent.

Thanks for figured out what I was doing wrong. That was really annoying. I'll start closing the right away until the fix is in.

Glad to have confirmation.

The fix has been submitted to Kovid and will be in next Friday's calibre release.

BTW: I find it easier to disconnect by using "Eject" from calibre (click the arrow on the Device toolbar icon and choose "Eject") than to use CC's disconnect. I am usually at calibre while I am connected so the two clicks aren't any trouble.

[kaufman]

Originally Posted by chaley

BTW: I find it easier to disconnect by using "Eject" from calibre (click the arrow on the Device toolbar icon and choose "Eject") than to use CC's disconnect. I am usually at calibre while I am connected so the two clicks aren't any trouble.

This information has been filed under the category of "you learn something new every day". Thanks.

[Purple Lady]

I created a new column in Calibre then connected wirelessly with my Nexus 7 and got an error while Calibre was sending metadata to the device. I sent the CC debug log.

calibre, version 1.28.0
ERROR: Error: Error communicating with device

Device closed the network connection

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 86, in run
File "site-packages\calibre\gui2\device.py", line 520, in _sync_booklists
File "site-packages\calibre\devices\smart_device_app\driver.p y", line 50, in _synchronizer
File "site-packages\calibre\devices\smart_device_app\driver.p y", line 1326, in sync_booklists
File "site-packages\calibre\devices\smart_device_app\driver.p y", line 50, in _synchronizer
File "site-packages\calibre\devices\smart_device_app\driver.p y", line 607, in _call_client
ControlError: Device closed the network connection

ETA: I connected again and it was successful.