CC V5.0.1 beta channel release

[chaley]

V5.0.1.1 Beta 1

This is the V5 beta release following the V5 alpha test.

Change log from V4, in no particular order:

• Dropped support for Gingerbread and Honeycomb (Android 2 and 3). See below for more information.
• Changed to newest Google tools and libraries. This change touches most of CC's user interface, usually is subtle ways, because the new tools try to enforce some "Material Design" guidelines. For example, the action bar is a bit different, dialog buttons have moved around a bit, and some animations have come or gone. It is this change that introduces the most risk that I broke something.
• Use Android's Storage Access Framework (SAF) when CC is running on Android 5 (Lollipop) or higher. This permits storing your library in arbitrary folders on the SD card. When you go to Settings / Formats and Folders and tap an extension or the default line, CC opens an Android folder chooser instead of the existing one. Note that I have no control at all over that chooser dialog.
  
  The app-private-folder hack is now only necessary on Android 4.4 (Kitkat).
  
  The tools and library changes were necessary to make this work.
• Changed first-letter grouping by dates to use decade and century ranges. If you group by first letter on dates, in order to see the new partitions you must rebuild the grouping information. The easiest/quickest way to do that is to go to Settings / Grouping Drawer / Group/sort on custom columns then uncheck something (anything), tap OK, go back into the dialog, check what you unchecked, then OK. At this point CC will do a complete rebuild. You can "Back" out of Settings then wait for the process to finish (the icon in the notification area goes away and Grouping becomes enabled again).
• Add an option in Settings / File names to remove accents (diacritical marks) on characters used in path names. This does not "ascii-fy" the path. It just removes accents. Exactly what is an accent is defined by the ICU (International Components for Unicode) consortium.
• Change templates to do what calibre does for several non-alphanumeric characters. See below for more information.
• Add an option to rename book files if a book with a different filename (path) is being downloaded
• The settings screeen(s) have reverted back to the "standard" for the android theme.
• Add a transition animation from the book list to book details. Don't do the animation if e-ink corrections are enabled. Add an option in Settings / Book details to disable the animation.
• Move the cloud connection database cache out of the library, instead storing it in the app private folder on the SD card if there is one, otherwise in main memory. This was required because of the change to use the SAF.
• Handle Android 6 asking for permissions at run time instead of install time.
• Add Android 6 auto-backup support. I have no idea if it works at all. It doesn't on my Nexus 5. There is a lot of chatter saying that one must do a system reinit after an upgrade to make it work and I am unwilling to do that on my "production" phone.
  
  If it works then it will backup CC's settings. The database is too large.
• CC now collects non-personal usage statistics. The first time you start CC after the upgrade you will see a dialog mentioning this and explaining how to opt out (I hope you don't). See our Privacy Policy for more information.
• Add some space between the Read and Delete buttons in Book Details
• Reorganize the Settings / File name preferences screen to put Use templates ... on top.
• Use CC's notification when receiving books over the wireless connection, in addition to the existing uses (database upgrade and bulk download in the cloud and content server).
• Support for the new calibre content server. This server is not generally available. My main concern at the moment is that I don't break anything when using the current calibre server.
• Updated languages: French, Russian, Spanish, Vietnamese.
• [BUG FIX]Don't use a "left over" template as the default if the "Use templates ..." setting is unchecked, but preserve such a template in case the option is rechecked later.
• [BUG FIX]Make CC really able to open azw3 files with FBReader
• [BUG FIX]Don't generate invalid file names if the length of the result of a template exceeds the maximum.
• [BUG FIX]Fix crash in Android M (6) when long-pressing on a book in the content server and cloud lists
• [BUG FIX]Make on-device work in the cloud connection work when a book is downloaded in book details
• [BUG FIX]Fix rare crash when adding books with strange internal metadata.
• [BUG FIX]Yet another try to make a selected language "stick".
• [BUG FIX]Make searching on custom comments (long text) fields work.

Change log from V5.0.0.7 Alpha 7

• Remove superfluous quotes from some strings
• Updated languages: French, Spanish, Vietnamese
• [BUG FIX]Make searching on custom comments work. Ignore html in comments when searching.

Comments:

• Re dropping support for Android 2 and 3: CC version 4 will remain on Google Play and will be offered to the these devices. Devices running Android 4 and higher will automatically be offered CC version 5. I don't intend to continue development of V4, but I will fix egregious bugs.
• Re non-alphanumeric characters in file names: CC now matches calibre's behavior in the following ways:
  • CC changes the following characters to underscores -- characters that are not:
    letters, numbers, or a space
    !, $, %, ^, &, (, ), =, -, {, }, \, [, ], @, ~, ', #, comma, period, _, and /.
  • CC next changes any leading period to an underscore and removes all trailing periods.
  • Finally, CC removes any empty path segments, e.g., aaa//bbb becomes aaa/bbb.
• Re rename book files: By default and since forever, when sending a book over the wireless device that already exists in CC's library, calibre + CC reused the old path, ignoring the results of the template. This means that if you changed a book's author from AAA to BBB and then resent the book, the file name would still contain AAA. If you downloaded a book that already exists via the content server or the cloud a similar thing happened. The book file was not renamed. This is good behavior in that it prevented creation of duplicate files. The bad behavior is that the file names might not match the metadata.
  
  When templates were introduced this behavior changed. Using the above example, if you changed the author from AAA to BBB then you would get two copies of the book on your device, one with path AAA ... and one with path BBB .... This has the good behavior that file names match metadata. It has the bad behavior that you can get multiple copies of the same book in your CC library.
  
  The changes attempt to get the best of both worlds. If templates are in use then if a book is downloaded by any method, the book file is renamed to match the template. File names match metadata and only one copy exists. The down side is that during wireless connects CC can "ignore" calibre and use a file name that calibre doesn't know about. Calibre interprets this as meaning that there are two copies on the device when in fact there aren't. CC now beeps and shows a message (a popup "toast") to tell you that calibre might be confused. Disconnecting and reconnecting eliminates the confusion.

[NSILMike]

It is totally non-functional for me. Crashes on start every single time... How do I get back to the non-beta versions? Or how soon will you be able to fix this issue?
My device is a samsung tab 4, model SM-T230NU, running android 4.4.2

[chaley]

Originally Posted by NSILMike

It is totally non-functional for me. Crashes on start every single time... How do I get back to the non-beta versions? Or how soon will you be able to fix this issue?
My device is a samsung tab 4, model SM-T230NU, running android 4.4.2

Color me unhappy with Google. They released a new support library that they strongly suggested we use. Well, it doesn't work with anything less than Android 5 when apps are packaged for release. It works fine on test versions. Grumble grumble.

I put a version of CC up on google play that uses the older library. To be sure, I ran this version (not a test version) on my Android 4.3 device.

My apologies.

[NSILMike]

Thanks for the quick response! However, either I'm reloading the wrong version, or it didn't help me. I went to Google Play, and uninstalled Calibre Companion, then rebooted (yes, probably unnecessary but color me a geek...) and reinstalled. I see only 2 versions of Companion in Google Play, the 'normal' and the 'demo.' Same thing happens after the reinstall... it immediately fails each and every time.

[chaley]

It takes an hour or two for a new version to appear. The broken version is V5.0.1.1. The corrected version is V5.0.1.2. The change log visible with the app on google play will also note that it fixes the problem.

Edit: you don't need to uninstall. The new version installs over the top of the old one.

[NSILMike]

Oops, wrong place... see below.

[NSILMike]

Thanks. I will post an update when I'm able to get it.

[NSILMike]

Got it, and it works great. Thanks for the fast and excellent support.

[Purple Lady]

These are on my Fire HD 8.

I tried to backup my settings and db to the external SD card and it failed with an error asking if the file was read only.

I successfully moved my mobi and azw3 books to the external SD card, but when I moved them back got the error in the screenshot. The files left behind on the external SD card do not have a period as the last character in the folder.

[chaley]

Originally Posted by Purple Lady

These are on my Fire HD 8.

I tried to backup my settings and db to the external SD card and it failed with an error asking if the file was read only.

This is by design. I thought that it would be too much trouble to incorporate the SAF into that dialog. I will revisit that.

If I don't incorporate the SAF then CC shouldn't let you navigate up from main memory. I will fix that. I will also add a comment to the Settings page saying that one can only backup to and restore from main memory, and if you want the backup on an SD card you must copy the file there yourself.

I successfully moved my mobi and azw3 books to the external SD card, but when I moved them back got the error in the screenshot. The files left behind on the external SD card do not have a period as the last character in the folder.

Ahh, another take on the same path problem, and one I should have anticipated.

For whatever reason you have books in your library that have "invalid" paths (paths with trailing periods in folder names). When CC moves books, it does not regenerate the path from the template but instead uses the existing path. Something in Android is stripping the trailing periods when the folders are created on the SD card, meaning that CC and Android now disagree about the path. Somehow CC is still able to open the book to copy it back to main memory, but it is not able to delete the book because of the folder name mismatch.

I see two ways to fix this:

1. Regenerate the path using the current template. Although this would guarantee a correct path, it could change the path in very significant ways. It also isn't clear what would happen if a user is depending on calibre templates.
2. "Clean" the path and store the cleaned version back with the book. This has the advantage of minimal change, but is slightly more complicated internally.

Both have the consequence that if you move the books back to main memory then the resulting path will be different from the original. The trailing periods will be gone. I prefer choice 2.

[chaley]

Originally Posted by Purple Lady

These are on my Fire HD 8.

I tried to backup my settings and db to the external SD card and it failed with an error asking if the file was read only.

Originally Posted by chaley

This is by design. I thought that it would be too much trouble to incorporate the SAF into that dialog. I will revisit that.

I am not going to change this to use the SAF. Writing zip files (a CC backup is a zip) through the SAF is too much of a pain, and in the end might not even work.

[chaley]

Regarding this change to CC

• Change the default template to {title} - {authors} ({id}) to ensure that books cannot overwrite each other in the rare case that two books have the same title and author. This is also calibre's default template. The default template is used by the content server and cloud connections if no options are checked in Settings / File names.

I am wondering whether this change is a good idea. Although it does solve a very rare problem, it is also arguably a "change for change sake" because the problem happens almost never (if ever).

The problem: the change could break CC file names for people who changed the file template in the wireless driver to be the same as the (old) default CC template ({title} - {authors}) because the change makes the content server and cloud connections use different names.

Does anyone think that the change is a good thing and that I should leave it there?

[Purple Lady]

Originally Posted by chaley

Ahh, another take on the same path problem, and one I should have anticipated.

For whatever reason you have books in your library that have "invalid" paths (paths with trailing periods in folder names). When CC moves books, it does not regenerate the path from the template but instead uses the existing path. Something in Android is stripping the trailing periods when the folders are created on the SD card, meaning that CC and Android now disagree about the path. Somehow CC is still able to open the book to copy it back to main memory, but it is not able to delete the book because of the folder name mismatch.

I see two ways to fix this:

1. Regenerate the path using the current template. Although this would guarantee a correct path, it could change the path in very significant ways. It also isn't clear what would happen if a user is depending on calibre templates.
2. "Clean" the path and store the cleaned version back with the book. This has the advantage of minimal change, but is slightly more complicated internally.

Both have the consequence that if you move the books back to main memory then the resulting path will be different from the original. The trailing periods will be gone. I prefer choice 2.

I have "invalid" paths because they were there before you made your changes. I have a lot of these, and I thought my azw3 books were this way too, but I didn't check before. The mobi ones that failed are back in internal storage with the periods - see screenshot with Silva, David B.

I think choice 2 is the better as well.

[Purple Lady]

Originally Posted by chaley

I am not going to change this to use the SAF. Writing zip files (a CC backup is a zip) through the SAF is too much of a pain, and in the end might not even work.

Can you leave it so someone can still navigate up from main memory? This way rooted users would still be able to store to the external SD. When a rooted device has the SDfix applied storage works like it used to before Kitkat with no changes needed to CC. I always wondered what the changes you made in CC for rooted users did since I never had it checked and was still able to use the external SD card.

[Purple Lady]

Originally Posted by chaley

Regarding this change to CC

• Change the default template to {title} - {authors} ({id}) to ensure that books cannot overwrite each other in the rare case that two books have the same title and author. This is also calibre's default template. The default template is used by the content server and cloud connections if no options are checked in Settings / File names.

I am wondering whether this change is a good idea. Although it does solve a very rare problem, it is also arguably a "change for change sake" because the problem happens almost never (if ever).

The problem: the change could break CC file names for people who changed the file template in the wireless driver to be the same as the (old) default CC template ({title} - {authors}) because the change makes the content server and cloud connections use different names.

Does anyone think that the change is a good thing and that I should leave it there?

I would say since the change could break things you shouldn't do it.