V5.0.0 alpha channel release

[chaley]

Change log and explanations:

1. Exchange the title and summary of animate covers
2. Remove period (.) as a legal character in CC template-generated path names, replacing it with an underscore.
3. Add an option to rename book files if a book with a different path is being downloaded

Explanation for the last two.

• Remove periods: CC has always changed periods (and other characters) to underscores when downloading books using the content server and cloud connections (without templates). Calibre does similar things when sending books via the wireless device. Up to now, templates have allowed periods, meaning that a template could generate a different name than calibre or the default file name mechanism. This change attempts to make the methods work better together. They should also prevent at least some of the problems Purple Lady is reporting.
• Rename book files: By default and since forever, when sending a book over the wireless device that already exists in CC's library, calibre + CC reused the old path, ignoring the results of the template. This means that if you changed a book's author from AAA to BBB and then resent the book, the file name would still contain AAA. If you downloaded a book that already exists via the content server or the cloud a similar thing happened. The book file was not renamed. This is good behavior in that it prevented creation of duplicate files. The bad behavior is that the file names might not match the metadata.
  
  When templates were introduced this behavior changed. Using the above example, if you changed the author from AAA to BBB then you would get two copies of the book on your device, one with path AAA ... and one with path BBB .... This has the good behavior that file names match metadata. It has the bad behavior that you can get multiple copies of the same book in your CC library.
  
  The changes I made attempt to get the best of both worlds. If templates are in use then if a book is downloaded by any method, the book file is renamed to match the template. File names match metadata and only one copy exists. The down side is that during wireless connects CC can "ignore" calibre and use a file name that calibre doesn't know about. Calibre interprets this as meaning that there are two copies on the device when in fact there aren't. CC now shows a message (a toast) to tell you that calibre might be confused. Disconnecting and reconnecting eliminates the confusion.
  
  I added an option to Settings / File Names to disable this new behavior, should someone prefer what it did before. The new behavior is [EDIT] disabled by default.

I will make the release sometime before my evening. I am still testing.

[jackie_w]

Originally Posted by chaley

Does it still happen with V5.0.0.2? I thought I fixed it. If it does still happen then I need another debug log, please.
Quite reasonable. CC should (and will) support this.

I hadn't realised there was an update. I've just tried it and no more hanging. Thanks

[copyrite]

I've tried the update out on two devices with no issues now to update on the device I primarily read on.

[chaley]

I have released V5.0.0.3. See post 31 above for details.

If you use CC templates I recommend that you enable the new option in CC / Settings / Filenames / Replace existing book files before you download any books using the wireless connection.

I hope that this will the last release for a while. I want to give people a chance simply to use the new version. In addition, I will be away-from-keyboard Monday through Wed.

[Purple Lady]

Originally Posted by Purple Lady

I am running V5.0.0.2.

Books with missing files show a mobi book is missing. The screen shot shows where the missing book is - in Konrath, J. A. (1). The other Konrath, J. A is missing the period at the end and contains a book that is not missing. Amazon seems to have a problem with a folder when the last character is a period, but only on the external SD. I will create another post for that - I thought it was only ES File Explorer

I can't successfully move all the books back to internal. See screenshots. All books with problems have an author with a middle initial, which should create a folder ending with a period.

Maybe my SD card is faulty? I will look for another.
Attachment 146294

Attachment 146295

Attachment 146296

I'm preparing to test the latest, so I'm looking at these problem books. All the ones that had an error because it couldn't delete the current file actually did copy to the new location, it just didn't delete the old. I don't know if this is expected behavior, but it makes things easier for me. I can just delete all books from external SD and re-download the one not found.

[Purple Lady]

Originally Posted by chaley

Change log and explanations:

Spoiler:

1. Exchange the title and summary of animate covers
2. Remove period (.) as a legal character in CC template-generated path names, replacing it with an underscore.
3. Add an option to rename book files if a book with a different path is being downloaded

Explanation for the last two.

• Remove periods: CC has always changed periods (and other characters) to underscores when downloading books using the content server and cloud connections (without templates). Calibre does similar things when sending books via the wireless device. Up to now, templates have allowed periods, meaning that a template could generate a different name than calibre or the default file name mechanism. This change attempts to make the methods work better together. They should also prevent at least some of the problems Purple Lady is reporting.
• Rename book files: By default and since forever, when sending a book over the wireless device that already exists in CC's library, calibre + CC reused the old path, ignoring the results of the template. This means that if you changed a book's author from AAA to BBB and then resent the book, the file name would still contain AAA. If you downloaded a book that already exists via the content server or the cloud a similar thing happened. The book file was not renamed. This is good behavior in that it prevented creation of duplicate files. The bad behavior is that the file names might not match the metadata.
  
  When templates were introduced this behavior changed. Using the above example, if you changed the author from AAA to BBB then you would get two copies of the book on your device, one with path AAA ... and one with path BBB .... This has the good behavior that file names match metadata. It has the bad behavior that you can get multiple copies of the same book in your CC library.
  
  The changes I made attempt to get the best of both worlds. If templates are in use then if a book is downloaded by any method, the book file is renamed to match the template. File names match metadata and only one copy exists. The down side is that during wireless connects CC can "ignore" calibre and use a file name that calibre doesn't know about. Calibre interprets this as meaning that there are two copies on the device when in fact there aren't. CC now shows a message (a toast) to tell you that calibre might be confused. Disconnecting and reconnecting eliminates the confusion.
  
  I added an option to Settings / File Names to disable this new behavior, should someone prefer what it did before. The new behavior is [EDIT] disabled by default.

I will make the release sometime before my evening. I am still testing.

I re-downloaded my one book, but the path looks wrong. In addition to substituting a period to underscore it also changed the comma to underscore. See the two Konrath folders in the screenprint.

[Purple Lady]

I just tried moving my one azw3 file to the external SD, and the path created still had the missing period at the end. I am using alpha 3.

[Purple Lady]

I just noticed that this release also puts underscores in the book file name as well. Was that intended? If the book file name changes, it means that syncing reading positions won't work anymore. I really rely on that.

[chaley]

Originally Posted by Purple Lady

I'm preparing to test the latest, so I'm looking at these problem books. All the ones that had an error because it couldn't delete the current file actually did copy to the new location, it just didn't delete the old. I don't know if this is expected behavior, but it makes things easier for me. I can just delete all books from external SD and re-download the one not found.

Re not deleting the files: no, that isn't intended behavior and it isn't behavior that I see. I suspect that it happened for you because of the various path name changes in the alpha releases. A book file could have been orphaned.

Originally Posted by Purple Lady

I re-downloaded my one book, but the path looks wrong. In addition to substituting a period to underscore it also changed the comma to underscore. See the two Konrath folders in the screenprint.
Attachment 146341

I decided to spend the time to determine what calibre does to paths and, if possible, do the same thing in CC.

Calibre first "ascii-ized" the text, getting rid of non-Latin characters. CC won't do this.

1. Calibre next changes the following characters to underscores -- characters that are not:
   • letters, numbers, or a space
   • !, $, %, ^, &, (, ), =, -, {, }, \, [, ], @, ~, ', #, comma, period, _, and /.
2. Calibre next changes any leading period to an underscore and removes all trailing periods.
3. Finally, calibre removes any empty path segments, e.g., aaa//bbb becomes aaa/bbb.

CC now does these things as well.

CC will optionally remove accents from characters.

These changes deal with the comma -> underscore problem. It won't happen any more.

Originally Posted by Purple Lady

I just tried moving my one azw3 file to the external SD, and the path created still had the missing period at the end. I am using alpha 3.
Attachment 146342

This implies that you didn't resend the book or that the "replace" option is not checked, because CC alpha 3 would not leave that dot in the folder name.

Originally Posted by Purple Lady

I just noticed that this release also puts underscores in the book file name as well. Was that intended? If the book file name changes, it means that syncing reading positions won't work anymore. I really rely on that.
Attachment 146343

Yes, it is intended, for two reasons. 1) The "illegal" characters must be changed but not removed in order to prevent the path segment from becoming zero length. 2) That is what calibre does.

Assuming you are using a CC template, the file names will be the same if you are running the CC V5 on all devices and if you resend all your books to every device. Why? CC V5 generates a different name than CC V4. If you don't resend the files then some books will have the V4 name and some will have the V5 name.

For testing, and again assuming you are using CC's templates, I strongly suggest

• that you use the "Replace existing book files" option
• that once you receive the next alpha (alpha 4, to be released today), you resend from calibre all the books on the device.

This will ensure that all the books have the "correct" path and that you don't end up with multiple copies of books in CC.

FYI: here are the scenarios I have tested:

On Android 4.3 w/o an SD card (Galaxy Nexus)

1. Uninstall CC
2. Delete all book folders
3. Install CC. Change no settings except to allow azw3
4. Connect to calibre
5. Send some books: 5 epub, 2 mobi, 1 azw3. One epub (the "strange" one) has

   title="Unknown !"$%^&*()_+=-{}[]@~'#<>?,./\|",
   author="Unknown 222g. A.....".

   This tests all the character substitution. It will also test the removal of periods in folder names.
6. When sending the strange epub calibre sent the path
   

   Unknown !_$%^&_()__=-{}[]@~'#___,.___ - Unknown 222g. A__. - (1412).epub

7. Disconnect
8. Check for missing books: none found
9. Turn on CC templates with template {title} - {authors}. Check the replace existing book files option
10. Connect to calibre. Send the strange book again. CC warns me that calibre might show the wrong number of books, which happens because the path to the book changed during the send.
    

    Orig calibre path is = Unknown !_$%^&_()__=-{}[]@~'#___,.___ - Unknown 222g. A__. - (1412).epub
    Template gen'ed path = Unknown !_$%^&_()__=-{}[]@~'#___,.___ - Unknown 222g. A.epub

    We can see that CC's template generated the same title and author string for the template as calibre did, with the exception of the book id what isn't in CC's template.
11. Disconnect.
12. Change the template to {authors}/{title} - {authors}.
13. Reconnect. Send the strange book again.
    

    Template gen'ed path = Unknown 222g. A/Unknown !_$%^&_()__=-{}[]@~'#___,.___ - Unknown 222g. A.epub

    We see that the trailing dots of the authors name are gone from the folder.

On Android 5.1 with an SD card(Amazon Fire):
All of the above steps produced the same results. Next are the SD card tests.

1. Use CC Formats and Folders to move mobi and azw3 to /CC on the SD card. ES File Explorer says that the files are now on the SD card and have been deleted from main memory. The folder for the strange book still doesn't have a trailing period.
2. In CC, delete the azw3 book. ES File Explorer says the book is gone.
3. Connect and resend the azw3 book.
4. The book appears on the SD card in CC using the CC template to create an author folder.
5. Move the epubs to the SD card by changing the default folder in CC. ES File Explorer says the books are on the SD card and gone from main memory.
6. Delete the strange book. ES File Explorer says the book and the author folder are gone.
7. Connect and resend the "strange" book. The book appears and yet again the author folder has no trailing dots.

[chaley]

V5.0.0.4:
Change log:

• Change CC template processing so that it does much the same as calibre. Exception: CC does not "ascii-fy" (convert non-Latin characters to Latin).
• Only tell the user one time per connect when calibre might show multiple books on the device when in fact there is only one. This situation can arise if the "Replace books" option in Files is checked.

[Purple Lady]

Originally Posted by chaley

Re not deleting the files: no, that isn't intended behavior and it isn't behavior that I see. I suspect that it happened for you because of the various path name changes in the alpha releases. A book file could have been orphaned.

Yes, this was because of the missing period in the folder name with alpha 2.

This implies that you didn't resend the book or that the "replace" option is not checked, because CC alpha 3 would not leave that dot in the folder name.

I'm pretty sure I did have replace option checked. I didn't resend, this happened when I changed the file location in settings. CC moved the book files, so resending was not necessary.

Yes, it is intended, for two reasons. 1) The "illegal" characters must be changed but not removed in order to prevent the path segment from becoming zero length. 2) That is what calibre does.

Assuming you are using a CC template, the file names will be the same if you are running the CC V5 on all devices and if you resend all your books to every device. Why? CC V5 generates a different name than CC V4. If you don't resend the files then some books will have the V4 name and some will have the V5 name.

For testing, and again assuming you are using CC's templates, I strongly suggest

• that you use the "Replace existing book files" option
• that once you receive the next alpha (alpha 4, to be released today), you resend from calibre all the books on the device.

This will ensure that all the books have the "correct" path and that you don't end up with multiple copies of books in CC.

I don't have a problem with the path names changing, only with the actual book file names changing.

I am not willing to resend all my books to all my devices. I know that I am probably an outlier because I have all my books on most of my devices. I am going to revert back to the last production version because I use Moon+ sync capability every day. I hope you include a warning that the new version will not have the same naming convention as the old version and that the ability to sync reading positions with some Android reading apps may no longer work unless all books on all devices are sent again. The "Use CC template everywhere" mentions Moon+ by name so you will need to change that as well.

I know that reverting to the last prod version means that I will need to purchase CC at Amazon, but that's a small price for me to pay to be able to sync reading positions. There's also now root available on the HD 8 and 10 models so I will be able to use the SD card too.

[chaley]

Originally Posted by Purple Lady

I am not willing to resend all my books to all my devices. I know that I am probably an outlier because I have all my books on most of my devices. I am going to revert back to the last production version because I use Moon+ sync capability every day. I hope you include a warning that the new version will not have the same naming convention as the old version and that the ability to sync reading positions with some Android reading apps may no longer work unless all books on all devices are sent again. The "Use CC template everywhere" mentions Moon+ by name so you will need to change that as well.

Sorry to see you go.

No, I won't include the warning. Any problem only affects:

• people who use CC templates AND
• use more than one device AND
• use syncing AND
• have trailing periods in items used to make folders OR
• if the template itself or fields used in the template contain character that are now allowed but were not allowed in V4 ({}-!$%^#@~)

This is a small group of people and an even smaller number of books. Adding the warning would confuse everyone for very little value.

I know that reverting to the last prod version means that I will need to purchase CC at Amazon, but that's a small price for me to pay to be able to sync reading positions. There's also now root available on the HD 8 and 10 models so I will be able to use the SD card too.

Buying from Amazon won't help much. I will release V5 there once the beta period is over.

Note that V5 will use SAF on Android 5 or later even on rooted phones, meaning that you probably cannot ever accept a V5 update.

If you drop out of the alpha and beta programs then Google Play will offer you V4 until V5 goes to production. Let me know if you want to drop out.

[Purple Lady]

Originally Posted by chaley

Sorry to see you go.

No, I won't include the warning. Any problem only affects:

• people who use CC templates AND
• use more than one device AND
• use syncing AND
• have trailing periods in items used to make folders OR
• if the template itself or fields used in the template contain character that are now allowed but were not allowed in V4 ({}-!$%^#@~)

This is a small group of people and an even smaller number of books. Adding the warning would confuse everyone for very little value.
Buying from Amazon won't help much. I will release V5 there once the beta period is over.

Note that V5 will use SAF on Android 5 or later even on rooted phones, meaning that you probably cannot ever accept a V5 update.

If you drop out of the alpha and beta programs then Google Play will offer you V4 until V5 goes to production. Let me know if you want to drop out.

I think that more file names than you think will be affected. Anyone who uses author sort in the file name will be affected - that will always include a comma the old way and an underscore the new way. Any author that has a period anywhere in their name will have a different file name as well. I didn't realize how many that was until this.

Buying from Amazon will allow me to use it without the play store, but I realize that I won't be able to update from either store.

[chaley]

Originally Posted by Purple Lady

I think that more file names than you think will be affected. Anyone who uses author sort in the file name will be affected - that will always include a comma the old way and an underscore the new way. Any author that has a period anywhere in their name will have a different file name as well. I didn't realize how many that was until this.

I suspect you haven't tried alpha 4. Commas in names are left alone. Periods are left alone unless they are at the beginning or the end of the segment. For example, a book with

title=Unknown. Mankind is doomed
Author=Weber, David

when used with the CC template "{authors}/{title} - {authors}" produces the path

Weber, David/Unknown. Mankind is doomed - Weber, David.epub

You can see that the periods and the commas are still there.

Regardless, I won't argue with you about it. Thanks for your help, and best wishes for the future.

[chaley]

Originally Posted by Purple Lady

Buying from Amazon will allow me to use it without the play store, but I realize that I won't be able to update from either store.

V4.3.3.4 from Google Play should work fine if installed on a device without the play store installed.