Sony PRS-650 Author Sort Sometimes Overwritten

[seanbob]

I've noticed this problem since the recent Calibre changes to the Sony device plugin. Perhaps it's related in some way with the thumbnail issue now occurring too? I'm on Calibre 0.7.38 currently.

When adding books to my PRS-650 with the "Use Author Sort as Author" option enabled, it sometimes sorts the authors properly and sometimes not.

I originally noticed it after formatting the internal memory completely and adding a large collection of books to the Reader. The file directory structure was created properly (using Author Sort) and the media.xml cache file listed the book metadata using Author Sort too.

Upon disconnect, the Sony Reader changed the authors back to "FirstName LastName" for some reason. I connected the Reader to the computer again, and noticed that the media.xml file had been changed likewise. I synced a book (that was already on the device) using Calibre and it changed it back again to the Author Sort and on that disconnect, the Sony retained the metadata.

Now as I add/remove books from the Reader, it seems to be hit or miss when it keeps the Author Sort or overwrites it (a similar behavior to what I'm seeing with the thumbnails which sometimes are and sometimes aren't overwritten by the Sony Reader after syncing with Calibre).

Am I doing something wrong? Or has a recent change affected the behavior? Is there anything I can do to help solve the problem? Thanks in advance for any help here...

[jackie_w]

chaley is the expert on these matters, but I'll see if I can help.

The thumbnail issue and proposed workaround is discussed in this short thread.

As for your author_sort problem... I've just sent 50 books to my PRS650. After disconnecting, the author_sorts appeared correctly. I then reconnected and disconnected again. The author_sorts are still intact.

I don't have the "Use Author Sort as Author" option enabled in the Sony device plugin, I have moved it into my Metadata Plugboard. Do you have one of these set up for your Sony?

How are you viewing your bookshelf when you see author instead of author_sort? In Collections ? See all by Author?

Have you double-checked the actual values of author and author_sort in calibre edit metadata for a badly behaving book?

[seanbob]

Here's some additional information based on my current Calibre setup.

I don't have the "Use Author Sort as Author" option enabled in the Sony device plugin, I have moved it into my Metadata Plugboard. Do you have one of these set up for your Sony?

I don't have the Metadata Plugboard set up at all for my Sony. I've just used the Sony device plugin to handle everything, as I've never needed anything too fancy. But... I would be curious to know how you have the Metadata Plugboard set up for your Reader in case I get the urge to tinker.

How are you viewing your bookshelf when you see author instead of author_sort? In Collections ? See all by Author?

I'm viewing the bookshelf using "See All" sorted by Author. Sometimes the authors are displayed (and sorted correctly) as "LastName, FirstName" and sometimes they change to "FirstName LastName".

From what I can gather, the Sony Reader is overwriting Calibre's metadata when it processes the library upon disconnect (perhaps grabbing new thumbnails and Author/Title information from the eBook itself rather than using the metadata already assigned to it in media.xml). However, this behavior appears random, as I haven't yet discovered any pattern as to why it sometimes overwrites the metadata and sometimes leaves it alone.

Viewing the books using the custom Collection "All Books by Author" (aba) works great every time, but I miss having the alphabetic index shortcuts on the right side of the screen when viewing a collection.

Have you double-checked the actual values of author and author_sort in calibre edit metadata for a badly behaving book?

I have recently edited the metadata for each book individually to ensure that their Author and Author Sort fields are correct.

When viewed on the Reader, this issue is never limited to a single book. Rather, it's all or nothing - with every book displaying Author on the device or every book displaying Author Sort (as preferred).

[jackie_w]

Originally Posted by seanbob

I don't have the Metadata Plugboard set up at all for my Sony. I've just used the Sony device plugin to handle everything, as I've never needed anything too fancy. But... I would be curious to know how you have the Metadata Plugboard set up for your Reader in case I get the urge to tinker.

Rather than me type it again, can I refer you to the beginning of post #1 of this thread where I listed the current settings in my Plugboard (I played a small part in testing chaley's new "All by Author" and "All by Title" special Collections). I also have an earlier post where my original Plugboard attempt was shown with a bit more detail and explanation.

You could try a much simpler Sony device_db:PRS505 Plugboard with only the {author_sort} -> {authors} bit if you wanted to do a little test. Once the Plugboard is set up only a connect/disconnect to calibre should be needed to realign all the metadata. It won't change filenames. If you do you should also probably disable your existing author_sort setting in the Sony device plugin. If the Plugboard works but the Sony device plugin doesn't then it does sound as if there may be a problem with the latter. Edited as not relevant to real problem.

Originally Posted by seanbob

Viewing the books using the custom Collection "All Books by Author" (aba) works great every time, but I miss having the alphabetic index shortcuts on the right side of the screen when viewing a collection.

I may be able to help with this one. Have you tried changing the default "by Added Order" to "by Author" when viewing special Collection "All by Author". The book order doesn't change but the alpha strip appears.

For better explanations of your current predicament... let's hope more expert advice will drop by.

[chaley]

You are dealing with two problems.

Sonys rescan the book if the book's file date is not the same as the date in its database. Calibre tries to write the correct date into the database, but sometimes it can't because of timezones. The reader will adjust the date to the GMT from the timezone it thinks it is in. There is no way for calibre to find the device's timezone, so it guesses by comparing the timestamps of existing books to the values in the database. If you empty your reader and send books, then there is nothing to compare to. In your case, calibre probably picked the wrong offset, resulting the dates being wrong from the reader's point of view and therefore all the information in the sony DB being replaced with information from the books. In the future, it would be better to send one book, disconnect & reconnect, then send the rest. This gives calibre a chance to find the device's timezone information. However, there is no guarantee, as there seems to be some conditions where the sony will regenerate the information even if the date is correct.

The covers problem is one that Jackie_w (who has a 650) and I (who doesn't) have spent hours fighting. We have tried to determine exactly how the Sony determines that it needs to rescan the book to regenerate the cover. We haven't succeeded.

Note that both problems go away if you set metadata management to automatic (preferences -> send to device -> metadata management). In this case, calibre will refresh the Sony DB on every connect, fixing the information that the reader unfixed. Note: in automatic mode, calibre will delete any collection that calibre didn't create, so do not set this option if you use other software such as Sony Reader or ADE to manage your device.

It is worth noting that people who never use ADE or Sony Reader seem not to have the regeneration-of-metadata problem. This is entirely anecdotal, and could be nonsense. We *do* know that Sony Reader messes with timezones and date/time, something that has caused grief for this programmer.

[seanbob]

chaley: You are definitely right about the date/time issue!

As a test, I formatted the 650's internal memory again and loaded a single book, then disconnected. The author name was displayed incorrectly as "fn ln". I then reconnected and disconnected the 650, without making any changes (but using automatic metadata management). The author name was now correctly displayed as "ln, fn" using author sort.

I do sometimes use ADE to add borrowed library books, but do not create collections. If I continue to use automatic metadata management in Calibre, will I have a problem using ADE to load/read library rentals on my device? Or is it only a problem if I've created collections for them in ADE?

I tried to find an answer for myself, but it just led to more questions...
I added DRM'ed library books into Calibre, which added them to the Reader just fine, but with a different look. It appeared as if their DRM had been stripped out since it no longer showed the usual expiration countdown. Is there an explanation for this behavior? Could they still be DRM'ed, but not showing it? Can Calibre automatically create a custom collection based on books that are DRM'ed, such as library rentals?

There's no urgency to these additional questions, just mild curiosity at this point. Thanks again for all your help!

[seanbob]

jackie_w: Thanks for all your help on the issue too! I appreciate the links to the plugboard posts and think they'll be useful when I eventually get around to tinkering with it. You were also right about this...

I may be able to help with this one. Have you tried changing the default "by Added Order" to "by Author" when viewing special Collection "All by Author". The book order doesn't change but the alpha strip appears.

You seem quite knowledgeable about the 650. Would you mind if I ask you 2 more quick questions that stemmed from that response?

1. I got the alpha strip to appear in the special collections (both title and author), but once I open a book or navigate to another screen, the sort is lost. Do you know if there's any way to change the default sort for a collection, or does it always have to be in "Added Order"?

2. What purpose does the Purchased Books collection serve? I tried loading in some DRM'ed eBooks I purchased (using both ADE and Calibre) and never saw anything in that collection. Does it only add books to it if they were purchased from the Sony eBook Store? Or maybe only if they were added through the Sony Reader software?

Thanks again for the help!

[jackie_w]

Re: 1 - If I open a book from the "All by Author" Collection sorted by Author, then Options - Back to List, the sort by Author seems to be retained. But if navigating Home - Collections - All by Author, then sort by Added Order seems to be the default starting position. I don't know whether calibre controls this default. All I can see in the media.xml file for Collections (playlist) is title, sourceid, id, uuid. I don't think items 1, 3 and 4 define sort sequence, I'm not sure what sourceid is but it doesn't sound much like a sort sequence. We'd need chaley again.

Re: 2 - I don't have any DRM'd books or Library books, so I was hoping you'd know more than me on that front. Your guess is as good as mine.

Not much help, I'm afraid.

[chaley]

Originally Posted by jackie_w

Re: 1 - If I open a book from the "All by Author" Collection sorted by Author, then Options - Back to List, the sort by Author seems to be retained. But if navigating Home - Collections - All by Author, then sort by Added Order seems to be the default starting position. I don't know whether calibre controls this default. All I can see in the media.xml file for Collections (playlist) is title, sourceid, id, uuid. I don't think items 1, 3 and 4 define sort sequence, I'm not sure what sourceid is but it doesn't sound much like a sort sequence. We'd need chaley again.

As far as I know, calibre has no role in what kind of sort/display the device uses. Calibre puts information into the DB. The reader displays it how it wants to.

For collections, calibre adds books so that the 'added order' is the order calibre thinks the collection should be in. This order is controllable by the user using tweaks and plugboards. The default is 'series index order' for collections built from series metadata columns and title_sort order for collections built from non-series columns. The tweak sony_collection_sorting_rules is used to indicate that another metadata column is to be used to sort a collection. Plugboards can be used to change the value of title_sort.

The sourceid tells the sony where the collection is -- main memory or on a card. IIRC, ids 1-L are in memory, L+1 - M are on card A, and M+1 - N are on card B. SourceID is set to L+1 or M+1, depending. I am operating from memory, so I might have the values wrong.

Re: 2 - I don't have any DRM'd books or Library books, so I was hoping you'd know more than me on that front. Your guess is as good as mine.

I also have no idea. I have a 300 that doesn't have such a collection, and in any event I have never put a DRMed book on my device.

Originally Posted by seanbob

I do sometimes use ADE to add borrowed library books, but do not create collections. If I continue to use automatic metadata management in Calibre, will I have a problem using ADE to load/read library rentals on my device? Or is it only a problem if I've created collections for them in ADE?

ADE puts books in a place that calibre doesn't know about. Because of this, calibre thinks that the books don't exist and deletes them from the database. The reader will re-add them, but their presence in collections is lost. I also think that bookmarks and the like are lost.

I have considered enhancing calibre's sony device driver to scan for books in these other special places. My problem is that I don't know where these special places are or what the side effects might be.

I tried to find an answer for myself, but it just led to more questions...
I added DRM'ed library books into Calibre, which added them to the Reader just fine, but with a different look. It appeared as if their DRM had been stripped out since it no longer showed the usual expiration countdown. Is there an explanation for this behavior? Could they still be DRM'ed, but not showing it? Can Calibre automatically create a custom collection based on books that are DRM'ed, such as library rentals?

My guess is that the clock appears for books in ADE's special folder. If the book is elsewhere, then the reader thinks that it is a normal DRMed book. The books are most definitely still DRMed.

As for creating a special collection: no, not easily. The problem is that calibre doesn't know a book is DRMed until it tries to manipulate it, something that happens outside the part of code that deals with collections. My suggestion is to tag (perhaps a custom column) these books when you add them, then build a collection from the column. If you are a programmer, then consider writing a plugin to add library books, tagging them in the process.

[seanbob]

Thanks for the great feedback! (My appreciation is long overdue, but I was away on my honeymoon and only recently returned.)

It looks like the Sony plugin got some additional enhancements while I was out and I can't wait to see what else might be in the works. Good job!

[jackie_w]

As of v7.44, you should find that any thumbnail issues are resolved.

Oh, and congratulations, by the way