Sony PRS-T1 in Calibre

[Robert22]

Hi Kolenka,

Because the "_id" field in the "books" table is different between the two versions of the database, and also the sequence of the books is different, it looks as though the T1 has removed the books out of the "books" table and then added them back in. Which would explain why there are no matches between the Collections and Books. But why the T1 is doing this I have no idea.

You can also see that the "collection" table is the same in both DB versions but that the "collections" table in the second version is empty.

[Kolenka]

Yeah, there's a Windows bug with the file_path column.

The driver sets:
Sony_Reader\media\books\Adams, Douglas\Mostly Harmless.epub

The T1 wants:
Sony_Reader/media/books/Adams, Douglas/Mostly Harmless.epub

So the T1 removes the books the driver added (since the path doesn't exist), and then adds its own when scanning the media. That's what is causing books to drop out of collections, and is very likely behind the duplication as well. Since the driver's path doesn't match the database.

I thought the book's "lpath" (local path) would be using *nix path separators, not the OS' path separator. At least that is what the code looked like it did. Looks like I was wrong.

I can piece together a fix for that tonight.

Sony, Sony, Sony... what the heck did you do? I also see a 7 hour discrepancy in the modified time of the files. Looks like when the T1 re-added the books, it set the modified_date to a unix timestamp in local time for the device. The driver sets it to the UTC unix timestamp.

Yet, the added_order column is in the same format, but UTC. Sony get's the facepalm award from me for inconsistent database design.

[Robert22]

Originally Posted by Kolenka

Yeah, there's a Windows bug with the file_path column.

The driver sets:
Sony_Reader\media\books\Adams, Douglas\Mostly Harmless.epub

The T1 wants:
Sony_Reader/media/books/Adams, Douglas/Mostly Harmless.epub

So the T1 removes the books the driver added (since the path doesn't exist), and then adds its own when scanning the media. That's what is causing books to drop out of collections, and is very likely behind the duplication as well. Since the driver's path doesn't match the database.

I thought the book's "lpath" (local path) would be using *nix path separators, not the OS' path separator. At least that is what the code looked like it did. Looks like I was wrong.

I can piece together a fix for that tonight.

Sony, Sony, Sony... what the heck did you do? I also see a 7 hour discrepancy in the modified time of the files. Looks like when the T1 re-added the books, it set the modified_date to a unix timestamp in local time for the device. The driver sets it to the UTC unix timestamp.

Yet, the added_order column is in the same format, but UTC. Sony get's the facepalm award from me for inconsistent database design.

Good catch... I was just about to send you the same info about the paths...

[anesthezea]

Originally Posted by dwanthny

I'm curious. did any or all of your testing start with a clean slate of zero books/collections on the ereader?

Mine started clean slate. After that second batch of books, I ejected the T1 and went to bed to read for awhile, but ran into some buggy behavior - page turns were terribly, terribly slow and the unit froze twice and had to be reset. I don't know if this is due to the problems with the driver/Calibre or if it's the T1 itself. In either case, the reader hasn't behaved in this way before now. I plan to take everything off of it today and reload books via Reader Library to see if things smooth out.

[Kolenka]

Yeah, your post just caught me while I was also investigating the time stamp discrepancy.

I even took a look at my own, and it looks like the T1 will always update the modified_date column on an unplug. But it only updates the rest of the metadata if the file's date is newer than what was in the database.

This means the device_db plugboards work for those in west of UTC, but not east. I'm not entirely sure of a good way to discover the offset, especially if there are no books in the database. If there are, I can detect the offset, but if there aren't, it's a bit of a crap shoot.

[Kolenka]

Originally Posted by anesthezea

Mine started clean slate. After that second batch of books, I ejected the T1 and went to bed to read for awhile, but ran into some buggy behavior - page turns were terribly, terribly slow and the unit froze twice and had to be reset. I don't know if this is due to the problems with the driver/Calibre or if it's the T1 itself.

Do you know if it was still scanning the books when you hit this behavior? I've noticed the T1 (unlike older models) scans for books in the background, and makes the device verrrry slow while it is doing it.

With the bug I'm seeing, the Sony is doing a full rescan of your books, which if you have a lot could easily cause this behavior. Once the Windows bug is fixed, the loading phase after an unplug should be a lot faster and on par with the Reader Library application.

[Pegster]

I've nothing really helpful to add to this discussion, but I wanted to thank all of you for taking the time to add to Calibre and for testing it. As a Calibre user, I'm extremely appreciative.

[jackie_w]

Originally Posted by Kolenka

Yeah, your post just caught me while I was also investigating the time stamp discrepancy.

I even took a look at my own, and it looks like the T1 will always update the modified_date column on an unplug. But it only updates the rest of the metadata if the file's date is newer than what was in the database.

This means the device_db plugboards work for those in west of UTC, but not east. I'm not entirely sure of a good way to discover the offset, especially if there are no books in the database. If there are, I can detect the offset, but if there aren't, it's a bit of a crap shoot.

I've only just loaded the beta v0.8.21. Does this explain why plugboards are not working for me? My Time Zone is "London, Dublin" and I'm using Windows XP SP3.

On the plus side so far, Send-to-Device Save template is working and so are collections.

Need to test more tomorrow.

[Kolenka]

It would be (GMT+1, so east of UTC). It's a really stupid bug on the part of Sony too.

If I change my device's timezone to be east of where it was before (say GMT-7 to GMT-6), it will trigger a rescan of the entire library (and undo the work of my device_db plugboards). It makes me a little dead inside that something like this made it through, and there are easy ways on *nix systems to go from localtime to a valid Unix Epoch timestamp. They had to go out of their way to create a bug like this.

It's weird that you aren't seeing the path bug others are seeing, though.

[DoctorOhh]

Originally Posted by Kolenka

Minus the bugs folks are having with duplication, it's already at that point.

I don't pretend to know how any of this works and Thanks a bunch for contributing your time and knowledge to this driver.

Originally Posted by Kolenka

The difference between manual/automatic management is outside of the driver, and doesn't require any specific support for it. Calibre tells the driver what the data on the device should be, and the driver makes it so.

EDIT: Forgot to mention I actually use Automatic myself.

If this was true then wouldn't Automatic management work on all devices the same way it works on the older Sony models? Currently I have not heard anyone other than Sony owners acknowledge that their metadata automatically updates just by connecting their device. I assumed this was because the Sony driver wrote directly to the media.xml file and most other devices or drivers don't have an equivalent item to update.

Anyways Bravo Zulu for stepping up and helping out with this driver.

[Kolenka]

Originally Posted by dwanthny

If this was true then wouldn't Automatic management work on all devices the same way it works on the older Sony models? Currently I have not heard anyone other than Sony owners acknowledge that their metadata automatically updates just by connecting their device. I assumed this was because the Sony driver wrote directly to the media.xml file and most other devices or drivers don't have an equivalent item to update.

That is a good point.

Calibre has an idea of what the metadata should be. It passes on this metadata to the driver and tells it to update the device with that metadata. A driver can ignore it if it chooses. This makes sense when there is nowhere to put that metadata coming from Calibre.

As long as the driver refreshes whatever the device has for metadata (media.xml, books.db) during 'sync_booklists', then automatic/manual doesn't really matter. And really, I find that the driver is simpler if you do your metadata work in 'sync_booklists'.

The only things missing in the T1 driver versus the 505 driver are:
- "Use Author Sort" option so you don't need a plugboard for it. (And that is something I've got working again)
- Being able to toggle "Scan all folders". It scans all folders on internal memory and the SD card by default. This helps catch downloads on the device. If you could turn it off, there would be some interesting bugs that just don't have a reliable fix. I have to leave it turned on for my 950 with the 505 driver, so I don't see this as a loss of functionality.

As far as I can tell, it is already at feature parity with the 505 driver with those two exceptions.

[Robert22]

Originally Posted by Kolenka

It would be (GMT+1, so east of UTC). It's a really stupid bug on the part of Sony too.

If I change my device's timezone to be east of where it was before (say GMT-7 to GMT-6), it will trigger a rescan of the entire library (and undo the work of my device_db plugboards). It makes me a little dead inside that something like this made it through, and there are easy ways on *nix systems to go from localtime to a valid Unix Epoch timestamp. They had to go out of their way to create a bug like this.

It's weird that you aren't seeing the path bug others are seeing, though.

Will this affect the change from DST to Standard time?

[Kolenka]

Originally Posted by Robert22

Will this affect the change from DST to Standard time?

Don't know for sure, but I think it will. For you and me, the timestamps are -7 hours from UTC (PDT). So that likely means it will drop back to -8 hours from UTC when PST starts.

That means when PDT starts again, your device will rescan the whole library, because it has set all your books' modified_date fields to the PST value.

[jackie_w]

Please forgive me for poor testing of plugboard. I was using device_db:PRS505 I have now spotted the device_db:PRST1 option. I will retest.

[Kolenka]

Originally Posted by jackie_w

Please forgive me for poor testing of plugboard. I was using device_db:PRS505 I have now spotted the device_db:PRST1 option. I will retest.

I expect that they still won't work. You will unplug, smile at the result, and then watch as the T1 undoes all the work.