Sony PRS-T1 in Calibre

[azxswq]

Originally Posted by graycyn

I would be happy to donate $15-20 via PayPal if this is what is needed to have this happen. With luck, enough other people who like the tags to collection feature would consider doing the same to make it happen sooner. It wouldn't actually take tons of people at say $10 each.

Keep us posted.



Understood.

Got my T1, (Black and also waiting on the red.) Love everything about it.

I am able to read the Micro SD books by adding it to a disk-drive basis and really would appreciate the help Kolenka and potentially Chaley. I for one is happy to donate money to get this done, fast, right and it would be even better than what it has now.

Just the fact that it is now android, instead of linux, should pose serious challenges. But I gather the file management is entirely different. This, plus the abandonment of Memory stick, means that Sony is conceding and it will be a new standard for future product as well.

[log]

Originally Posted by kovidgoyal

I just got through 24 hours of flying so I'm a little loopy, but, IIRC, on Unix (OS X and linux) calibre only ejects the drives it knows about, i.e. not the SETTINGS, etc. partitions. On windows it ejects everything.

Thanks for response, didn't know you were already aware of this one, not a major issue anyway main volume is handled properly.

Originally Posted by Kolenka

I've opened up a bug with an early patch for the T1. It supports collections mostly similar to how the 505 driver supports it. Seems to handle additions, deletes, and editing collections directly well enough. Doesn't support any of the advanced thumbnail stuff that can be done on the 350/6xx/9xx models though. Should be easy to add, but I don't know if I'll be the one to do it.

Nice work Kolenka, looks like you also are updating the books database when just adding/removing books also. I guess this means that calibre will do the heavy lifting, and the T1 will not need to rescan the books metadata like the file system approach. Because of this, will the Sony firmware will still automatically scan the epub for the cover image and automatically generate it, or will it require the full thumbnail support implemented in new driver?

I also see that you are now using Sony_Reader/media/books as the main file system path on the device instead of calibre. Do you think this will pose any issues for those who use built in features on device? Only reason I ask is I did notice current beta T1 driver is not using this directory and thought maybe there could be a potential conflict if that directory will be shared with Sony desktop clients or the built in Reader Store/Public Library/Google Books features on the device.

In any case, great work with this so far! Really look forward to trying this out.

[Kolenka]

Originally Posted by log

Nice work Kolenka, looks like you also are updating the books database when just adding/removing books also. I guess this means that calibre will do the heavy lifting, and the T1 will not need to rescan the books metadata like the file system approach. Because of this, will the Sony firmware will still automatically scan the epub for the cover image and automatically generate it, or will it require the full thumbnail support implemented in new driver?

The Sony will notice there is no thumbnail and generate it, like with earlier models. The filesystem scanner only adds the book to the database, and the software will load/generate the thumbnail as-needed. It's cleaner that way.

The T1 will perform a scan anyhow, but it won't find anything to do, and your collections/etc will be available immediately while the scan happens in the background.

Originally Posted by log

I also see that you are now using Sony_Reader/media/books as the main file system path on the device instead of calibre. Do you think this will pose any issues for those who use built in features on device? Only reason I ask is I did notice current beta T1 driver is not using this directory and thought maybe there could be a potential conflict if that directory will be shared with Sony desktop clients or the built in Reader Store/Public Library/Google Books features on the device.

The T1 always downloads to the download folder, so the Store/Library/Web downloads won't conflict (plus I'd assume they'd figure out name conflicts on download anyhow). It hasn't been an issue with my device so far.

Actually, that brings up a bug in the initial patch: It doesn't actually put files into sub-folders based on the configuration like it should be. I just took a look and there is a typo: 'SUPPORT_SUB_DIR' instead of 'SUPPORTS_SUB_DIR'. D'oh.

Also, looks like detection of books being in your library isn't great when plugging in a device, due to the Sony only storing the first author in the database. Not sure of a good way to tackle that.

[log]

Originally Posted by Kolenka

The Sony will notice there is no thumbnail and generate it, like with earlier models. The filesystem scanner only adds the book to the database, and the software will load/generate the thumbnail as-needed. It's cleaner that way.

The T1 will perform a scan anyhow, but it won't find anything to do, and your collections/etc will be available immediately while the scan happens in the background.

Sounds good. Makes sense that it always will scan always after unplugging USB since they need to detect any changes to file system to support drag and drop to filesystem functionality, but as you mention there will be no additional work to do since all new files added by calibre will already be in the database.

This is great since no functionality of existing T1 driver is lost, and eventually when calibre library image for thumbnail support is added, will be a very nice feature that we can eliminate the step of tweaking the epub to change the cover image.

Also, looks like detection of books being in your library isn't great when plugging in a device, due to the Sony only storing the first author in the database. Not sure of a good way to tackle that.

I haven't noticed this yet with current beta, and did some digging to see how it works with standard file system.

If you have a epub with multiple authors delimited by ampersand (&) in calibre, it gets loaded by firmware filesystem scanner as you mentioned, with only the first author listed on the main book view. Interesting thing is if you open book, and go Menu -> More -> Setting -> Info, it shows all the authors, this time delimited by comma (,). I did check my books table, and there is single row for the book, and only the first author listed. My guess is on info screen it actually reads epub metadata.

This book correctly is detected as being in my library in current calibre beta also, so maybe there is some other variables besides the authors exact match that is causing your book to not be detected properly in calibre library as being on the device?

[Kolenka]

Originally Posted by log

If you have a epub with multiple authors delimited by ampersand (&) in calibre, it gets loaded by firmware filesystem scanner as you mentioned, with only the first author listed on the main book view. Interesting thing is if you open book, and go Menu -> More -> Setting -> Info, it shows all the authors, this time delimited by comma (,). I did check my books table, and there is single row for the book, and only the first author listed. My guess is on info screen it actually reads epub metadata.

That's correct. The Sony scanning routine only stores the first author in the database, but can read the rest from the ePub if you get info on it. My driver is trying to mimic Sony's behavior as much as possible, so it only stores the first author in the database.

Originally Posted by log

This book correctly is detected as being in my library in current calibre beta also, so maybe there is some other variables besides the authors exact match that is causing your book to not be detected properly in calibre library as being on the device?

The current beta reads the metadata from the ePub directly. The driver I wrote reads from the database for speed. The intent is to avoid querying multiple locations for data if at all possible. But, that means only the metadata in the database can be used for matching. The USBMS driver relies on more metadata inside the ePub itself (UUID/ISBN for example), which is more reliable on producing matches. But, that information isn't stored in books.db.

I know the problem, but as I said, I don't know how best to tackle it. It seems kinda wasteful to have to pop open each ePub to read information, but it might be unavoidable. Maybe it needs to be an option that people can turn off if it gets too slow for them.

Also keep in mind that the cover image used for the sleep display is pulled from the first page of the book. Even if a different thumbnail is used. So Calibre won't be able to make your sleep screen look any better if the ePub needs tweaks to get a decent cover.

[kovidgoyal]

@Kolenka: I haven't had a chance to review your code, but on general principles...

The USBMS classes maintain a cache of metadata read from the ebook files in the file metadata.calibre in the root of the device. You should allow USBMS to read metadata from the files as normal (this will automatically use the cache wherever possible). Then take the resulting books and match them against entries in the SONY db. This gives you excellent performance (after the initial scan) and it allows you to present correct and complete metadata to the user in the device view without having to rely on the vagaries of SONY's metadata reading implementation.

See for example the books() and sync_booklists() methods in the old sony driver.

[log]

Good to know about sleep display, didn't realize until you mentioned it that the cover image of book is not stored in cache directory, only the library browser thumbnails -- this is primary reason I think most would be interested in the feature.

[Kolenka]

Originally Posted by kovidgoyal

The USBMS classes maintain a cache of metadata read from the ebook files in the file metadata.calibre in the root of the device. You should allow USBMS to read metadata from the files as normal (this will automatically use the cache wherever possible). Then take the resulting books and match them against entries in the SONY db. This gives you excellent performance (after the initial scan) and it allows you to present correct and complete metadata to the user in the device view without having to rely on the vagaries of SONY's metadata reading implementation.

See for example the books() and sync_booklists() methods in the old sony driver.

Hmm, indeed. I used the Kobo as more of a template, than the 505 driver.

I was suspicious that this wouldn't work as expected due to sync_bookslists being called where it was. It was a bit hard to track the order of things and the end result. So I tested it by downloading and adding a book to a collection. Amazingly, that collection showed up in Calibre as I expected. Had to make a couple other small tweaks in the process, but it has killed the bug (along with another from rewriting sync_booklists from earlier), and perf is good enough.

I've included an updated patch with the feedback. It's at the point now that it's pretty usable. Really just needs thumbnail support (not 'vital', IMO) and maybe some fine tuning (more user options, perhaps) to catch up to the existing 505 driver.

[kovidgoyal]

Cool, I look forward to reviewing it. Unfortunately I am currently running around trying to find a place to live So it will have to wait a few more days.

Glancing over the code I notice that you have lrf missing from FORMATS, has SONY really dropped LRF rendering support in the T1?

[sabredog]

Kovid

Yes it has, as per this

I did not use LRF much, but my daughter has been, though on a PRS-300.

[Robert22]

Originally Posted by kovidgoyal

Glancing over the code I notice that you have lrf missing from FORMATS, has SONY really dropped LRF rendering support in the T1?

I saw that it was missing from the specs. Just to make sure I manually copied 2 files, an EPUB and LRF, to the T1 in the same directory. After unplugging, the EPUB showed up on the Reader but not the LRF so it does seem that they have dropped the support for LRF.

[kovidgoyal]

A pity. They should have made the effort to port a reader for LRF to android. It's not very nice to force people to unnecessarily convert books.

[Kolenka]

I just made sure the list matched the official specs.

I'm not terribly surprised, though. With how long LRF has been deprecated, and the fact that they had to delay features out of the v1.0 firmware to make the ship dates... I'd have cut LRF instead of cutting automatic periodical delivery (which they delayed to a future update).

Part of the decision is likely based on understanding that 350/650 users may not be willing to upgrade so quickly after their purchase.

Gah, I just realized something. I had it reading the thumbnail path into the booklist before, and rewriting books() removed it. I should probably add that back in.

[graycyn]

Just want to say thanks, Kolenka for working on this!

[jersysman]

I want to give Kolenka my thanks as well! Hopefully, when my T1 comes in, I'll have this Calibre plugin to make handling my books as great as Calibre always has been.

+1