[GUI Plugin] Manage Sony x50 Reader Book List

[MisterMoonlight]

Ok i sent you (PM) the debug output with the debug plugin. It looks like the media.xml filename is wrong because it has mix of \ and / characters...

Thanks

[kpw]

Originally Posted by MisterMoonlight

Ok i sent you (PM) the debug output with the debug plugin. It looks like the media.xml filename is wrong because it has mix of \ and / characters...

Thanks once again! This is turning into a very intersting issue...

You're right that the cache file path was not being normalized on read/write, so the plugin was using a path with a mix of \ and / characters. I've fixed that and have another driver that I hope you can try out for me. I'd appreciate it if you can generate the same trace file as last time as well - I've added trace entries to show the file system stats for each file whose timestamp I update both before and after the update, as well as a load/dump of the books from the cache file, again both before and after the media.xml file update.

Hopefully the normalized paths will fix the issue. The non-normalized paths haven't been a problem on the computers I've been using to test with, but that's what makes debugging so much fun!

If it's still not working with the fix, then the trace might point to something.

I'll keep my fingers crossed.

[MisterMoonlight]

I have tried the new debug plugging and the bug is still there. It is not related to wrong filename with \ and /.

By looking at the trace output, we can see that the plugin seems to do the job correctly according to the trace. But the files seems not to be written at all. When looking at the media file after the plugins run, we can see tha the date has not been changed

I will PM that info...

[MisterMoonlight]

After discovering that your plugins appear to do the job as expected for the date changes and no errors can be seen from the log file, i have made a little experiment.

I attempted to modify the media file manually and save it on the reader.

I have discovered an unexpected behavior:

The media.xml file is rebuilt by the reader everytime after the modification. If the reader is restared with the manually modified file (verified to be ok after manual modifications), it will simply override the modification because the file is rebuilt all the time!!!

So the real question now is why aren't you seeing this behavior yourself???

I have the latest sony firmware v1.0.00.14080 with the PRS+ v2.0.12 beta 3rd party addon...

[kpw]

Originally Posted by MisterMoonlight

So the real question now is why aren't you seeing this behavior yourself???

That is a very good question! I wonder if the relevant factor is that you have PRS+ installed. I'm using the standard Sony firmware, so there may be some different behaviour there.

I haven't heard from anyone else using the plugin with PRS+, so I don't really know how they would work together. My unbderstanding with PRS+, though, is that it leaves the standard firmware intact and just provides some additional function, but I don't really know a lot about it.

Another thought I had is that perhaps the file updates are being cached in memory at some level (maybe at the OS). If that's the case and the device is unplugged without doing a "safe remove", either through Calibre or through Windows, then the updates may be lost and you would end up with the original file content. That sounds like a bit of a stretch to me though, as I would expect that to be sporadic and not consistent like you are seeing. But I thought I'd point the possibility out. If you are doing a "safe remove", then this shouldn't really be the problem though...

I'm not completely sure where to go with this right now. I'll do some research into file write caching, and also with PRS+, to see if I can get any bright ideas.

[DoctorOhh]

Originally Posted by kpw

That is a very good question! I wonder if the relevant factor is that you have PRS+ installed. I'm using the standard Sony firmware, so there may be some different behaviour there.

This is easy enough to check out.

MisterMoonlight shutdown your device then turn it back on while your device is connected to your PC. This should bring it up in safe mode with no PRS+ showing or involved. Now see if the problem still happens. When you're done testing shutdown your device and turn it back on not connected to your computer and PRS+ will be back as usual.

[kpw]

Hi MisterMoonlight,

I don't have anymore real information to pass on, but I do have another theory that I'd like to throw out there. I'll give you my "bottom line" first before I ramble on with my rationale for why I think it might help...

Bottom Line: It might be worth your while to check your reader drive file system for errors, using the Windows "Check drive" facility (I'm not sure exactly what it's called in Win XP). If there are errors, let the tool fix them and then try things again.

Ramble: From everything we've seen, I'm getting the feeling that we have a problem specifically with the media.xml file, and that might be preventing updates from being applied to it. Here are the reasons I have for this:

• I noticed something unexpected in the last trace file you sent me. The trace dumps out the file stats for the book files whose timestamp get updated, both before and after the timestamp adjustment. What I expected to see (and what I see on my system) is that the "before" stats should have the original timestamp that matches the cache (i.e. media.xml), and the "after" stats should show the updated timestamp. What I see in your trace though, is that both "before" and "after" stats reflect the updated timestamp. I think we're getting this because the file timestamps were updated successfully on previous runs of the plugin action. So that tells me that the EPUB file updates are successfully going through.
• With the stock Sony firmware, when it does a scan of the book files on the drive, it detects if any timestamps differ from the cache entries for those books. If it finds a discrepancy, it updates the record in the media.xml file with the right "date" value. This is not happenning with your reader since the media.xml file continues to show the old "date" values for the updated books. Perhaps this means that the Sony firmware is attempting to update the file but is running into the same problem the plugin is?
• I understand you're running PRS+, but I would be really surprised if that would interfere with the base Sony metadata processing, so I wouldn't expect the behaviour you're getting to be normal.
• I'm not sure what would cause the media.xml file to not accept updates, but perhaps some file system errors are getting in the way. With caching at the OS or device driver level, maybe write errors don't get caught at the Python level (this is pure speculation, since I don't know what happens at that low level).

Anyway, I'm guessing a "Check Drive" won't hurt anything. If the drive is clean, then we can just ignore my rambling and scratch our heads some more...

[MisterMoonlight]

You have forgotten an important detail: i have tried to manually edit the file and i can really write it and it is rewritten after device boot.

I have done a check disk on it also and got no errors.

I have made this test:

1- I have a copy of the manually modified media file (with yesterday date). I delete the media.xml file on the reader, take a copy of my modified backup file to media.xml (which is now yesterday date). Than i compare the backup and the media.xml. They are the same dated from yesterday. Then i safely remove the drive, shutdown the devices and after restarted the media.xml has been modified to the reader and becoming different from the manually backup file i used few minutes ago. I can see also that the new media.xml file after reboot has the date/time matching when it was rebooted.

The file "cacheExt.xml" is modified also on the reader after reboot...

I have tried to power up without PRS (usb plugged) and it changes nothing at all...

Conclusion: media.xml file is rewritten with old date data from my 3 choosen books by the reader itself !!!

[MisterMoonlight]

Ok

Excuse me for this stupid comment, but why not updating the 3 choosen books date instead (touch the date to most recent date) on the device instead of playing with the media.xml and getting a risk that any present of future firmware get into problem with that file

If i am touching the books date manually in the reader, after restarting the reader promotes them to the 1st page without any need to touch the media.xml file...

[kpw]

Originally Posted by MisterMoonlight

Ok

Excuse me for this stupid comment, but why not updating the 3 choosen books date instead (touch the date to most recent date) on the device instead of playing with the media.xml and getting a risk that any present of future firmware get into problem with that file

If i am touching the books date manually in the reader, after restarting the reader promotes them to the 1st page without any need to touch the media.xml file...

The plugin does that, in fact. The media.xml file update is just synchronizing the cached date with the updated file date. There are some side effects to letting the reader do that instead (if you use the "Author sort for author" option), so I update the cache file to avoid those side effects.

It's interesting that manually "touching" the epub files is working for you, but the programatic "touch" that the plugin is doing is not. I'm not sure why that would be happening. And I'm really confused about how the media.xml file gets reset to the old content.

By the way, I installed PRS+ on my reader and the plugin is working fine so far. I'll keep trying it out to see if it changes, but so far, it looks like there is no conflict.

In any case, I'm not sure what else to try at this point, but I find your latest experiment interesting, so I'll mull that over...

[MisterMoonlight]

In your tests, try to set the device language to french to see if it causes any problem...

[MisterMoonlight]

The plugin does that, in fact. The media.xml file update is just synchronizing the cached date with the updated file date. There are some side effects to letting the reader do that instead (if you use the "Author sort for author" option), so I update the cache file to avoid those side effects.

Please give me additionnals details about the side effect. If the reader is set to sort by date (and this is what we are attempting to made the date more current date), the most recent book will be promoted to the 1st page (this is a feature of the sony firmware). So i don't really understand what are the possible side effects

[kpw]

Originally Posted by MisterMoonlight

Please give me additionnals details about the side effect. If the reader is set to sort by date (and this is what we are attempting to made the date more current date), the most recent book will be promoted to the 1st page (this is a feature of the sony firmware). So i don't really understand what are the possible side effects

The main side effect involves the "Use author sort for author" setting in the Calibre Sony device interface plugin configuration. When you have this set, the Sony interface plugin sets the cache author field in the media.xml (or cache.xml for SD/MS) for a book entry to the "author sort" value rather than the default "author". So in the reader, the author shows up in the form "LN, FN" rather than "FN LN" (or whatever the author sort value happens to be, presumably "last name, first name"). This is done while the reader is connected to Calibre, of course, so it is working with cache data that has already been processed by the Sony firmware. When the book timestamp is changed, if the cache isn't synchronized with that date, then the Sony firmware will detect that the the book has changed and will process the book entry to synchronize it with the book file metadata. As part of this synchronization, in addition to updating the date field, it also resets the author field to the author value in the book metadata, which is not "author sort". So to prevent the book record from being processed, the plugin synchronizes the date. Then the Sony firmware thinks everything is fine and it doesn't override what Calibre has set.

By the way, I'll try the French setting on my reader and see if that changes anything. I'll keep you posted on how that goes.

[kpw]

Originally Posted by MisterMoonlight

In your tests, try to set the device language to french to see if it causes any problem...

I tried setting to French (General Settings/Menu Language - that was the only language setting I could find). Rebooted the device to have the change take effect. Things still worked fine with the plugin, so this didn't have any impact on my reader.

[kpw]

Originally Posted by MisterMoonlight

If i am touching the books date manually in the reader, after restarting the reader promotes them to the 1st page

Hello MisterMoonlight,

I've had to make an update to my plugin to fix a problem in Ubuntu (just installed that recently, so I've only just ran tests on that platform). The update required is to ensure that any time I update the timestamp on a book EPUB file (i.e. "touch" the file) the new timestamp is never earlier than the current computer date/time.

Given the above test you ran where a manual "touch" of an EPUB file resulted in more satisfactory results, it's possible on your environment that "touch"ing the file with an older date/time value was problematic. I assume the "touch" you performed was using the current system time.

It's a long shot, but I no longer have a WinXP environment to verify this conjecture on (I replaced my XP with Ubuntu). If you haven't given up on it altogether here's my plugin with the fix that you're welcome to try. I'll be posting it in a day or two after I do some more thorough testing.