View Single Post
Old 06-29-2010, 06:15 AM   #42
chaley
Grand Sorcerer
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 12,485
Karma: 8025704
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
@FizzyWater: Thank you for sending me the log. I found a (i hope 'the') problem. It was in a place I never would have looked, and has nothing to do with Sonys and collections. A program component supplied by the Python people is using a very inefficient method to write to disks. Interestingly enough, they even say so in the middle of their code. (For those who want to know, the metadata cache code uses JSON, which is using unbuffered writes. Combine that with using unbuffered I/O with removable storage, the norm on XP I think, and you have performance death.)

Under Win7, both my phone card and my PRS-300 are forced to a performance-enhanced mode (write behind is forced on), so I can't completely measure the effects of the fix. However, even in my case I see a 10-times improvement, with 1500 books taking 1 second instead of 10. My guess is that you will see more improvement than that.

In addition, using your log and the one June sent earlier, I isolated two spots in the Sony driver that could be improved. Fizzywater's log indicates that building collections took 11 seconds for 1500 books. With the code changes, that time should drop to below 7 seconds, perhaps as low as 5. A drop in your bucket, but as one supermarket here says, "every little helps".

I have neither an XP machine nor a 505 to test with, so I can't be sure of the magnitude of the improvement. My guess is that in FizzyWater's case, the time will drop from approximately 220 seconds to approximately 25. It could drop to as low as 10, but I am not holding my breath.

@FizzyWater: the remapping messages in the log are not a problem. Remapping happens because calibre puts the books before the playlists in the Sony database, which can change their ID. It looks like the new book you added was given id 200. Also, the numbers in the front of the log are time in tenths of seconds from when the device was connected. Looking at the log, you will see that the vast majority of the time is taken between USBMS: starting sync_booklists and USBMS: finished sync_booklists. Fortunately for me, from calibre's point of view not much happens between those two messages, so it was relatively easy to chase the problem to its external source.

Last edited by chaley; 07-04-2010 at 06:30 AM. Reason: Fix June's name.
chaley is offline   Reply With Quote