0.7 -- Sending metadata to device is SSSSLLLOOOOOOOOOOW

[jotran2003]

I think there's a problem with the beta version.
Two records of the same book is displayed.

[DoctorOhh]

Originally Posted by jotran2003

I think there's a problem with the beta version. Two records of the same book is displayed.

That's a little too vague to be of any help.

Are you saying out of 100s or 1000s of books you have that one is displayed twice? Or are you saying every book on your device is displayed twice or are you saying ....?????

Please add some detail to help the developers.

[grizedale]

I have installed the beta and thank you the 'Save to disc' function now works.

There is still a very long loading time once I unplug the USB though, in 0.6.55 it is 28 seconds, in this Beta it is 2 minutes:45 seconds.

Re the loading time, I have noticed that 0.7.1 puts a metadata.calibre in the root of my device, what purpose does this serve because I have deleted it to no detriment and the loading time is now back to 28 seconds.

[chaley]

Originally Posted by jotran2003

I think there's a problem with the beta version.
Two records of the same book is displayed.

Do you two formats of the book on the device? For example, mobi and epub?

[chaley]

Originally Posted by grizedale

I have installed the beta and thank you the 'Save to disc' function now works.

There is still a very long loading time once I unplug the USB though, in 0.6.55 it is 28 seconds, in this Beta it is 2 minutes:45 seconds.

I assume you have a Sony? The Sony reader decides from time to time to rescan all the books in the library. I know this happens when Sony's book ID changes, but it can happen at other times for mysterious reasons.

Calibre will change the Sony ID if there is a mismatch between Calibre's cache (the .metadata.calibre file you mention below) and the Sony database. A mismatch will always occur on the first insertion into 0.7. It can occur at later times if the metadata for a book changes.

Re the loading time, I have noticed that 0.7.2 puts a metadata.calibre in the root of my device, what purpose does this serve because I have deleted it to no detriment and the loading time is now back to 28 seconds.

That file is calibre's metadata cache. If you delete it, calibre will rebuild it on next connect by scanning all the books on the device, considerably slowing down calibre-to-device connection. In addition, calibre will resend metadata for all files on the library that it also finds on the device, which may trigger collection rebuilding on the Sony. Finally, for books not matched, the collection information could be changed back to what they would have been when the book was originally put on the device.

The reason the Sony load time went back to 28 seconds is that during that connection, the Sony DB did not change in ways that forced a rescan. If a collection rebuild occurs, and if it changes anything, the Sony load time will be long on the next disconnect.

All in all, deleting that file is counter-indicated.

[grizedale]

but it is only 0.7.2 that puts the metadata.calibre file on my device, I have gone back to 0.6.55 and this file does not appear!

[chaley]

Originally Posted by grizedale

but it is only 0.7.2 that puts the metadata.calibre file on my device, I have gone back to 0.6.55 and this file does not appear!

Caching was added in 0.7.0. All versions of calibre from here on will have it.

[grizedale]

Originally Posted by chaley

Caching was added in 0.7.0. All versions of calibre from here on will have it.

Can there not be an option not to have this file installed? if not I will not be upgrading anytime soon, this is not a problem as 0.6.55 does everything I require.

I have also noticed that each time I open a new book on the device it formats for a short while and, again, this does not occur in 0.6.55

[chaley]

No. That file is critical to calibre's function. You are of course welcome to stay with the 0.6 series. Note, however, that there will be no further maintenance of it.

I don't understand your objection to the file's presence. On my PRS-300, I ran the following experiments (100 books on my device):
1. File present. Connect to the computer w/o calibre running. Disconnect. Sony time: 9 seconds.
2. Connect to the computer w/o calibre running. Delete the file. Disconnect. Sony time: 9 seconds.
3. Connect to the computer. Start Calibre. Calibre rebuilds the file. Disconnect. Sony time: 40 seconds. (This is expected because the file was rebuilt.)
4. Connect to the computer. Start Calibre. Disconnect. Sony time: 9 seconds.
5. Connect. Start Calibre. Add a book. Disconnect. Sony time: 41 seconds (it did a rescan).
6. Connect, calibre, disconnect. Sony time: 10 seconds
7. Connect, calibre, delete a book, disconnect. Sony time: 43 seconds (another rescan)
8. Connect, calibre, disconnect. Sony time: 9 seconds
9. Connect, delete the file, disconnect (no calibre): 10 seconds

As you can see, the presence or absence of the file makes no difference at all in the sony disconnect time. Running calibre also makes no difference. What makes a difference is changing something on the device.

[grizedale]

Chaley, thank you for your time and comments.

Looking at your results I could only assume it was something on my device causing this problem so, I backed up all my books, formatted my device, and, using version .7.1 reintroduced all my books and, I am happy to say, I am now getting the same reults as you.

Thanks again

[tafweb]

Originally Posted by jotran2003

I think there's a problem with the beta version.
Two records of the same book is displayed.

I am getting the same. When I upload a revised version of a book Calibre seems both to update the existing record and also create a new record for the book, hence the book appears twice in Calibre even though there is only one copy on my Sony PRS-505. If I delete one of the records, both disappear! This is not a real problem as long as I remember to delete the original entry before uploading the revised one.

[chaley]

Originally Posted by tafweb

I am getting the same. When I upload a revised version of a book Calibre seems both to update the existing record and also create a new record for the book, hence the book appears twice in Calibre even though there is only one copy on my Sony PRS-505. If I delete one of the records, both disappear! This is not a real problem as long as I remember to delete the original entry before uploading the revised one.

Are you saying that it always creates two lines when you send an updated book to the device?

If so, have you changed the send-to-device template? Or used some other software to load the books? What I am really wondering is whether the book is in fact on the device twice, but in two different places. If this is true, things will get confused. Calibre's author and title matching will match both to the same book, probably creating two lines that could end up pointing at the same file. The Sony software will find both and create two records, but its collections will likely get messed up.

If you delete one in calibre then disconnect, the other should come back on the next connection.

Bottom line: if you change naming conventions, it is best to clean all books off the device and reload. If you use more than one piece of software to manage the device, ensure that the same book is not added by both of them.

[tafweb]

Originally Posted by chaley

Are you saying that it always creates two lines when you send an updated book to the device?

If so, have you changed the send-to-device template? Or used some other software to load the books? What I am really wondering is whether the book is in fact on the device twice, but in two different places. If this is true, things will get confused. Calibre's author and title matching will match both to the same book, probably creating two lines that could end up pointing at the same file. The Sony software will find both and create two records, but its collections will likely get messed up.

If you delete one in calibre then disconnect, the other should come back on the next connection.

Bottom line: if you change naming conventions, it is best to clean all books off the device and reload. If you use more than one piece of software to manage the device, ensure that the same book is not added by both of them.

Here is an example:

1. Send new book to device - result: one entry on the list in Calibre for Card B.
2. Send the same book to device again - result: two entries on the list in Calibre for Card B - both showing the book to be in the same location on the device. Confirmed only one book present on the actual device.
3. Disconnect device
4. Reconnect device - result: the book is still listed twice in Calibre.
5. Delete one of the entries in Calibre - result: error message as follows:
   

   =====================================
   ERROR: Error: Error communicating with device
   (3, 'The system cannot find the path specified')
   
   Traceback (most recent call last):
     File "site-packages\calibre\gui2\device.py", line 62, in run
     File "site-packages\calibre\gui2\device.py", line 288, in _sync_booklists
     File "site-packages\calibre\devices\prs505\driver.py", line 151, in sync_booklists
     File "site-packages\calibre\devices\prs505\sony_cache.py", line 343, in update
     File "site-packages\calibre\devices\prs505\sony_cache.py", line 426, in update_text_record
     File "genericpath.py", line 64, in getctime
   WindowsError: [Error 3] The system cannot find the path specified: u'S:\\Bairnsfather, Bruce\\Bullets and Billets\\Bullets and Billets - Bairnsfather, Bruce_2010.lrf'
   ==========================================

   The book is no longer present on the device but there is still one entry in Calibre's list for Card B.
6. Delete remaining entry from Calibre's list - result: everything now back to normal.

[chaley]

Thank you for the details!

It seems that while fixing a problem with metadata updating, I broke the test for identical books, with the result that two entries were put into the cache. This regression appeared in the 0.7.2 beta, and will disappear in the official 0.7.2 release.

If you find yourself in the situation where you have two lines for the same book, you can fix it by deleting the .metadata.cache file. You will, of course, pay the scan time penalty on your next connect to calibre, but you won't have doubled books.

[slantybard]

chaley - I was curious if you can comment on the philosophy behind changing how calibre handles metadata and the devices? Ie, why change from 0.6 series method and the new 0.7 series method. Thanks for any info.