Tag troubles

[chaley]

Originally Posted by JvdW

Now I should tell you that I use Calibre on more than one computer to sync news, books etc. The first example is from a system at home (0.7.2?) and the second is from Calibre at work (0.7.2). I Think that might complicate things since at work I can plugin my Reader and have Calibre detect it, unplug it and it scans rather quickly then (10secs). Only the first time it took a long time and only then are the dates changed.
The problem is I can't figure out where those dates are coming from since those books are on my Reader, not in the library at work but in my library at home. (which on second thoughts my not be true, will check and report if relevant)

The dates come from the books on the device. Calibre's library is irrelevant.

The dates are checked (and possibly changed) on the first scan, whenever a book is added or deleted, or whenever collections are changed. In these cases, calibre scans the books folders on the device for files (books), gets the 'ctime' date from the file, breaks it into components (year, month, etc) using the local timezone (required by FAT file systems), then creates a date string using the format

"%w, %d %m %Y %H:%M:%S GMT"

It appears that one of your machines is reporting an incorrect ctime for the books. It isn't reasonable that all were created on 1 Jan 2008 at noon. That looks like someone's idea of a good default date.

What OS are you running at work and at home? Are both of them locale-aware and do both have valid locales? From what you say, it appears that the work machine is doing it right, but the home machine is not.

[JvdW]

Originally Posted by chaley

The dates come from the books on the device. Calibre's library is irrelevant.

What OS are you running at work and at home? Are both of them locale-aware and do both have valid locales? From what you say, it appears that the work machine is doing it right, but the home machine is not.

Sorry, I shoud learn to give all relevant information instead of rambling about.
Work OS: WinXP SP3
Home OS: Win 7

Will have a look at the locales.

FYI: I just uploaded my NRC Handelsblad.epub of today at work to my Reader and it took more then a couple of seconds to get to the start screen. Didn't time it but probably around 1:30 or more.

Only change in media.xml is the addition of the said epub and a change in counter at the top of the file:

<xdbLite xmlns="http://xscool.net/xdb/1" xmlns:xs1="http://www.kinoma.com/FskCache/1" default="string" nextID="110">

nextID=112 on my Reader now.

Regards,

Joop

[chaley]

Originally Posted by JvdW

FYI: I just uploaded my NRC Handelsblad.epub of today at work to my Reader and it took more then a couple of seconds to get to the start screen. Didn't time it but probably around 1:30 or more.

Only change in media.xml is the addition of the said epub and a change in counter at the top of the file:

<xdbLite xmlns="http://xscool.net/xdb/1" xmlns:xs1="http://www.kinoma.com/FskCache/1" default="string" nextID="110">

nextID=112 on my Reader now.

There must be another change somewhere, because 2 IDs were used. It is probably a new collection, but it is worth being sure.

To verify, you copied the original of the media.xml from the device just before starting calibre to put the book on the reader? My concern is that if it is a copy from some time back (like from before you put the device into your home machine), you might be comparing to the 'right' one when the actual media.xml before insertion was different.

You can test if the Sony SW is rescanning the library by sending some .txt file to the device. Create a small file, import it into calibre, set its author and title to something other than Unknown (e.g., Joe Blogs, My Precious Text File) and send it to the device. After you unplug the device, check on it to see if the author and title are still there. If they are, then Sony and calibre are in agreement on the date. If they are not, then Sony and calibre had a fight, and Sony won. This test works because .txt files contain no metadata, so a Sony scan with mismatching dates will reset the metadata to Sony defaults.

[JvdW]

Originally Posted by chaley

There must be another change somewhere, because 2 IDs were used. It is probably a new collection, but it is worth being sure.

To verify, you copied the original of the media.xml from the device just before starting calibre to put the book on the reader? My concern is that if it is a copy from some time back (like from before you put the device into your home machine), you might be comparing to the 'right' one when the actual media.xml before insertion was different.

Yes, I copied the file from the device before starting Calibre.

You can test if the Sony SW is rescanning the library by sending some .txt file to the device. Create a small file, import it into calibre, set its author and title to something other than Unknown (e.g., Joe Blogs, My Precious Text File) and send it to the device. After you unplug the device, check on it to see if the author and title are still there. If they are, then Sony and calibre are in agreement on the date. If they are not, then Sony and calibre had a fight, and Sony won. This test works because .txt files contain no metadata, so a Sony scan with mismatching dates will reset the metadata to Sony defaults.

Did just that and the file is still there in media.xml. The reader shows the metadata set in Calibre.

I had a look at the locale settings but they are the same between my 3 systems.

Also started searching for the mismatch in dates between one system and the other and think I might have found it. Due to setting the alternate date format in PRS+ and a bug in that part of the code (solved in latest release) I had a bit of an accident. The creation date is 16 may 2010 but the modification date is 1 jan 2010. Looks like Windows 7 and Windows XP in combination with python end up using a different date in media.xml causing the Reader to scan all books to see what is right.
I think if I erase my reader and re-upload its content everything should be fine again. Its just a little late right now in my timezone so that will have to wait til tomorrow.

Regards,

Joop

[DoctorOhh]

Originally Posted by JvdW

Sorry, I shoud learn to give all relevant information instead of rambling about.
Work OS: WinXP SP3
Home OS: Win 7

Will have a look at the locales.

FYI: I just uploaded my NRC Handelsblad.epub of today at work to my Reader and it took more then a couple of seconds to get to the start screen. Didn't time it but probably around 1:30 or more.

I'm sorry if you already stated this info. Are you accessing the exact same library folder from each computer? If not how are you syncing the two library folders?

[JvdW]

Originally Posted by dwanthny

I'm sorry if you already stated this info. Are you accessing the exact same library folder from each computer? If not how you syncing the two library folders?

No, I'm accessing different libraries. Is that a problem then because I never had these long timeouts when I was using Calibre 0.6.x on all systems.

I'll backup my books from IM and SD card and format both and re-upload the books to see if the problem goes away. Looks like one OS uses ctime and the other mtime. Can imagine that since you can't have a ctime after mtime (c=create, m=modify)

Regards,

Joop

[DoctorOhh]

Originally Posted by JvdW

No, I'm accessing different libraries. Is that a problem then because I never had these long timeouts when I was using Calibre 0.6.x on all systems.

Calibre 0.6.x didn't try to cache the metadata on the device to save time.

The first time you connect your device to a calibre's library calibre creates a file on your device that in the future is used to cache info and make things quicker. Thus the first connect is much longer than subsequent connects.

If you change to a different library the info is going to be different and ....

I'll let Chaley address this, I might not even be in the ballpark.

[chaley]

Originally Posted by JvdW

I'll backup my books from IM and SD card and format both and re-upload the books to see if the problem goes away. Looks like one OS uses ctime and the other mtime. Can imagine that since you can't have a ctime after mtime (c=create, m=modify)

I was wrong when I said ctime. Calibre uses mtime.

Your point could be significant. FAT and FAT32 provide both mtime and ctime. The two will be the same when a book is first sent to the device. However, when you resend a book with calibre, it does an open w/truncate, which should change the mtime but not the ctime. If (some version of) the sony SW uses ctime, updating a book will get calibre and sony permanently out of sync.

I will try some experiments on my 300, but what I see may not be the same behavior as other devices.

Perhaps the best thing to do would be to delete before add, which would result in the two timestamps always being the same. There would be no visible performance penalty (milliseconds), because writing the file takes the vast majority of time and both methods must do that.

[chaley]

I performed two experiments with my PRS-300 on Win7 with standard firmware. The first experiment was sending a text file to the device, unplugging it, replugging, and sending it again. The second was resetting the mtime of the text file to what it was before sending it the second time.

The first experiment shows that after the first sending of the text file, the author and title on my Sony are what I set in calibre (correct) and the ctime and mtime are identical (correct). After sending it again:

• the ctime does not change (correct),
• the mtime does change (correct),
• the author and title don't change (correct)

After changing the ctime and mtime (the second experiment):

• the Sony reset the time in its database to the new mtime (correct)
• the author became blank (correct)
• the title became the file name (correct)

In all cases the time to disconnect was 15 seconds +/- 1 second.

The experiments show that my Sony is using mtime instead of ctime, and that if the mtime in the database does not match the mtime of the book file, Sony rebuilds the metadata in its database by scanning the books. Because the book in the experiment is a text file, the only metadata the Sony SW can find is the file name. Conclusion: my 300 is behaving as expected and desired.

Your connecting the device to two different machines does raise a possibility. Calibre's preferred matching method for for the on-device function is the book's UUID (unique ID). This ID is assigned by calibre when the book is imported, assuming that the book doesn't already have one.

If a book on the device has a UUID different from the book in the library, title and author matching is used. If these match, then the metadata from that library is sent back to the device to correct the UUID. As a side effect, the calibre device driver scans the Sony db to ensure that the titles and authors agree with the new metadata. During this process, the mtime on the device is fetched and written to the Sony db using the format %w, %d %m %Y %H:%M:%S GMT (dayname, daynumber monthname yearnumber hour:minute:second GMT).

If the two machines use a different method for obtaining the date, this process can result in confusion. If one machine uses ctime and one mtime (it is hard to believe this could happen, but it is possible), then when the device is plugged into the one using ctime, the Sony will rescan the library when the device is disconnected. It should not rescan again on that machine until a book is added or changed, or a collection changed (anything that sends metadata).

When the device is connected to the other machine, the UUIDs could change, causing the process to start over again. The Sony database will be checked and modified as needed. Because the assumption is that this second machine uses mtime, the sony should not rescan on this machine. However, because the UUIDs changed, the device will rescan when connected & disconnected from the first.

If the custom firmware uses ctime, then the Sony will rescan every time you change machines, or add/change a book or collection. If the custom firmware has a date string format different from the Sony standard firmware, then the device will rescan under the same conditions.

[JvdW]

Originally Posted by dwanthny

Calibre 0.6.x didn't try to cache the metadata on the device to save time.

The first time you connect your device to a calibre's library calibre creates a file on your device that in the future is used to cache info and make things quicker. Thus the first connect is much longer than subsequent connects.

The problem I'm having is not that Calibre takes a lot of time to correct its cache file but that there seems to be a conflict in the generated media.xml file that the Sony software scans after the usb connection is disconnected.

Regards,

Joop

[mark2410]

ahh

seeing as it appears all the trouble ive been having its not a bug its a "feature"

i cant live without the reader library being able to connect and i cant have calibre wiping it every time it connects. i dont like this use only this app or you cant use it at all.

[chaley]

Originally Posted by mark2410

ahh

seeing as it appears all the trouble ive been having its not a bug its a "feature"

i cant live without the reader library being able to connect and i cant have calibre wiping it every time it connects. i dont like this use only this app or you cant use it at all.

Sorry, but what are you trying to say? What is the 'feature' you are referring to?

[mark2410]

Originally Posted by chaley

Sorry, but what are you trying to say? What is the 'feature' you are referring to?

calibre wiping everything not in its library, ie anything thats been added using the sony reader library gets deleted

[chaley]

Other than yours, I know of no assertion that calibre has *by itself* deleted a book from a library. We *would* hear about it, because calibre has thousands of users and if it was deleting books without permission, it would be considered highly antisocial.

I recognize that it is probably useful to you to make posts like this (makes you feel better, hmmm?). That said, it would be even more useful if you provided some information about the event, such as what was lost and what you were doing. "Everything" is a lot.

[DoctorOhh]

Originally Posted by mark2410

calibre wiping everything not in its library, ie anything thats been added using the sony reader library gets deleted

Calibre has never caused a single book to be deleted.

It is just not in its code, which is open for all to see. Ignorance may be bliss but when you expose that ignorance in writing on a public forum it becomes libelous.

If you explain what actually happened, maybe we can shed light on this perception that books were deleted and possibly create a better program in the process.