Metadata Updates

[gsparks2]

How do CC & Calibre determine if metadata needs updating on the device? Does it look at the time stamps of the file or does it do a detailed comparison between Calibre and CC?

Do CC/Calibre use the same date-time when writing book files to the device?

Reason I am asking: I was examining the files in the device folder and noticed that some of the files had times that were in the future. Not sure how that happened.

Edit: both the device and pc date/times were set to the same time zone with network updates of time.

[chaley]

Originally Posted by gsparks2

How do CC & Calibre determine if metadata needs updating on the device? Does it look at the time stamps of the file or does it do a detailed comparison between Calibre and CC?

The driver in calibre compares the last_mod time from calibre's library to the last_mod time that CC received during the last connect. If they are different (larger or smaller) then the driver sends new metadata to CC.

Do CC/Calibre use the same date-time when writing book files to the device?

No. The timestamp of the format on the device is controlled entirely by Android.

Reason I am asking: I was examining the files in the device folder and noticed that some of the files had times that were in the future. Not sure how that happened.

Neither am I. As I said above, CC never directly sets the mod time for a file, instead depending on Android to take care of that when a file is created/opened for write. The implication is that during at least one connect the time on the device was not correct.

Edit: both the device and pc date/times were set to the same time zone with network updates of time.

While investigating your questions I did discover that CC is sending the file modtime as UTC but the driver in calibre was interpreting it in the local time zone. This means that the auto-format-sync feature might get confused for some time if a format is changed in calibre and immediately synced to CC. I can imagine situations where the file isn't sent for up to 12 hours, and others where the file is sent on every sync for up to 12 hours. I will submit the driver fix to Kovid.

[gsparks2]

Thank you.

Edit: The last paragraph described a situation that may be causing metadata to be repeatedly updated on sequential connections following initial transfer to device.

(Still trying to get all my books on the backup reader, almost up to 16k now)

[chaley]

Originally Posted by gsparks2

Edit: The last paragraph described a situation that may be causing metadata to be repeatedly updated on sequential connections following initial transfer to device.

You are referring to the paragraph about UTC vs local time? That problem affects only book (format) syncing, not metadata syncing.

It is very easy to change the mod time for a book without directly changing the book. For example, changing the case of a tag in book 1 will change the mod time of every book with that tag. The same thing is true of any shared value such as series or authors. Changing the template of a column built from other columns changes the mod time for every book in the library.

[gsparks2]

ok, thanks.

What I am seeing is:
Connect CC to Calibre as wireless device.
Select books in Calibre to send to device.
Calibre sends books to device.
Calibre sends metadata to device.
<<connection lost for any number of reasons>>
Reconnect CC to Calibre as Wireless device.
CC sends metadata of every book to Calibre.
Calibre analyzes books on device.
Calibre sends metadata back to device.

During all of this, I have not changed any metadata in Calibre nor have I changed the book files.
Edit: I may be here: http://www.mobileread.mobi/forums/sh...d.php?t=220906

[chaley]

Originally Posted by gsparks2

ok, thanks.

What I am seeing is:
Connect CC to Calibre as wireless device.

At this point, no metadata is exchanged?

Select books in Calibre to send to device.
Calibre sends books to device.
Calibre sends metadata to device.
<<connection lost for any number of reasons>>
Reconnect CC to Calibre as Wireless device.
CC sends metadata of every book to Calibre.

This is very significant. More below.

Calibre analyzes books on device.
Calibre sends metadata back to device.

Calibre sends all metadata?

CC sending all metadata to calibre implies that the device book cache in the driver in calibre is corrupt. In this case it throws it away and asks for a new copy. The only way I know that it could become corrupt is for calibre to run out of memory or disk space.

Are you running calibre x64?

Are any of your drives running low on disk space? (I don't remember what OS you are running?

Edit: I may be here: http://www.mobileread.mobi/forums/sh...d.php?t=220906

Possibly. Most of the underlying causes were fixed long ago. On the other hand, Gingerbread devices tended to be light on memory, and Gingerbread memory management is rather flawed, so we could be running into out-of-memory cases. These will be noted in CC's log file, and if they happen during connections then the connection is usually rudely and abruptly broken.

If you don't mind, the best way to proceed here is for me to see debug logs both from calibre and from CC. Instructions are in our FAQ answer How do I get help? Please remove the multipie.co.uk address when you send CC's debug log. No need to bring my son into this.

[theducks]

This may be similar to what happens to time when you dual boot Linux and Windows.

Windows sets the Clock chip in the PC to Local time.
Linux use UTC and adjusts the display by Locale

There is a Registry setting you can add to make Windows use UTC

BTW I used to see the future date when my Astak USB worked (broken now traces and I only use the SD card)

[gsparks2]

Originally Posted by chaley

At this point, no metadata is exchanged?
This is very significant. More below.
Calibre sends all metadata?

CC sending all metadata to calibre implies that the device book cache in the driver in calibre is corrupt. In this case it throws it away and asks for a new copy. The only way I know that it could become corrupt is for calibre to run out of memory or disk space.

Are you running calibre x64?

Are any of your drives running low on disk space? (I don't remember what OS you are running?
Possibly. Most of the underlying causes were fixed long ago. On the other hand, Gingerbread devices tended to be light on memory, and Gingerbread memory management is rather flawed, so we could be running into out-of-memory cases. These will be noted in CC's log file, and if they happen during connections then the connection is usually rudely and abruptly broken.

If you don't mind, the best way to proceed here is for me to see debug logs both from calibre and from CC. Instructions are in our FAQ answer How do I get help? Please remove the multipie.co.uk address when you send CC's debug log. No need to bring my son into this.

On your first question, the sequence begins with no books on device before sending books to device, so no metadata transfer.

Not all Metadata is resent. Just most of it. (this time 14900 of 16057).

I will send debug logs. It may be tomorrow as I am in a lengthy run of calibre sending metadata to device. If/when it stops again I will send logs.

Device is the Android Gingerbread device. Device has 512 MB ram and 16 GB internal with 13.3 GB free. Book files are going on 16 GB external SD card with 6 GB free after loading 16057 books.

Calibre is x64 latest version 2.26

My PC runs Windows 7 Professional with 9 GB ram and first gen core i9 processor. while processing, calibre is using 1.1 GB memory. PC has ~5 GB memory in standby.

[gsparks2]

Originally Posted by theducks

This may be similar to what happens to time when you dual boot Linux and Windows.

Windows sets the Clock chip in the PC to Local time.
Linux use UTC and adjusts the display by Locale

There is a Registry setting you can add to make Windows use UTC

BTW I used to see the future date when my Astak USB worked (broken now traces and I only use the SD card)

Thank you. I do not have dual boot on my PC.
Interesting thought on using UTC on PC. I am in US central time zone. I wonder if I can adapt.

[chaley]

Originally Posted by gsparks2

Interesting thought on using UTC on PC. I am in US central time zone. I wonder if I can adapt.

I don't see any reason why you should bother.

[gsparks2]

me either

[gsparks2]

Charles,
I cannot duplicate the situation.

[chaley]

Originally Posted by gsparks2

Charles,
I cannot duplicate the situation.

Thanks for letting me know. I suppose this is actually good news.

[gsparks2]

chaley,
Is this the expected behaviour of CC and Calibre?

Calibre Job times:
Start with 8940 books on device.
Upload 60 books to device 2m 44s
Send metadata to device 0m 19s
<disconnect CC>
<Wait ~30s until Calibre disconnects.>
<connect as Wireless Device in CC>
Get device information 0m 18s
Set library information 0m 1s
Get list of books on device 0m 43s
<during this time CC quickly sends metadata for the 60 books to Calibre, less than 10 seconds then Calibre analyses the books on device>
Send metadata to device 2m 30s
<during this time CC displays progress bar for receiving metadata for the 60 books>
Total elapse time: ~6m 35s

It seems that metadata is sent several times back and forth. I am just trying to understand what is going on.

Thanks!

[chaley]

I am away with no access to my computers, so this is from memory.

Originally Posted by gsparks2

chaley,
Is this the expected behaviour of CC and Calibre?

Calibre Job times:
Start with 8940 books on device.
Upload 60 books to device 2m 44s
Send metadata to device 0m 19s
<disconnect CC>
<Wait ~30s until Calibre disconnects.>
<connect as Wireless Device in CC>
Get device information 0m 18s
Set library information 0m 1s
Get list of books on device 0m 43s
<during this time CC quickly sends metadata for the 60 books to Calibre, less than 10 seconds then Calibre analyses the books on device>

I would not expect CC to send any metadata at this time. It must send some base information to tell calibre what it needs to know to check its cache. If metadata for new books is being sent back to calibre then the last mod time in calibre's cache is being changed after the book is sent. How is a mystery. Do you have the reading list plugin installed with it keeping track of what is on the device? That would update metadata for the newly sent books at some point in the process.

Send metadata to device 2m 30s
<during this time CC displays progress bar for receiving metadata for the 60 books>
Total elapse time: ~6m 35s

This is consistent with the above: calibre and CC disagree about the metadata the cc has. Calibre must resend it.

It seems that metadata is sent several times back and forth. I am just trying to understand what is going on.

Thanks!

You can get a glorious amount of detail of you run calibre in debug mode. It will log the state of the cache, how many books are in it, how many are not, what it sends, and in many cases why.