Calibre 0.6.14 incorrect publish date

[elmoglick]

Just upgraded to the newest version and retrieved a number of newsfeeds. All were created with incorrect publish date metadata. It appears to ignore the current local time and uses GMT. However, the correct date is displayed within Calibre, but shows tomorrow's date on the Kindle. Metadata examination shows a 6 hour time discrepancy.

I've found a workaround - highlight each item in Calibre, click Edit Metadata, then just click OK to resave (the publish date is already correct).

El

[kovidgoyal]

To be clear, you're saying the date displayed in calibre is correct, but not on the Kindle?

[elmoglick]

Originally Posted by kovidgoyal

To be clear, you're saying the date displayed in calibre is correct, but not on the Kindle?

Exactly. I followed these steps:

1) I fetched a few newsfeeds. They all listed with correct date.
2) Plugged in the Kindle and waited until it showed up in Calibre
3) Highlighted the newsfeeds
4) Clicked on Send to Device

When I disconnected and then looked at the Kindle, all the newsfeeds I had just transferred had tomorrow's date.

I found that within Calibre, the publication date metadata is displayed properly, but checking the files with Mobi2mobi, they actually had tomorrow's date (off by 6 hours, to be precise, which is my timezone's GMT offset, coincidentally). If I then changed them to today's date, they would display correctly on the Kindle.

Also, I found that simply displaying and resaving each feed's metadata (in Calibre) corrected the metadata mismatch as well.

Pretty weird, eh?

[kovidgoyal]

Ah I will have to change the MOBI output code to put in local date/times. Open a ticket for it.

[alison87]

Was this bug ever resolved? This sounds a lot like the bug i am having here: https://www.mobileread.com/forums/sho...d.php?t=120474 Kovid, if i follow your advice and do "show info" in Calibre's ebook viewer it shows the UTC time whereas viewing the Calibre metadata shows the local time. It seems the MOBI is embedded with the incorrect time, which causes the Kindle to see the file as published on the wrong date.

[kovidgoyal]

I'm in GMT -7, I downloaded a periodical at 20:41 and sent it to my Kindle 3 and the Kindle 3 correctly shows the date as Mon, Feb 7

[alison87]

Hi Kovid, thanks for the quick reply! I really appreciate all the work you put into this great piece of software.

I have just tried the same thing (i am GMT-5) and it still showed up on my Kindle as tomorrow's date. The interesting thing is that if i turn off auto-send to my Kindle email address and let it sync via USB then the date is correct. Is there some additional metadata sent when syncing via USB?

[kovidgoyal]

Not that I'm aware of, when sending by email the file does get processed by amazon, so it may be that something is being changed by them. look at the file delivered by email and see the pubdate in it using calibre's viewer and then compare that to the same file sent by USB.

[alison87]

I just did some testing and i have confirmed that the USB sync is sending some additional metadata to the Kindle that the transfer by email is not. I'm including some output that might help in figuring out what is going on here:

This is from the mobi that is generated by Calibre and saved in the Calibre Library directory:

$ ebook-meta Seth\'s\ Blog\ -\ calibre.mobi 
Title               : Seth's Blog
Author(s)           : calibre
Publisher           : calibre
Comments            : Seth Godin's riffs on marketing, respect, and the ways ideas spread.
Language            : en
Published           : 2011-02-09T04:07:33.098608+00:00

Note that the published date here is in GMT. If i run ebook-meta against the metadata.opf file in the same directory it also shows the timestamp and published date in GMT.

This is from the azw that is received by the Kindle via Calibre sending to my @kindle.net email address:

$ ebook-meta Seth\ s\ Blog\ calibre\ azw-asin_37cbe674befbf88966412ed98abf292d73da35f8-0-azw-type_PDOC-v_0.azw 
Title               : Seth's Blog
Author(s)           : calibre
Publisher           : calibre
Comments            : Seth Godin's riffs on marketing, respect, and the ways ideas spread.
Tags                : News, Seth's Blog
Language            : und
Published           : 2011-02-09T04:07:33.098608+00:00

Note that almost all the metadata is retained from the original mobi file - including the GMT published date. Only the language is now undefined.

This is from the azw that appears on the Kindle after doing a USB transfer from Calibre:

$ ebook-meta Seth\ s\ Blog\ azw-asin_e95a6168f94c984e9375ae214f62e0d7b9d6614f-0-azw-type_PDOC-v_0.azw 
Title               : Seth's Blog
Author(s)           : calibre
Publisher           : calibre
Comments            : Seth Godin's riffs on marketing, respect, and the ways ideas spread.
Language            : en
Published           : 2011-02-08T23:03:47.132918+00:00

Note that here (unlike the email transfer) the language has been retained, but the published date has somehow been changed to my local date. Interestingly, the timezone offset is now incorrect (it wasn't published at 11pm GMT, it was 11pm GMT-5), but this does at least allow the ebook to display with the correct date in the Kindle UI.

Is it possible the process of mobi to azw conversion being done to copy the book to the Kindle via USB is updating metadata that is not being updated when sending the same mobi to an @kindle.net email address?

Also: is it normal for mobis to not store the timezone information in their published date field? It seems like this problem might go away if the full timezone was stored in the original mobi.

[kovidgoyal]

Umm the published field is storing the timezone. See the part that says +0:00 which means it is GMT. It's quite possible that the USB kindle plugin is delibrately setting an incorrect timestamp in the published date field to get around the Kindle's idiocy with time zones. I don't have the time right now to check. But in any case you should report the bug to Amazon. Maybe they'll fix it for their upcoming 3.1 firmware update.

[kovidgoyal]

I just took the time to check:

Here's the pubdate for a news download sent to the kindle by USB, before disconnecting the kindle, i.e. before the kindle has had a change to mess iwth the metadata, but after calibre is done with it:

Published : 2011-02-09T05:23:30.271993+00:00

And here's the pubdate in the same file after emailling it to myself (so again before Amazon gets is hands on it)

Published : 2011-02-09T05:23:30.271993+00:00

They are identical, I've now wasted enough of my time on this will'o the wisp.

[Quasii]

Thanks for looking into it Kovid. I'm having the same trouble and just want to clarify: Are you saying the fault is with Amazon? So there is nothing we can do?

Thanks again, your program is the bomb.

[Quasii]

I just was looking at the pubdate on today's paper, and when I plugged my kindle in by USB it resent it with the correct date (being now 9 hours later than when I originally had it autosent) when I looked at the two pubdates side by side they were identical even though on my kindle one has yesterdays date and one has todays.

Does this mean that it is an Amazon issue?

[alison87]

I believe there are two issues here. One issue is that the Kindle is not recognizing timezones in news sources, so a news source is shown as being published on its GMT date instead of the local time. This is clearly a bug and something i will raise with Amazon.

There does seem to be some other weirdness going on in the way files are copied from Calibre, though. I did the same test as Kovid (send email vs send via USB). The email always maintained the correct published date but in my case the USB transfer changed the published date. I didn't take my Kindle out of USB drive mode in between the time i sent the file and checked the metadata, so i can't see how the Kindle itself could be altering it. I don't know why it changed for me and not for Kovid, but there must be some code somewhere that is doing it. If you read the original post on this thread it's indicated that editing and re-saving the same metadata inside Calibre also "corrects" the date for future syncs, which implies there might be some sequence of events that causes the published date to be updated to local time whenever Calibre re-writes a mobi file. I'll take a look at the source code when i have time to see if i can figure it out.

[alison87]

I added some debugging code today and after some further testing it looks like i was wrong - Calibre doesn't appear to be changing the date when copying via USB. The mobi file is simply dropped into /documents/News, overwriting any previous files. I am guessing that yesterday i had so many versions of the same file floating around on my Kindle i got confused.

What is 100% certain is that the Kindle will always display the date of news items at GMT instead of in your local timezone, which is an Amazon bug. I'm not sure where to file bugs with them, but i'll try going through customer support and see what happens. We can temporarily get around the bug by outputting a bad timestamp in Calibre - override create_opf() in your recipe and replace the calls to nowf() with datetime.now().replace(tzinfo=tzutc()).

Kovid, can you think of any other case where custom recipe builders might want to override the published date and/or timestamp of the OPF file? Perhaps one case might be to use an RSS feed's pubDate tag rather than the actual current date. If this is useful for other people, perhaps you could add a get_opf_pubdate or get_opf_timestamp method that could be overridden by user recipes - that way it would be very easy for users to add the Kindle-specific hack to ignore the real timezone.