Sync to Furthest Page Read not working with sideloaded AZW3 files

[CheriePie]

I apologize if this has been covered elsewhere in this forum. I searched and it didn't turn up anything recent.

So here's my situation: I consistently read on 2 different Kindles: my Paperwhite and my Fire HD 8.9, and thus rely on the Sync to Furthest Page Read feature to sync me up between the two devices all the time.

This feature works fine in almost all circumstances EXCEPT when I'm using a sideloaded AZW3 file that's been converted with Calibre. (And YES, the book's metadata is using the Amazon ID in the ID field pointing to the ASIN of the Kindle version of the book. This is what makes it work with Calibre-converted MOBI files.)

AZW3 files coming directly from Amazon sync fine across both devices. As do MOBI files converted with Calibre and sideloaded to both devices. But if I use the AZW3 output format in Calibre, then nope.

With the AZW3 format book, when trying to Sync Furthest Page Read, my PW will report: "Unable to retrieve the furthest location read. If this is the first time you have opened this book, your device will update your location when you close the book or your device goes to sleep." Well it's not the first time I've opened the book and the device has been to sleep several times since I have so obviously that error message is fairly meaningless.

Likewise on my Fire, the error message: "You are currently at the furthest read location across all your devices." when obviously I'm not means this is just another meaningless error as well.

NOW, if I take those same files and reconvert them in Calibre to MOBI files, the syncing starts to work again.

Sooooo.... I'm thinking that there might be some important metadata Calibre isn't storing when writing AZW3 files, especially since the AZW3 files coming from Amazon, both books and personal docs, sync fine.

Any ideas? For now, I have had to resort to downgrading any of my sideloaded AZW3 files to MOBI so the sync will continue to work, but I'd love to get this worked out so that I don't lose any advanced Kindle 8 formatting if a specific book is using it.

[kovidgoyal]

Amazon does not want you reading side loaded books on your devices. They want all your books to come from them or at least live in their cloud.

Over the years they have been gradually and consistently disabling functionality for side-loaded books. They do it slowly and subtly hoping that people wont notice. I for one am tired of finding workarounds for their malice or their incompetence. If you do figure it out let us know, and I'll be happy to make the change (provided it does not break something else).

[CheriePie]

Thanks for the reply!

Yeah I get lots of pre-release books for review, usually in AZW3 format and sent directly to my Paperwhite via it's kindle email address. I then log into Amazon and send it over to my Fire too. Unaltered, they sync just fine between the two of them. But in a couple instances where I've pulled down the file to make a few changes in Calibre, add a table of contents for example, once I put the modified AZW3 files back on each of my Kindles, the furthest read page no longer syncs. If I instead make those same changes and convert it to MOBI, then Furthest Page Read will continue to sync fine. O_o

So I figure there's gotta be something specific we can point to that's missing from the altered vs the unaltered AZW3. Perhaps I can study the headers and metadata of each with a utility like Mobi2Mobi or whatever the equivalent is for AZW3 files, and maybe that'll point out any key changes.

Of course, if you or anyone else has a better idea, or wants to look at these files side-by-side yourself--the syncing and the non-syncing AZW3, as well as the syncing MOBI--then let me know and I'll send 'em on over. Since these are pre-release books sent out from the publisher to a limited audience, I don't think they'd care for me posting them publicly.

[kovidgoyal]

If I were to guess I'd say amazon is detecting that the azw3 file was altered by non amazon tools and deliberately disabling the sync. They imagine they can get away with this because azw3 is a "new" format. They did something similar with .apnx files a while back. Similarly, they deliberately changed the way that their newer firmware loads covers from mobi files to try to force side-loaded books into the "Personal docs" ghetto.

As I said, feel free to experiment, if you find some header field that can be modified to do the trick, let us know.

[CheriePie]

Will do kovid... it'll be a "when I find the time" type of thing, but I'd definitely like to see if I can figure it out and contribute to the community by getting the azw3 sync working. Thanks again for your replies, and have a great weekend!

[extesy]

I also have the same issue, so I looked into azw3 books in binary editor and for books originated from amazon, their ASIN (such as B007TBSL9U) is stored near the title field in the beginning of the file. By some reason ASIN is always specified twice.

On the other hand, in AZW3 books generated by Calibre, I couldn't find ASIN string at all. My guess is that Calibre is simply not writing this metadata into AZW3.

Kovid, I can send you a couple of original AZW3 files with all metadata fields for your experiments if needed. I really would like Calibre to add those missing fields.

[kovidgoyal]

calibre is not going to write ASINs into MOBI headers. It writes a uuid instead. I have explained why many, many times before. You just have to wait until Amazon implements syncing for personal AZW3 books (if ever). Or use MOBI.

[CheriePie]

Hmmm well at least now we know why it's not syncing for our sideloaded AZW3 files, and that it's something in the headers as I suspected. Thanks for looking into and confirming that extesy!

However, since Kovid can't or won't implement the writing of the ASIN to the AZW3 (I don't know his reasons, perhaps a search would turn those up), then maybe there's an alternate tool that will do so. Perhaps maybe Mobi2Mobi (or it's related GUI) can also write AZW3 files in addition to Mobi. Something to investigate I suppose. Unfortunately, I haven't had much free time at all so I'm just putting it out there...

[DoctorOhh]

Originally Posted by CheriePie

Perhaps maybe Mobi2Mobi (or it's related GUI) can also write AZW3 files in addition to Mobi. Something to investigate I suppose. Unfortunately, I haven't had much free time at all so I'm just putting it out there...

I have never owned a Kindle, does this wiki page come close to helping?

[WallyJ]

Kovid,

You mentioned in a post in this thread that we should be able to sync the furthest page read between Kindle devices/apps if we are using MOBI books. I have been trying this for some time and though I can sideload books with covers, I can't get them to sync. Can you point me to a guide on how to properly insert the ASIN and what settings to use in Calibre so that my "furthest page syncing" will work with my sideloaded MOBI books.

Thanks for all you do!!

WallyJ

[WallyJ]

Bump...

Kovid, did you get a chance to read my last post? Anyone else able to assist?

Thanks!

WallyJ

[WallyJ]

Been watching this thread for quite a while to get syncing of sideloaded books of any type to work with syncing.

The covers work fine on my Paperwhite, but the books never leave the device to sync with other devices.

Any advice would be appreciated.