Problem with synchronization of last read page

[tirion]

Hey guys

I hope I'm not starting an awfully redundant thread here.
I searched around a lot and found bits and pieces, but no satisfying answer.
So I figured a thread, dedicated to this problem, might not be the worst idea, since many people seem to have the same question.
If I'm wrong, I will be very thankful if you could just point me to the right discussion and will delete this thread to not clutter the forum unnecessarily.

But now to the point!

The Setting
My method is to convert the eBook to MOBI with the setting shown in the attached screenshot.
I do send the book via OTA to my Kindle. I know I could also send it via USB but here I read, that there is no chance for making the LRP-sync happen so..

I'm not using the quality check plugin as I couldn't quite get behind the necessity for it. And as far as I found out, it does not really improve on the LRP-sync anyway.

The Problem
The synchronization for the LRP is not exactly working for me.
Interestingly it does sometimes, but not consistently (it appears to me, that it works better to get the LRP from Android to Kindle than the other way around. Could be wrong about that, though).
I also don't get the covers since - I think - the beginning of 2019 anymore, but that is not really an inconvenience.
(It's a nice-to-have, though, but it's not focus/topic of this thread)

Also possibly of interest: the transferred eBook is listed as document, not book.

The Question
How can I make the synchronization of LRP via whispersync happen reliable again?
Switch to AZW3 or another format (if so, with which output setting)?
Is there a wrong setting?
I am missing an important plug-in?

Setup
Kindle PaperWhite 3 (2015) WiFi
Firmware Version: Kindle 5.12.2 (3569620025)
Calibre 4.2 (18 Oct, 2019)

Thanks a lot in advance for any pointers!

[BionicGecko]

Indeed you do need to use "send-to-kindle" in order for syncing to work. Unfortunately books not purchased on Amazon and sent OTA like this will always be listed as "documents", there is no way around that. It's actually called the "Personal Document Service".

As for covers, it depends on how you convert your books. If in Calibre you choose "Both" as MOBI File Type, then your Kindle will actually receive an AZW3 file when you download it (unless it's a very old model). If you choose "Old", then you'll get an actual MOBI file. Covers do work on MOBI files, but not on AZW3. On the other hand, AZW3 supports better formatting options.

Regarding your question, syncing the last page read has always been a bit hit-and-miss for me, regardless of the file type. Sorry I can't be of much help here, but I'm definitely listening if someone knows a way to make it more reliable!

[tirion]

Seems like can't edit my own post.
The screenshot of the MOBI setting didn't make it, so here goes the 2nd try.

[odamizu]

Originally Posted by tirion

...
The Problem
The synchronization for the LRP is not exactly working for me.
Interestingly it does sometimes, but not consistently (it appears to me, that it works better to get the LRP from Android to Kindle than the other way around. Could be wrong about that, though) ...

The Question
How can I make the synchronization of LRP via whispersync happen reliable again?

I had the same hit-or-miss syncing problem between iOS and Kindle devices with Calibre mobis.

What fixed it for me was switching from Calibre to KindleGen to do my conversions.

The easiest way to do this is to retrieve the epub file from your Calibre library, change the extension to .png or .txt (I seem to recall that .txt only works on Mac), then use Send to Kindle, and Amazon will run it through KindleGen for you and deliver it to all your devices. eink Kindles will get azw3; not sure what format iOS and Android get, but whatever it is works on my iOS devices.

If you use the Send to Kindle app, input Title & Author in dialog box. This is preferable to the Send to Kindle email address, which uses the filename for the title and the email address for the author.

[barryem]

I do a lot of syncing between multiple Kindles and between Android phones and Kindles. I do use Send to Kindle to send my books to the library. I then download them. These are always in mobi files when I send them.

I've found that it's fairly reliable if I'm careful. When I finish reading I sync. I sync before I begin on another device. Then I'm patient. It usually works. If I forget to sync when I'm done it often fails.

I've worked out a system to help with this. I make it a point to remember a short phrase from my current page when I quit. I make sure I remember it. The phrase from my current book is "older sister's husband". It's short and probably won't occur too many times. So when I begin reading I can use text search and it will probably show me less than a full page of that phrase, each with enough context that I can easily recognize the one I was reading.

I did a bit of experimenting on this when I first started doing it. It's typically about twice as fast as letting the device sync for me. Also it takes a bit of thought and effort so it's a bit more trouble and somewhat distracting. For that reason I mostly use it these days as a backup for the times the Kindle fails. I did use it exclusively for a while since it is faster.

Another thing I try to do is end a session at the end of a chapter and remember the name/number of the next chapter. That's effortless when I can do it but it isn't always practical.

Barry

[odamizu]

Originally Posted by barryem

I do a lot of syncing between multiple Kindles and between Android phones and Kindles. I do use Send to Kindle to send my books to the library. I then download them. These are always in mobi files when I send them.

I've found that it's fairly reliable if I'm careful. When I finish reading I sync. I sync before I begin on another device. Then I'm patient. It usually works. If I forget to sync when I'm done it often fails ...

That doesn't sound normal to me. I never do a manual sync, it all happens automagically for me.

[tirion]

Originally Posted by BionicGecko

Indeed you do need to use "send-to-kindle" in order for syncing to work. Unfortunately books not purchased on Amazon and sent OTA like this will always be listed as "documents", there is no way around that. It's actually called the "Personal Document Service".

Alright, but I don't see that as an issue, as long as it does not have any negative effect on whispersyn and/or the availability of the book cover.

Originally Posted by BionicGecko

As for covers, it depends on how you convert your books. If in Calibre you choose "Both" as MOBI File Type, then your Kindle will actually receive an AZW3 file when you download it (unless it's a very old model). If you choose "Old", then you'll get an actual MOBI file. Covers do work on MOBI files, but not on AZW3. On the other hand, AZW3 supports better formatting options.

Well, don't have either the newest nor the oldest Kindle.
Is the setting "Both" a universal go, no matter the hardware?
Or is there benefit of "New", as long as the device is not too outdated for that option (and if so, how do I know that?)

Originally Posted by BionicGecko

Regarding your question, syncing the last page read has always been a bit hit-and-miss for me, regardless of the file type. Sorry I can't be of much help here, but I'm definitely listening if someone knows a way to make it more reliable!

Crap, so it actually is possible that this won't work well at all with books, not purchased from Amazon? :/

Originally Posted by odamizu

I had the same hit-or-miss syncing problem between iOS and Kindle devices with Calibre mobis.

What fixed it for me was switching from Calibre to KindleGen to do my conversions.

The easiest way to do this is to retrieve the epub file from your Calibre library, change the extension to .png or .txt (I seem to recall that .txt only works on Mac), then use Send to Kindle, and Amazon will run it through KindleGen for you and deliver it to all your devices. eink Kindles will get azw3; not sure what format iOS and Android get, but whatever it is works on my iOS devices.

If you use the Send to Kindle app, input Title & Author in dialog box. This is preferable to the Send to Kindle email address, which uses the filename for the title and the email address for the author.

Wait, let me see if you understood you correctly:
I create (or take if already at hand) the ePub version of the book in question.
Then I change the extension to ".png" (an image format?) or ".txt" and send it via eMail to the address of my Kindle.
That does the trick for you, to make whispersyn work more reliable?

Originally Posted by barryem

(...)I've found that it's fairly reliable if I'm careful. When I finish reading I sync. I sync before I begin on another device. Then I'm patient. It usually works. If I forget to sync when I'm done it often fails.

I've worked out a system to help with this. I make it a point to remember a short phrase from my current page when I quit. I make sure I remember it. The phrase from my current book is "older sister's husband". It's short and probably won't occur too many times. So when I begin reading I can use text search and it will probably show me less than a full page of that phrase, each with enough context that I can easily recognize the one I was reading.

I did a bit of experimenting on this when I first started doing it. It's typically about twice as fast as letting the device sync for me. Also it takes a bit of thought and effort so it's a bit more trouble and somewhat distracting. For that reason I mostly use it these days as a backup for the times the Kindle fails. I did use it exclusively for a while since it is faster.(...)

Well, that is a rather unique approach I have to admint But very creative, I give you that.
I just tried to work with the manual sync, but got nothing out of it unfortunately. At least not with this book. Other books do sync. Same conversion setting, same format, I dont get it..

The remembering part is a nice workaround actually, but not quite what I'm looking for.
Thanks for sharing, I think I will make use of it, though, every now and then.

[tirion]

Originally Posted by tirion

Well, don't have either the newest nor the oldest Kindle.
Is the setting "Both" a universal go, no matter the hardware?
Or is there benefit of "New", as long as the device is not too outdated for that option (and if so, how do I know that?)

Okay, I did the experiment and got a very positive outcome.
Changed the output of MOBI file type to new, deleted the old one from Calibre, Kindle, Android and Amazon.
Then - of course - converted it to the new MOBI and send it via mail to my Kindle.
Whispersync worked without any manual intervention.

Now I'm not sure if this keeps up all the time and with all books, but a positive first outcome non the less.
I will keep you posted.

Regarding the cover nothing changed, though.

Which leads me to a new/additional question:
Can I add the cover by just manually copy the image file onto the Kindle? I tried but no luck.
Maybe with a certain name? In a certain location on the Kindle (because on the Android the cover shows no problem)? Or a certain image size?

[BionicGecko]

Originally Posted by tirion

Alright, but I don't see that as an issue, as long as it does not have any negative effect on whispersync and/or the availability of the book cover.

It does not affect whispersync, but it does indirectly affect the availability of the book covers. There was a recent discussion on this topic, it's a good read: https://www.mobileread.com/forums/sh...d.php?t=321635


Basically, if you consider those three things:

1. Syncing of last page read across devices
2. Book covers
3. Advanced text formatting

Unless your Kindle is still on a fairly old version of the firmware (i.e. lower than 5.8.5), it is not possible to have all three. You can have:

• 1 and 2 (but not 3) by using the "Old" MOBI format
• 1 and 3 (but not 2) by using "Both" as the MOBI format, or converting to AZW3
• 2 and 3 (but not 1) by sideloading your books via USB

So the answer to your question:

Originally Posted by tirion

Can I add the cover by just manually copy the image file onto the Kindle? I tried but no luck.
Maybe with a certain name? In a certain location on the Kindle (because on the Android the cover shows no problem)? Or a certain image size?

is unfortunately no.

Originally Posted by tirion

Well, don't have either the newest nor the oldest Kindle. Is the setting "Both" a universal go, no matter the hardware?
Or is there benefit of "New", as long as the device is not too outdated for that option (and if so, how do I know that?)

Indeed when using "Both" you get a file that contains two versions of the book, one in the old MOBI format, and one in AZW3. When you send the file to amazon, their servers keep both versions. If you request your book from an older Kindle, then you will get the MOBI version, because those kindles don't support AZW3. If your Kindle does support it, then the AZW3 version will be downloaded to your Kindle.

As far as I know there is no "New" option, because "New" would be a plain AZW3 file, so you would convert directly to AZW3 in Calibre instead.

[odamizu]

Originally Posted by tirion

... Wait, let me see if you understood you correctly:
I create (or take if already at hand) the ePub version of the book in question.
Then I change the extension to ".png" (an image format?) or ".txt" and send it via eMail to the address of my Kindle.
That does the trick for you, to make whispersyn work more reliable

Yes.

KindleGen is Amazon's mobi creation software. It converts epub (and other formats) into a mobi file that contains an azw3/KF8 component and an old mobi format component. This is similar to Calibre's mobi-both.

I found that while Calibre mobi-both files usually work, occasionally they fail to sync or have other problems. When that happens, running the epub through KindleGen always fixed the problem for me, so I switched to KindleGen.

You can run KindleGen in a variety of ways:

• download the command-line tool from Amazon
  
• use Amazon's Kindle Previewer, which has KindleGen embedded in it and has a more user-friendly GUI
  
• use Sigil with Doitsu's KindleGen plugin
  
• use Send to Kindle
  

Send to Kindle is the easiest. Officially, Send to Kindle does not accept epub files, but if you change the file extension to .txt or .png (yes, an image format), Send to Kindle thinks it's .txt or .png rather than .epub and will accept it, run it through KindleGen, and voila, you get a dual-mobi file in your Kindle Personal Documents library that syncs. iOS (and Android??) will receive old mobi format, so the cover works, but eink Kindles get azw3/KF8, so no cover.

Using the Send to Kindle desktop app is preferable to email because then you can input the title and author. If you Send to Kindle via email, the title will be the filename and the author will be the email address.

[tirion]

Originally Posted by BionicGecko

Basically, if you consider those three things:

1. Syncing of last page read across devices
2. Book covers
3. Advanced text formatting

Unless your Kindle is still on a fairly old version of the firmware (i.e. lower than 5.8.5), it is not possible to have all three. You can have:

• 1 and 2 (but not 3) by using the "Old" MOBI format
• 1 and 3 (but not 2) by using "Both" as the MOBI format, or converting to AZW3
• 2 and 3 (but not 1) by sideloading your books via USB

What I noticed is, that if I change the MOBI Output setting from "both" to either "old" or "new", the sync between my Android phone and Kindle worked like a charm!
But with the setting on "both" it was back at the lottery.
The cover did never show up on my Kindle, though, no matter the setting.

Originally Posted by BionicGecko

Indeed when using "Both" you get a file that contains two versions of the book, one in the old MOBI format, and one in AZW3. When you send the file to amazon, their servers keep both versions. If you request your book from an older Kindle, then you will get the MOBI version, because those kindles don't support AZW3. If your Kindle does support it, then the AZW3 version will be downloaded to your Kindle.

This does sound like an optimal solution, but - for me at least - proofs very fragile in comparison to how reliable whispersync works on "new" or "old" setting for MOBI.

Originally Posted by BionicGecko

As far as I know there is no "New" option, because "New" would be a plain AZW3 file, so you would convert directly to AZW3 in Calibre instead.

There is a "new" setting, actually. I took a screenshot of it.

Originally Posted by odamizu

Send to Kindle is the easiest. Officially, Send to Kindle does not accept epub files, but if you change the file extension to .txt or .png (yes, an image format), Send to Kindle thinks it's .txt or .png rather than .epub and will accept it, run it through KindleGen, and voila, you get a dual-mobi file in your Kindle Personal Documents library that syncs. iOS (and Android??) will receive old mobi format, so the cover works, but eink Kindles get azw3/KF8, so no cover.

So, in essence it should be the same as having the MOBI Output set to "both"? Because here the sync does not really work as expected.
Or is there something fundamentally different when pushing the file through KindleGen locally and not automatically like when sending the book via mail?

Originally Posted by odamizu

Using the Send to Kindle desktop app is preferable to email because then you can input the title and author. If you Send to Kindle via email, the title will be the filename and the author will be the email address.

I gotta say, I never noticed that either the title or the author got screwed up when sending the book via mail to my Kindle. Neither on the phone nor the eBook reader itself.
You mean when e.g. "clicking" and holding on the book cover to get to the detailed info of the book, right?

[tirion]

Actually, why not using AZW3 directly instead of creating a MOBI with "both"?

[jhowell]

Originally Posted by tirion

What I noticed is, that if I change the MOBI Output setting from "both" to either "old" or "new", the sync between my Android phone and Kindle worked like a charm!
But with the setting on "both" it was back at the lottery.

...

So, in essence it should be the same as having the MOBI Output set to "both"? Because here the sync does not really work as expected.
Or is there something fundamentally different when pushing the file through KindleGen locally and not automatically like when sending the book via mail?

There are differences in a MOBI fle produced using the "both" option of calibre and a Master MOBI created using kindlegen either by running it yourself or as the result of emailing a source file to Amazon. One difference is that calibre's MOBI "both" lacks a DATP record which contains the data needed for proper synchronization of reading position between formats.

Originally Posted by tirion

Actually, why not using AZW3 directly instead of creating a MOBI with "both"?

Amazon will not accept an AZW3 file via email so you can't add it to your account and it will not sync between devices.

[Waitingforatrain]

Sync seems to work best for me if I convert files to old mobi before sending to Kindle with Calibre. I get covers too.

[odamizu]

Originally Posted by tirion

... Because here the sync does not really work as expected.

Please elaborate on exactly what's happening that is not as expected.

You are using the .txt/.png trick to email an epub and it's not syncing?

I gotta say, I never noticed that either the title or the author got screwed up when sending the book via mail to my Kindle. Neither on the phone nor the eBook reader itself.
You mean when e.g. "clicking" and holding on the book cover to get to the detailed info of the book, right?

The "screw up" is only if you use the .txt/.png trick to email the book.

If you email a .mobi, it's fine (it gets the author/title from the opf).