Looking for an offline method to sync book highlights/notes on KOReader

[kokaros]

Hello forum,

Let me preface this by saying that I have indeed read through many threads here in an attempt to find an answer, before creating my own.

These past few days I've started playing around with my Kindle PW3. As my Kindle was already running the latest FW v5.12.4, I had to go with the serial port JB. I then proceeded with the installation of the JB "hotfix", MR Package Installer, KUAL, USBNetwork and KOReader.

The main reason as to why I wanted to do the jailbreak in the first place was to have the ability to use a reader that would:

1. Have dark reader/night mode option (Amazon hasn't released this feature to PW devices older than the PW4)
2. Would render PDF documents better than the stock reader
3. Would allow me to backup/export my highlights and notes in a format which would be importable on:

b)the same book on other e-reader devices
a)the same book on the same device, e.g. in case I decide to delete the book and its contents and re-import it along with its highlights and notes at a later point[/LIST]

This leaves us with KOReader as the perfect candidate for points 1) and 2). Now point 3) is where it gets tricky...
Lets consider .epub or any format other than .pdf, where KOReader cannot save the highlights within the document itself.
How can I achieve all the requirements I've listed above without using a WiFi connection? Using the stock kindle reader I know that I could just plug in my device, copy the .sdr folder which contains the .azw3f and .azw3r files, which in turn contain the highlights and notes, and store them on my PC - but these types of files wouldn't be compatible on a Kobo lets say.

The reason why I'm not so keen on using a WiFi connection is because my Kindle is running the latest firmware. From what I've read on the forum it would probably update OTA and I don't want to take any chances with losing the JB. Still, even if the JB were to survive, I really don't wanna go through the trouble of installing all the packages I've installed post JB again.

I also have one more question. Disregarding everything I said above (i.e. assuming WiFi wasn't a problem), does Amazon's Whispersync sync notes and highlights? e.g. I have book A which I've taken notes and highlights on, I perform a sync, but then I delete its notes and highlights locally. If I do another sync after the delete, will Whispersync fetch the deleted notes and highlights that were previously synced from the server?
If this is true, then I've also though of another possible solution... Does KOReader have a feature which exhibits the behavior I've just described above for Amazon Whispersync? If so, then I could try to downgrade my FW to a version where methods that block OTA updates were actually working so turning the WiFi on wouldn't be a problem and I could use this sync option with WiFI through KOReader - but then again would a downgrade work with the latest FW version that I am currently running?

I've tried to explain everything as clearly as I could but please, if there's anything that wasn't clear let me know and I'll try my best to explain it better.
I really hope someone can help me out with this, I'd really appreciate it!

Thanks.

[NiLuJe]

Whispersync has nothing to do with KOReader, so I'm not sure what you were asking there...

KOReader's own metadata handling is put in the book's sidecar folder (e.g., for an ePub: ${filename}.sdr/metadata.epub.lua). The actual content of the file is portable, it doesn't depend on a specific filename, so you can move stuff around manually without any issue.

KOReader's sync feature can sync *progress*, but not notes/higlihts (AFAIK, I'm not familiar with the details).

You can also use @embryo's tool to export notes/highlights to more human-readable formats.

TL;DR:

Originally Posted by kokaros

Using the stock kindle reader I know that I could just plug in my device, copy the .sdr folder

Doing just this happens to be portable in KOReader's case .

[ilovejedd]

Originally Posted by NiLuJe

You can also use @embryo's tool to export notes/highlights to more human-readable formats.

In addition to human-readable exports, it's also capable of merging highlights created on multiple devices in native KOReader format.

I've never tested what it does when it encounters collisions though.

[kokaros]

Thanks for the prompt replies guys.

Originally Posted by NiLuJe

Whispersync has nothing to do with KOReader, so I'm not sure what you were asking there...

Yes, I am aware that Whispersync has nothing to do with KOReader. The answer I was looking for there is the one you gave me here:

Originally Posted by NiLuJe

KOReader's sync feature can sync *progress*, but not notes/higlihts (AFAIK, I'm not familiar with the details).

Also, thank you for letting me know about this:

Originally Posted by NiLuJe

KOReader's own metadata handling is put in the book's sidecar folder (e.g., for an ePub: ${filename}.sdr/metadata.epub.lua). The actual content of the file is portable, it doesn't depend on a specific filename, so you can move stuff around manually without any issue.

I had no idea KOReader was creating locally a .sdr folder, just like Kindle's native reader does, containing the book's highlights and notes.
Having read your replies, I still have a few more questions. Please bear with me

FIRST:
Going back to point 3) of my post:

Originally Posted by kokaros

3)Would allow me to backup/export my highlights and notes in a format which would be importable on:

a)the same book on other e-reader devices
b)the same book on the same device, e.g. in case I decide to delete the book and its contents and re-import it along with its highlights and notes at a later point

I can see how I can backup the highlights and notes (the .sdr folder) on my PC as NiLuJe explained, and have the ability to import them again on the same kindle, so point b) is covered (I've just tested this). How about point a)? Does KOReader use the same format for the folder .sdr and contained files .lua across other e-reader devices? I.e. if I put the exact same ebook on another device which uses KOReader, can I just copy the .sdr folder and be sure that my highlights and notes will appear there as well?

SECOND:

Originally Posted by ilovejedd

In addition to human-readable exports, it's also capable of merging highlights created on multiple devices in native KOReader format.

I've already downloaded and played around with the KoHighlights program. I've noticed the "Merge/Sync" option in the top bar menu. When you hover over it, you get the following message:
"Merge the highlights from the same book in two different devices, and/or sync their reading position. Activated only if two entries of the same book are selected."

What does it mean with that last sentence "Activated only if two entries of the same book are selected."? Do I need to have 2 separate e-reader devices plugged in at the same time, with both of them having the same book stored on them to have the "two entries of the same book" appear within the program, so that in turn this feature works?

THIRD:
Does Amazon's Whispersync sync:

a) the book file itself
b) the book's highlights
c) the book's notes

across other kindle devices owned by me? I.e. if I were to buy a new Kindle and register it with the same account as the one I am using on my current one, will I get a copy of the things I've listed above on my new kindle?

FOURTH:
If the answer to my THIRD question is yes, does this apply to books I've transferred to my original Kindle through USB, or is it just for books that were transferred to it via the "Send to Kindle" software or bought through Amazon's store?

FIFTH:
Do we have any confirmed successful downgrade attempts of FW v5.12.4 running on Kindle PW3 here on the forum?

SIXTH:
Even if we don't have any confirmed successful downgrades, what can I do prior to a downgrade attempt, to be 100% sure that if my device gets bricked I'll be able to easily unbrick it?

SEVENTH:
What FW version would you recommend downgrading to, so that I can be 100% sure that even with WiFi turned on I won't get a forced OTA update?
Having the JB without being able to connect to the WiFi is pretty lame

I hope this post isn't too overwhelming

[NiLuJe]

1. Yep, that's what I meant by "portable" .

2. I've never used it (I'm not a heavy annotation user, period), so I'll let someone more familiar with it field that one .
I would assume that's what @ilovejedd alluded to earlier, but I have no idea how it works in practice .

3. b & c, IIRC. (Again, never really used Kindles with WiFi on, so, err, take that with a grain of salt).

As for a., depends on what you mean, exactly. If it's not archived in your Library, a *device* sync will pull a device-specific version of the book that ought to be compatible with existing hl/notes on other devices. I *assume* it's true even if one device gets a KFX and the other an old M7 (i.e., worst-case scenario, as far as format generations are concerned), but I may be wrong on that point.

4. May apply to sideloaded content if it's flagged as EBOK with the proper ASIN (which Calibre can do). Again, take that with a grain of salt, I haven't followed those shenanigans in a good long while, and I was never the target audience to begin with.

5. Can't downgrade current Kindles. No diags and no serial shell on a PW3 means you *probably* can't do it over serial either.

6. None, RP & CRP can't work on those devices.

7. Final 5.9.x (or the exact 5.10.x before the KFX DRM update, I'm not familiar with the exact version, I think @ilovejedd knows though ).

[ilovejedd]

2. Alas, it's been 2 years since I last used KoHighlights. Nowadays, I just use either GoodReader or Marvin for iOS for books I need/wish to highlight.

I backup my Kindles to the PC, though:

\backup
    \kindle
        \pw3a
        \pw3b
        \pw4a
        \pw4b

I remember I had KoHighlights pointed to the full kindle backup folder.


7. All FW 5.10.x are safe.

FW 5.10.1.3 is the last version with old KFX DRM but the update blockers all work up to FW 5.10.3.

The update.bin.tmp.partial folder method stopped working in FW 5.11 and SKIP_UPDATE_CHECK file method stopped working in FW 5.12.

[kokaros]

Thank you so much @NiLuJe and @ilovejedd. You've both been extremely helpful

Hope an OTA update blocker comes out for newer FW versions soon