Kohighlights - Merge / Sync annotations & positions across several devices

[edojan]

Below I describe some essential elements of my setup that work well to synchronize my highlights and comments across several Koreader devices using Kohighlights (a.k.a. "KH"). KH is a freeware app that is actively being developed by @embryo . The app has significantly evolved during the last year to become an extremely capable annotation manager designed to work with Koreader metadata. You may find this information helpful when creating your own setup.

Note, this is not intended to be an exhaustive guide. If you need more detail, feel free to ask or consult the forums. E.g. there are plenty of resources describing how to install Koreader on your android or Linux devices or how to install user patches in Koreader. Also, make sure to read the newly released KH user guide here.

But first, a little bit of theory to put things in context.

File-level vs annotation-level synchronization

There are two basic ways to synchronize annotations across your Koreader devices:
1. File level syncing (the older metadata files on your devices are replaced with the newer files)
2. Annotation level syncing (more accurately should be called merging since the individual annotations taken on different devices are merged and then written into the respective metadata files)

File-level sync

Probably the most commonly used tool for the file-level synchronization is Syncthing. The idea here is to have Syncthing running on a centralized cloud server / local PC, as well as on all Koreader devices that you wish to sync. The program would continuously sync the individual book metadata files with the server overwriting the older files with the newer ones.
While this may sound like a great method, in practice there are a number of issues that can make or break the actual implementation.

Among the issues that plague the file level synchronization methods is the need to follow a strict sequential pattern when using your Koreader devices:
1. Create annotations using one device
2. Sync changes with the server
3. Sync the second device with the server
4. Proceed with annotating on second device
5. etc.

This is not very practical in most cases. For example, my typical reading pattern is not linear. When I read / annotate my books I often switch back and forth between my reading devices. To illustrate, my setup consists of three devices running Koreader: Kobo reader, Android phone, and Windows Desktop / WSL. I may start annotating a book on Kobo, continue on my desktop, return back to Kobo, then resume it on my android phone, etc. Or in a different order. You get the idea.

Another important consideration while using Syncthing is the need to keep your devices powered-on and connected to a centralized server. If this condition is not met, then there is a good chance of creating version conflicts and having your annotations overwritten by the metadata file with the latest timestamp.
Again, I find this difficult to maintain. I often revise / edit my annotations at whatever device I happen to be at a given time. Some of my devices may be on, and others may be off. Likewise some may be connected to the network and some not.

Annotation-level sync

In contrast to the file-level, the annotation-level synchronizers such as Kohighlights (KH) look inside of the book metadata files, find the differences, and merge them across all files included in the sync group. This eliminates vast majority of issues associated with the file-level syncing. One can read / annotate in a non-linear fashion while the devices may or may not be connected to network. Also, the risk of losing information is much lower given that the annotation data is merged, not overwritten. Additionally, one can setup KH to archive a copy of your annotations in its internal database as a backup. A potential disadvantage is the need to sync the devices manually (KH makes this process easy once you set up your sync groups)

KH is a freeware app specifically created to work with Koreader book metadata residing on several devices. KH installers for Windows and Linux can be downloaded from Github.

Synchronization process in KH is discrete, not continuous. From time to time the user must connect all the devices he wishes to sync to a PC running KH. Once the devices are connected, the user can create new "Sync groups" -which are essentially "jobs" for each book to be synchronized. Each sync group contains paths to the metadata files to be synchronized and by default is named after the book title.

Some things you should keep in mind:

1. By default Koreader stores highlights and notes (annotations) in metadata.epub.lua files. These files are automatically created in .sdr folders matching your book's epub filename. Example: my_favorite_book.epub will have its metadata in: **my_favorite_book.sdr/metadata.epub.lua** All Koreader devices you wish to sync must be connected to the PC running KH with the paths to the metadata files accessible from the windows explorer as letter drives.

2. Annotations you create on your devices are tied to the "page number" keys in the metadata files. Given that .epub is a reflowable format it doesn't have an intrinsic concept of a page. Therefore, instead of the "real" page numbers, epub readers calculate "pseudo" page numbers which are dependent on respective device resolutions and font size options selected. Hence the page numbers displayed by one epub reader are often differ from those displayed by other readers. This presents a problem for merged annotations for they will appear out of sequence.
To overcome this limitation the good folks from the Koreader dev team as well as the community created the "logical pages" user patch and allowed it to write the "logical pages" into the [pageref] key of the book metadata file. The patch counts every 1818 characters of the text to be a logical "page". If you install this patch on all your readers, the page numbering will be harmonized across all your devices irrespective of the resolutions or font sizes. Once the user patch is installed for all your Koreader devices, make sure that you enable "Use reference pages: check box under "Reference pages" menu item.

3. You can run Koreader in Windows (under WSL), and set Koreader as the default epub reader using the following workaround: a. create a .bat file containing a script to launch Koreader from WSL b. Set the windows default for .epub to the bat file. This works well enough and in my case allows me to keep using the Calibre library as my epub organizer and read / annotate the epub books using Koreader, instead of the default Calibre reader. Note, that by default Koreader will create / store the book metadata files containing the annotations in the same directories where Calibre keeps the corresponding epubs enabling easy creation of the synch groups in KH.

4. If you want to sync Koreader on an android phone then your phone must be accessible to KH as a letter drive. This is something that android doesn't support out of the box. The solution is simple: install on your phone an FTP server or a Webdav server app and on the windows PC install a program that can map an FTP / Webdav connection as a drive letter. Here is what works well for me on android: Owlfile is a free app that has a build-in Servers feature. On windows to map the connection as a letter I am using a free version of RaiDrive.

5. KH can export your annotations to a better looking (and much more usable) markdown output where the book's Table of Content chapter / sub-chapter names are mapped to the corresponding markdown heading (##, ###, ####, etc.) producing a clean hierarchical structure. This section of the KH user guide explains the process and provides an example of the output. Note, for this to work your annotations must be taken while “2-save-full-chapter-path.lua” user patch is installed. That is the existing annotations (those you took without the user patch) will continue to show the immediate chapter headings only.


The rest is quite straightforward.
Install KH, connect your devices, create sync groups and start syncing!
The attached is a screenshot showing KH interface to create and manage sync groups (a similar pic can find in the KH user guide).

Sync group layout elements
(1) Add a new group (KH will create a new empty group at the bottom of the list)
(2) Sync all groups (KH will run all active sync groups at once)
(3) Delete selected group(s) (also available via right-click menu)
(4) Refresh the source files (KH will re-read the metadata files specified in the Sync Paths of the selected group)
(5) Sync Position (check this box to sync the reading positions)
(6) Merge Highlights (KH will merge the annotations - highlights and comments across the devices in the group)
(7) Sync with archived (KH will update the archived version in its internal database)
(8) The name of the sync group (typically the same as the book's title). This can be renamed via right-click menu
(9) Sync this group: KH will only sync this group, not any other group. Same is available via right-click menu
(10) Add / Remove a member of the sync group
(11) Select file paths pointing to book metadata files (e.g. after adding a new member to the group)
(12) Toggle switch of active / inactive (makes group active or inactive. Only active groups are synced when used Sync All (2).
(13) Path error (red color indicates an issue with the file)
a) File path is inaccurate / metadata file is missing or not accessible (use "Refresh" button after correcting the issue to make KH to re-read the path)
b) Metadata files MD5 value is not matching to other books in the group (see advanced features for some ways to correct).

[hius07]

Nice manual! Some day we will add "logical pages" to the core.

[edojan]

Originally Posted by hius07

Nice manual! Some day we will add "logical pages" to the core.

Looking forward to that. Thank you for creating the user patch to record full TOC path!

[RandomBoi]

Does KOhighlights only syncs via USB, right? No chance to sync wirelessly?
The project is really interesting.

[edojan]

Originally Posted by RandomBoi

Does KOhighlights only syncs via USB, right? No chance to sync wirelessly?
The project is really interesting.

Correct. Currently, all syncing is via USB only, although android phones can bes connected wirelessly via FTP or SFTP clients (but the connection must be mapped as a drive letter for Kohilights to be able to see it)

[embryo]

Perhaps if you can sync wirelessly with Dropbox or any other service like this, you can then sync the local files?..

[RandomBoi]

Originally Posted by edojan

Correct. Currently, all syncing is via USB only, although android phones can bes connected wirelessly via FTP or SFTP clients (but the connection must be mapped as a drive letter for Kohilights to be able to see it)

And I guess if my ereaders are all Android, those would work that way, too? That way I shall I be able to set up multiple android device wirelessly?

[RandomBoi]

Originally Posted by embryo

Perhaps if you can sync wirelessly with Dropbox or any other service like this, you can then sync the local files?..

I hope to have the time to attempt this soon using SyncThing and report here.