[GUI Plugin] KOReader Sync

[harmtemolder]

KOReader calibre plugin

Hi,

I'm looking for someone who wants to help me maintain this plugin, because I don't use it all that much anymore. Please contact me if you're up for this.

Harm

A calibre plugin to synchronize metadata from KOReader to calibre.

KOReader creates sidecar files that hold read progress and annotations. This plugin reads the data from those sidecar files and updates calibre's metadata based on them. It is inspired by the Kobo Utilities plugin, that synchronizes reading progress between the original Kobo firmware ("Nickel") and custom columns in calibre.

Note that at the moment the sync is primarily one-way—from the KOReader device to calibre, and only works for USB and wireless devices. For the latter, you'll need KOReader 2021.04 or newer.

Pushing metadata from calibre to KOReader currently works only for books which do not have KOReader sidecar files, and of course requires the raw metadata column to be mapped. The use-case is for setting up a new device, or if a book was removed from your device and you've now added it back. This has been tested for calibre's Connect to Folder and Custom USB Device modes. It does not seem to work for the Kobo Touch device driver nor with wireless connections, but I (@charlesangus) find those don't communicate perfectly with Calibre/KOReader in any case... I haven't disabled it for other devices - it may be a quirk in my setup which is causing it to fail, and it may work fine for you.

Releases will also be uploaded to this plugin thread on the MobileRead Forums. If you are on there as well, please let me know what you think of the plugin in that thread.

Using this plugin

Download and install

1. Go to your calibre's Preferences > Plugins > Get new plugins and search for KOReader Sync
2. Click Install
3. Restart calibre

Alternatively

1. Download the latest release from here.
2. Go to your calibre's Preferences > Plugins > Load plugin from file and point it to the downloaded ZIP file
3. Restart calibre

Setup

1. Pick and choose the metadata you would like to sync and create the appropriate columns in calibre. These are your options:
   • A Floating point numbers column to store the current percent read, with Format for numbers set to {:.0%}.
   • An Integers column to store the current percent read.
   • A regular Text column to store the location you last stopped reading at.
   • A Rating column to store your rating of the book, as entered on the book's status page.
   • A Long text column to store your review of the book, as entered on the book's status page.
   • A regular Text column to store the reading status of the book, as entered on the book status page (Finished, Reading, On hold).
   • A Yes/No column to store the reading status of the book, as a boolean (Yes = Finished, No = everything else).
   • A Date column to store the date on which the first highlight or bookmark was made. (This is probably around the time you started reading.)
   • A Date column to store the date on which the last highlight or bookmark was made. (This is probably around the time you finished reading.)
   • A Long text column to store your bookmarks and highlights of the book, with Interpret this column as set to Plain text formatted using markdown. (Highlights are an unordered list with their metadata in an HTML comment.)
   • A regular Text column to store the MD5 hash KOReader uses to sync progress to a KOReader Sync Server. (Progress sync in the KOReader app.) This might allow for syncing progress to calibre without having to connect your KOReader device, in the future.
   • A Date column to store when the last sync was performed.
   • A Date column to store when the sidecar file was last modified.',
   • A Long text column to store the contents of the metadata sidecar as JSON, with Interpret this column as set to Plain text. This is required to sync metadata back to KOReader sidecars.
2. Add KOReader Sync to main toolbar when a device is connected, if it isn't there already.
3. Right-click the KOReader Sync icon and Configure.
4. Map the metadata you want to sync to the newly created calibre columns.
5. Click OK to save your mapping.
6. From now on just click the KOReader Sync icon to sync all mapped metadata for all books on the connected device to calibre.

Things to consider

• The plugin overwrites existing metadata in Calibre without asking. That usually isn’t a problem, because you will probably only add to KOReader’s metadata. But be aware that you might lose data in calibre if you’re not careful.
• Pushing sidecars back to KOReader currently only happens for sidecars which are missing. For now, manually delete the <bookname>.sdr folder from the device before attempting to push the sidecars back to KOReader for any books you would like to overwrite the current metadata with Calibre's metadata.
• When pushing missing sidecars to the device, no attempt is made to convert Calibre's metadata to account for changes in KOReader's sidecar format. Old metadata may work unpredictably if it's from a different version of KOReader.

Supported devices

This plugin has been tested successfully with:

• Kobo Aura connected over USB, which means it will probably work for all comparable Kobo devices (KOBO and KOBOTOUCH)
• Kobo Aura H2O over USB (KOBOTOUCHEXTENDED, see #6 for details)
• Kobo Aura connected wirelessly, which means it will probably work for all calibre connect devices (SMART_DEVICE_APP)
• A connected folder (FOLDER_DEVICE)
• Kindle Keyboard (KINDLE2, see #1 for details)
• Tolino Vision 4 HD (TOLINO, see this comment for details)
• PocketBook Touch Lux 5 (which uses the POCKETBOOK626 driver, so it will probably work for all comparable PocketBook devices, see #8 for details)
• PocketBooks that use the POCKETBOOK622 driver

This plugin is not compatible with:

• MTP_DEVICE (see #2 for details)

Issues

If you encounter any issues with the plugin, please submit them here.

Acknowledgements

• Multiple tweaks and bug fixes by Glen Sawyer
• Additional functionality by Charles Taylor
• Contains SirAnthony's SLPP to parse Lua in Python.
• Some code borrowed from--and heavily inspired by--the great Kobo Utilities calibre plugin.

Contributing to this plugin

Notes & Tips

• My first attempt was actually to sync calibre with KOReader's read progress through the progress sync plugin and a sync server. Read here why that did not work. This plugin might actually make that possible now by allowing you to store KOReader's MD5 hash in calibre...
• calibre allows you to auto-connect to a folder device on boot, which greatly speeds up your workflow when testing. You can find this under "Preferences" > "Tweaks", search for auto_connect_to_folder. Point that to the dummy_device folder in this repository. (I have included royalty free EPUBs for your and my convenience.)
• If you're testing and don't actually want to update any metadata, set DRY_RUN to True in __init__.py.
• I work in PyCharm, which offers a remote debugging server. To enable that in this plugin, set PYDEVD to True in __init__.py.You might need to change sys.path.append in action.py.
• The supported device drivers can be found in the SUPPORTED_DEVICES list in config.py. Adding a new type here is the first step to adding support, but make sure all features are tested thoroughly before releasing a version with an added device

Testing in calibre

Use make to load the plugin into calibre and launch it:
[pre][code=shell]make dev
[/code][/pre]

Alternatively, build a release and load that:
[pre][code=shell]make zip load
[/code][/pre]

Building a release

Make sure you have the dependencies and have set the correct version number in __init__.py, pluginIndexKOReaderSync.txt and Makefile. Also update Changelog. Then:
[pre][code=shell]make zip
[/code][/pre]

Debugging a release

1. Download the required release from here
2. Add it to calibre by running this in your terminal: calibre-customize -a "KOReader Sync vX.X.X-alpha.zip", where X.X.X refers to the version you downloaded
3. Start calibre in debug mode with calibre-debug -g
4. Configure the KOReader plugin as described here
5. Connect your device
6. Run the sync by clicking the KOReader icon in your toolbar
7. Check the details of the message when it's done if any/all books have been synced correctly
8. Check your (custom) columns for one of those books to see if their contents are what they should be
9. Check the output in your terminal for lines containing koreader to see what it did

Changelog

Spoiler:

All notable changes to this project will be documented here.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Unreleased

0.5.2-beta - 2023-05-22

• Many thanks to @elmodor and igorius for their help!

Added

• Added config option to only sync if the metadata is newer than the data stored in calibre (will fallback to "Percent read column" if no "Date Modified column" exists or can not be obtained)
• Added config option to not sync if the book has already been marked as finished (via "Percent read column" or "Reading status column")
• Added a yes/no column for read status (based on changes from igorius at MobileRead)

Changed

• Pylint cleanup
• Update README to match new columns
• Update dummy device and library to match new columns

Fixed

• Fixed crash for wireless connected devices while trying to get the "Date Modified column" value
• Fixed setting correct sync status for column_status if no status is sent from KOReader

0.5.1-beta - 2022-12-27

Added

• Add support for Date Synced column (stores date of last sync from KOReader to Calibre)
• Add support for Date Modified column (stores date modified of KOReader Sidecar)

Changed

• Standardized results message format
• code cleanup to pass linting

Fixed

• Error in results message
• Fix error in debug_print definition

0.5.0-beta - 2022-12-27

Added

• Add "Sync Missing Sidecars to KOReader" functionality

Changed

• Vendor in slpp.py instead of adding it as a separate dependency to reduce fragility

0.4.1-beta - 2022-11-08

Changed

• Use calibre's built-in UTC timezone (source), because tzdata isn't available on Windows (see #13)

0.4.0-beta - 2022-11-02

Added

• Calculate first and last bookmark date columns

0.3.2-beta - 2022-09-16

Added

• Enable POCKETBOOK632

0.3.1-beta - 2022-09-15

Added

• Enable USER_DEFINED

0.3.0-beta - 2022-09-13

Changed

• Don't break for unknown device class, but try to sync anyway

0.2.7-alpha - 2022-02-18

Added

• Enable TOLINO, for real this time

0.2.6-alpha - 2022-02-04

Added

• Enable POCKETBOOK622

0.2.5-alpha - 2021-12-20

Added

• Enable POCKETBOOK626

0.2.4-alpha - 2021-12-12

Added

• Enable TOLINO

0.2.3-alpha - 2021-11-23

Added

• Enable KOBOTOUCHEXTENDED

0.2.2-alpha - 2021-06-22

Fixed

• Skip metadata sidecars that cannot be decoded (e.g. from a very old version of KOReader)

Changed

• Use path instead of lpath for book paths to go around MTP_DEVICE lowercasing the latter
• Disable MTP_DEVICE because it cannot be supported (see #2)

Added

• Enable KINDLE2

0.2.1-alpha - 2021-04-26

Added

• An extra column for read progress as integer (because that’s what the Goodreads Sync plugin expects)

0.2.0-alpha - 2021-04-24

Added

• Support for highlights and bookmarks
• Counts to post-sync alerts
• An .editorconfig and .pylintrc to define code layout

Changed

• README.md to reflect current state of development

0.1.4-alpha - 2021-04-11

Fixed

• Multiple tweaks and bug fixes by Glen Sawyer

0.1.3-alpha - 2021-04-04

Added

• Support for SMART_DEVICE_APP devices, i.e. KOReader's wireless connnection

0.1.2-alpha - 2020-11-21

Added

• Support for KOBO and KOBOTOUCH devices

0.1.1-alpha - 2020-11-18

Added

• Support for all possible filetypes
• Variables to easily enable a dry-run when debugging and remote debugging

0.1.0-alpha - 2020-11-18

Added

• Everything needed for a first working version of the plugin
• dummy_device and dummy_library for easy debugging
• Makefile to build a plugin release as a ZIP file
• TODO to keep to-dos in one place
• environment.yml, in case anyone wants to recreate my Conda environment
• This README.md

[PeterT]

Please consider attaching the plugin to your post as opposed to making people head elsewhere

[harmtemolder]

Sorry about that, will add the latest ZIP right away

[harmtemolder]

I cannot seem to edit my original post, so I’ll attach the latest release of the plugin here. It should now support Kobo devices (tested on my Kobo Aura) as well as generic USB and folder devices.

EDIT: I crossed the threshold to be able to edit my original post, so added release 0.1.2-alpha there and removed it here.

[jd2733]

I’m most interested in getting highlights off of koreader. I was able to setup your plugin via usb, but don’t see mention of highlights in your documentation.

Thanks for working on it!

[harmtemolder]

Originally Posted by jd2733

I’m most interested in getting highlights off of koreader. I was able to setup your plugin via usb, but don’t see mention of highlights in your documentation.

Great to hear you were able to set it up. Unfortunately I haven’t gotten around to adding syncing of highlights. They are high on my wishlist as well, just haven’t had the time lately to work on the plugin.

[bitnerd]

Ooh, I'll be watching this closely. I would love to be able to do this over the network

[twowheels]

EDIT: Found it, shortly after posting, ignore the rest. The assignment is indirect as the status flags are enumerated and then displayed as buttons on the book status page, within the book.



I was just exploring the source to see how hard it would be to fix something that's been bugging me and I saw the "abandoned" status flag, with the associated "On hold" text, which led me to a google search that brought me here.

I can't seem to figure out how to "abandon" a book. I have one book that I want to abandon for now (got bored with it), but the only option I see when holding down on the book is "Mark as read", "Add to favorites", etc, but nothing to "abandon" it or put it on hold. That said, your plugin description above says that it only syncs from device to Calbire, so it must be something that can be set on the device, yet I can only find comparisons against the value, nothing that sets the value in the source making me think that it's an incomplete feature or one that has only been partially removed.

[harmtemolder]

I just uploaded v0.1.3-alpha which now supports KOReader's wireless connnection, but that will not work until this pull request is part of your KOReader app.

[harmtemolder]

Originally Posted by bitnerd

Ooh, I'll be watching this closely. I would love to be able to do this over the network

Since the most recent release of KOReader this works. Enjoy

[harmtemolder]

Originally Posted by jd2733

I’m most interested in getting highlights off of koreader.

I’ve added support for highlights. Let me know what you think

[SpaceGun]

Hello! I'm using 0.2.0 with calibre 5.17 (installed from calibre's "Get new plugins" menu). When I attempt the "Sync from KOreader" option I get the following error:

calibre, version 5.17.0
ERROR: Unhandled exception: <b>UnicodeDecodeError</b>:'utf-8' codec can't decode byte 0x97 in position 10: invalid start byte

calibre 5.17 embedded-python: True is64bit: True
Linux-5.4.0-71-generic-x86_64-with-glibc2.2.5 Linux ('64bit', 'ELF')
('Linux', '5.4.0-71-generic', '#79-Ubuntu SMP Wed Mar 24 10:56:57 UTC 2021')
Python 3.8.5
Interface language: None
Successfully initialized third party plugins: Annotations (1, 15, 0) && KOReader Sync (0, 2, 0) && Kobo Utilities (2, 13, 1)
Traceback (most recent call last):
File "calibre_plugins.koreader.action", line 327, in sync_to_calibre
sidecar_contents = self.get_sidecar(device, sidecar_path)
File "calibre_plugins.koreader.action", line 217, in get_sidecar
parsed_contents = self.parse_sidecar_lua(contents.decode())
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x97 in position 10: invalid start byte

The error persisted while using version 0.2.1 from the sourcehut, along a very similar one when I tried it (0.2.1) from a Windows partition. Any leads would be much appreciated!

[harmtemolder]

I’ve sent you a PM to get to the bottom of this

[bitnerd]

Seems to work quite well wirelessly on Calibre 5.18 with plugin 0.2.0

The modified date wasn't coming across but I may have a typo so I'll keep messing with it

[SpaceGun]

harmtemolder has very kindly sorted out the issue I was having. All the features I've tried (bookmarks, percent read, reading status) are syncing correctly. Thanks!