Koreader Highlights utility

[embryo]

KOHighlights



KOHighlights is a utility for viewing and converting the Koreader's history files to simple text, html or csv files.
This is a totally re-written application using the Qt framework (PySide).
The original KOHighlights (using the wxPython) can be found here.





Usage

• Load items by:
  1. Selecting the reader's drive or any folder that contains books that where opened with Koreader. This will automatically load all the history files from all subdirectories.
  2. Drag and drop files or folders. This will load the files and/or all the files inside the folders.
• View the highlights and various info for a book by selecting it in the list.
• Save the highlights to the "Archive" and view them, even if your reader is not connected.
• Merge highlights from the same book that is read in two different devices and/or sync its reading position.
• Double click or press the Open Book button to view the book in your system's reader.
• Save all the selected books' highlights in:
  1. a text/html/csv file for every selected book or
  2. a single text/html/csv file with all highlights combined
• Clear the .sdr folders with the metadata or the books in the eReader.

Downloads

Check the latest release on the Downloads Page.
Read the version history at App's Page.

[embryo]

• v2.3.1.0
  Fixed:
  • When syncing highlights the newer notes replace the older ones.
  • Some fixes in the User Guide formatting and GUI.

• v2.3.0.0
  Added:
  • New User Guide. Can be accessed from the User Guide tab in the About dialog
    or online at GitHub.
  • System tab in the About dialog, with some basic system information.
  • Drag and drop support for changing the current database on the fly.
  • Experimental (and slightly hidden) support for splitting highlight chapters
    if exported to markdown.
  Changed:
  • Hitting the `Delete` keyboard key while in Books View, will just remove the
    selected book entries from the layout without deleting anything from the
    storage.
  Fixed:
  • Drag and drop of metadata files/folders was disabled.
  • Sorting highlights in Highlights View by Chapter name could lock the app.
  • Creating a new database file was not possible.

• v2.2.1.1
  • Custom Markdown templates were not applied.

• 2.2.0.0
  Added:
  • New Preferences panel.
  • Custom template for the Markdown export.
  • Right click an erroneous Sync Group item to ignore the MD5 checksum for syncing.
  Fixed:
  • A missing metadata file during loading could crash the app.
  • If the statistics plugin was disabled, the metadata could not be archived.
  • Some smaller bugs.

• v2.1.1.0
  • Completely erasing a highlight's note, made this note appear as blank in KOReader.

• v2.1.0.0
  Added:
  • Support for displaying the book's reference page numbers (if any).
  • Right-click to copy the Archived version to all other versions of a Sync group.
  • Right-click to open all versions of a Sync group in Books View.
  Changed:
  • No total pages in the "Book Info" if the book uses reference pages.
  • The arrangement of the columns in Highlights View is remembered between sessions.
  Fixed:
  • No Archived items loaded if the app starts in the Archived mode.
  • Problems with .csv/.md export from Highlights View.
  • Portable mode did not remember last used View.
  • Some window artifacts on app's startup.

• v2.0.4.0
  • Small sync bugfix.

• v2.0.3.0
  • Could not archive a book if there was a bookmark before the first highlight.

• v2.0.2.0
  • Support for displaying the user edited (inside KOReader) metadata.
  • Right clicking menu in the Highlights View lets you copy any column's value.
  • Sync Group paths are now collapsible.
  • Some more GUI changes.

• v2.0.0.0
  This is a major update for KOHighlights.
  It is adding a new concept, the Sync Groups.
  Sync Groups can be used to sync reading position and merge highlights between multiple books in different devices.
  These are groups of paths to different sidecar folder contents (metadata .lua files), that can be synced together and to the database.
  There are some GUI additions, like dark/light themes and new icons.
  Also, the app has now preliminary support for the upcoming changes to the KOReader's metadata structure.
  Finally, there are some compatibility changes.
  Because of the transition from Python 2/PySide to Python 3/PySide2, Windows XP are no longer supported and the Installer must uninstall any previous 1.x version.
  For Linux, the supported distributions are from Ubuntu 18 and up.
  
  
  • Sync Groups as a way to sync position/merge highlights between books on different devices.
  • Option for Dark/Light/No theme and old/new icons.
  • Support for the upcoming changes to the KOReader's metadata.
  • Upgrade code to Python 3, Pyside2, so no more WindowsXP and Ubuntu 16 support.
  • When merging highlights between different devices, the app tries to re-compute the page numbers, so that are not all over the place.
  • Merging highlights will now keep the latest version of their comments.
  • Better wording with fewer items for the Export menu, if only one item is selected.
  • Can now delete highlights from the Highlights view.
  • Removed the requirement for the Statistics plugin to be enabled.
  • Some times there was an error when deleting a highlight. Fixed.
  • Delete a highlight from the highlight list some times did not work. Fixed.
  • Display wrong highlight count if more than one highlight on the same page. Fixed.

• v1.7.6.0
  • Problematic PyInstaller Windows executable.

• v1.7.5.0
  • Added PySide6 support.
  • "Find in Books" for a highlight was finding the book but did not select the highlight.

• v1.7.4.0
  • Some problems with relative paths in Linux.

• v1.7.3.0
  • Could not parse highlights due to changes in KOReader latest release.

• v1.7.2.0
  • This is a Windows only release that fixes the startup problem that v1.7.1.0 introduced.

• v1.7.1.0
  • This is a Windows only release that adds a portable mode for... Windows.
    When KOHighlights runs in Portable mode, will use a portable_settings directory to store its settings, that is located inside the installation directory of the app.
    Because of this, it is advised to not install the app inside the Program Files folder if you indent to use it as portable.
    There are two ways to start the app in Portable mode:
    • Run the KoHighlights Portable.exe that is located next to the KoHighlights.exe.
    • Run KoHighlights.exe with a -p argument. E.g. create a shortcut for the KoHighlights.exe and add a space and a -p argument at the end of the target filename.

• v1.7.0.0
  • Export highlights as markdown files.
  • Chapter title is available as part of the highlight data.
  • The document path from the metadata is used to locate the book if the .sdr folder is in the "DocSettings" directory of KOReader.
  • Fixed: "Scan Directory" and "About" shortcuts weren't working.

• v1.6.0.0
  • New toolbar button "Filter" to filter the current view's items by searching for keywords in Highlights and/or Comments and/or Book titles..
  • Added an arrow to the "Archived" view button, to invoke a drop-down menu with actions for the Archive database (other than right-clicking on it)
  • Problem when exporting highlights from books with very long filenames (Github issue here).</li>

• v1.5.0.0
  • "Show in Highlights" menu has a new entry, "Date Format", where you can customize the appearance of the date using format codes (info here).
    This influences the way the date is formatted when the highlights are exported, and the way its presented at the "Book Info" panel.
    It is not supported in the CLI side of the app though.
  • Fixed exporting highlights to csv from "Highlights View" that did not work.
  • Fixed some display problem when the system uses a dark theme.

• v1.4.4.0
  • A new column with the number of comments.
  • User's review is visible in "Book Info", if one exists.
  • More checks are added to the highlight extraction from the metadata file.

• v1.4.2.0
  • Right clicking the "Archived" button produces a menu with actions for the Archive database. That way we can use a database file that is located anywhere, even on the cloud.
  • Exporting without the Page numbers but sorted "By Date" hung the app.
  • Exporting highlights from multiple books to individual files produced text files that contained html code or the opposite.

• v1.4.0.0
  • Export highlights to csv is now supported (in GUI and CLI), using tabs as separators and double quotes (") as text qualifiers.
  • Selecting the exported file type/mode is done with popup questions.

• v1.3.0.0
  • Option to sync/merge a loaded book with a metadata file from any location.
  • Option to recalculate (and re-save in the metadata) the MD5 for a book.
  • Option to open the selected book's metadata location with the system's file manager.
  • Keyboard shortcuts to copy selected highlights/comments.
  • All found "metadata.lua" files are loaded (if more than one are present in the .sdr directory).
  • When checking if two books are the same (for sync/merge), the app now uses any MD5 present in the book's metadata, and if none exists, it calculates one.
  • Fixed: Canceling export of highlights still opened a "Success" dialog.
  • Fixed: Sometimes the modified date had decimal digits.
  • Fixed: Some cosmetic bugs.

• v1.2.2.0
  • Option to ignore the English articles at the start of the title while sorting.
  • Option to merge highlights/sync position with the archived version (using right click on loaded book).
  • New "Rating" column.
  • There is an optional question dialog when adding/merging books in the archive.
  • "Abandoned" book rows now have red text.
  • Single launch App.
  • No more Linux.x86 binary (nobody used it anyways...).
  • Fixed: Some problems with drag&dropping books.
  • Fixed: Wrong "Delete" button when in Archived mode.
  • Fixed: Better Python 3 compatibility (for the .py version).

• v1.0.0.0
  • New Archive mode. Using a local database in which we can add/upgrade any loaded book's metadata (with the highlights of course) that we get from the reader, and access it even if the reader is not connected.
  • We can now export to html from "Highlights" view too.
  • The user interface has 2 modes now ("Loaded" and "Archived"). In any of these modes we can view the highlights using two views: as rows of "Books" or as rows of "Highlights".
  • Fixed: Could not hide Comments while viewing or exporting highlights.
  • Fixed: Drag & dropping multiple folders, was loading only the last one.

Older versions:

Spoiler:

• v0.9.1.0
  • Code refactoring.
  • Fixed: Comments in highlights with more than one paragraphs could not get changed.

• v0.9.0.0
  • Option to export highlights in HTML format.
  • Linux binaries now support the command-line interface.
  • Fixed: The sorting direction (ascending-descending) sometimes was not saved.

• v0.8.0.0
  • Option to change the size of the toolbar buttons with right-click.
  • Linux binaries for x86/x64 with PyInstaller.
  • The Python script has now a command-line interface for batch conversions.
  • Fixed: The Python script can run with Python2/3.
  • Fixed: Reloading the same files after clearing the book list, works now.

• v0.7.0.0
  • Option to sort highlights by Date or Page (in "Books" view).
• v0.6.1.0
  • Merging the highlights of a book that is read in two different devices is now possible (beta). It must be from the same reader engine though. Read about it here.
  • Syncing the reading position for a book that is read in two different devices is also possible now.
  • We can now edit highlight comments while in the "Highlights" view, using the right-click menu or double clicking.
  • From "Highlights" view, using the right-click menu, we can locate the current highlight to the "Books" view.
  • Changed: When scanning a directory using the "Scan Directory" button, the previously loaded books are not unloaded from the list.
  • Fixed: The same book's metadata are not loaded more than one time any more.
  • Fixed: The book's description text is now read only (as it should).
• v0.5.0.0
  • In "Highlights View", using the right-click menu, we can copy to clipboard the selected highlights/comments.
  • Added "Tags" and "Description" to the "Book Info".
  • Removed "Total time" and "Status" from the "Book Info" because the relevant keys where removed from the more current KoReader versions' metadata.
  • Fixed: Some highlights where missing when exported to text file.
  • Fixed: Better handling of lua comments.
• v0.4.0.0
  • New "Highlights View", where we can see all the highlights from the opened books and sort them by date, book etc.
  • Fixed: Update book info when navigating using the arrows.
• v0.3.5.0
  • Support for comments (kind of), using the "Edit" highlight option of KoReader (click on a highlight).
  • Edit/Add/Delete a highlight's comment.
  • Fixed: Deleting an "Edited" highlight could not delete the associated bookmark too.
  • Fixed: New lines in the highlight are now properly displayed.
• v0.3.4.0
  • Delete some or all the highlights from a book.
  • Copy some or all the highlights from a book in the clipboard.
  • Highlights are now a list with items that can be selected individually for copy/deletion, not just a text.
  • Highlight's elements like page, date and even highlight text can be individually turned on/off.
  • Fixed: If there was more than one highlight in a page, only the first was displayed.
  • Fixed: Right click menu appeared even if the book list was empty.
• v0.3.2.0
  • Page & Date reference for each highlight.
  • Option to show/hide page & date info that also affects if that info is going to be written in the saved text files.
  • Highlights are now sorted by page number.
  • Right clicking and selecting "View Book" did nothing.
  • If another file was in an '.sdr' folder (like an '.old' backup file) it could be selected instead of the normal metadata file.
  • Saved texts were displayed wrong in Windows Notepad due to Unix line breaks.
• v0.3.1.2
  • New "Book Info" panel.
  • Option to delete the .sdr folders for specific books or even the books themselves.
  • Option to view a book with the system's viewer.
  • Added "Book Type" and "Read percent" columns.
  • Color-coded rows based on Finished/Completed state and the presence or not of the actual book.
• v0.2.1.1
  • Added a "Modified" date column and option to pass
    folders or files as arguments.
• v0.2.0.1
  • Totally re-written application with new GUI
  • New GitHub repository
  • Initial release
• v0.1.5
  • Release version, nothing more to add
• v0.1.1
  • Better Highlights detection
  • Cosmetic changes
• v0.1.0
  • Initial release

[Ken Maltby]

What does "Batch Convert" mean, in this case? Are they being converted to simple .txt format/files?

The History files processing should be a real good thing for those working with highlights. Perhaps Markismus could place his notes in the History files, when he makes his note taking plugin/wiget.

Luck;
Ken

[embryo]

Originally Posted by Ken Maltby

What does "Batch Convert" mean, in this case? Are they being converted to simple .txt format/files?

Yes.
There are two ways to do this.
One, is to select a directory with many history files, and a directory for the output text files.
The other, is to just drag and drop many files on the program and it will convert the ones that end to ".lua" AND have highlights in them to text files in the same directory as the dropped files.

[embryo]

Just find out that KoReader has its own place here!

It would be very nice if the moderators could move this thread there, where it would be more helpful.

Thanks

[Ken Maltby]

Originally Posted by embryo

Just find out that KoReader has its own place here!

It would be very nice if the moderators could move this thread there, where it would be more helpful.

Thanks

I did add a post with a link to the utility's Github page. Since your utility should work with any KOReader installation where the history/.sdr files can be accessed, it should get moved.

Luck;
Ken

[embryo]

Updated the first post for more info.

In a nutshell, I just release the v2 of the KoHighlights utility.

Its a different application, written from scratch using Pyside (Qt for Python) that has the same name with the old one because it has the same goal of reading and converting the KoReader's highlights to text.
But it has a nicer interface and its ready to also import the notes that the great Koreader developers will implement on day :-p

[boriar]

Thanks for a new version. I will try asap

[johnnyb]

Very nice, thank you! And right at the time when the Evernote plugin ceased to work!

[sbaguz]

Great idea!
Just downloaded: I'm going to test it...

[embryo]

Updated KoHighlights to v0.2.1.1.
Added a "Modified" date column and option to pass folders or files as arguments.

[embryo]

Updated to v0.3.1.2

• New "Book Info" panel.
• Option to delete the .sdr folders for specific books or even the books themselves.
• Option to view a book with the system's viewer.
• Added "Book Type" and "Read percent" columns.
• Color-coded rows based on Finished/Completed state and the presence or not of the actual book.

[johnnyb]

Thanks!
Tried installing but it won't work because it will not accept my Visual Studio installation (which I downloaded from MS).
Edit: I'm on Win 10.

[embryo]

Originally Posted by johnnyb

Thanks!
Tried installing but it won't work because it will not accept my Visual Studio installation (which I downloaded from MS).
Edit: I'm on Win 10.

I checked with the files that the installer suggests if missing from your system (Visual C++ 2008 9.0.30729.6161) also from Microsoft (x64, x86) in Win 10, and it works OK.
What is your version?
Is it x64?
Maybe you should also install the x86 version.

[johnnyb]

Originally Posted by embryo

I checked with the files that the installer suggests if missing from your system (Visual C++ 2008 9.0.30729.6161) also from Microsoft (x64, x86) in Win 10, and it works OK.
What is your version?
Is it x64?
Maybe you should also install the x86 version.

I actually tried both versions (system is 64bit but I also installed the 32bit version), with and without reboot and all, to no avail...

Edit: Great! I just had the system install a bunch of updates and repaired the two installations and now it works! Thanks for your help!

Edit 2: Fantastic tool!