Is there any way to backup/restore reading progress to Calibre?

[OrangeFlavored]

I know I asked this a year or two ago and the answer was basically no, but curious if something has changed since then. Back when I was using my old Kobo H2O I always liked being able to automatically backup all my reading progress to Calibre each time I plugged it in using KoboUtilities plugin, or push it all back to the device when necessary. Even integrated with goodreads so it would update my finished status on there when it did so... Kinda miss the functionality since switching to my Likebook Mars and using KOReader as the main reader app.

Experimented around a bit with the auto-sync feature built into KOReader which could maybe potentially serve a similar purpose but I think it's limited to the specific book you're reading? Or is there a way to push reading progress for your entire library at once? I usually leave wifi off since it has a large impact on battery life for my device so updates are kind of sporadic and mostly happen when I specifically go out of my way to let it sync.

Also doesn't necessarily have to go through Calibre (though that would be preferred) but if someone knows of another good quick way to backup/restore your reading positions and started/finished/etc status for your entire library at once that might be good enough for me for now, since I could probably just use KOReader on whatever new device I get in the future anyway and import that data.

Edit: Oh, actually looking at the contents of my SD card it looks like that data is stored in a pretty easily readable form in the same folder as the epub files... I guess backing that up should be easy enough.

[ilovejedd]

The <book_filename>.sdr folders (stored in the same directory as the books) should contain that info.

[OrangeFlavored]

Yeah for now just automated the backup process using Windows Task Scheduler, figured I'd share my process if anyone else is interested in doing so:

First open up Event Viewer and go to Applications and Services -> Microsoft -> Windows -> DriverFrameworks-UserMode and right click Operational, Enable Log. Then plug your device in, refresh the list, find a bunch of Verbose items that popped up, go to Details, copy down the InstanceId and paste it into this:

<QueryList>
  <Query Id="0" Path="Microsoft-Windows-DriverFrameworks-UserMode/Operational">
    <Select Path="Microsoft-Windows-DriverFrameworks-UserMode/Operational">*[System[Provider[@Name='Microsoft-Windows-DriverFrameworks-UserMode'] and (Level=5) and Task = 33 and (EventID=2004)]]
	and
	*[UserData[UMDFHostAddDeviceBegin[(InstanceId='PASTE_YOUR_INSTANCE_ID_HERE')]]]</Select>
  </Query>
</QueryList>

Also make sure to replace any & with &amp; since they'll screw up your filter. Then you go to Task Scheduler and make a new task with the trigger set to "Custom event filter" and click Custom, Edit Event Filter, XML, check "Edit query manually" and paste that in there. From here you can have your task do whatever is appropriate for your particular e-reader, in my case I couldn't figure a good way to copy files from a MTP device on the command prompt so I just enabled USB debugging and did it through the ADB like so:

adb shell mkdir /sdcard/Download/Temp
adb shell cp -r /sdcard/Books/*.sdr/ /sdcard/Download/Temp
adb pull /sdcard/Download/Temp G:\Backup\KOReader\%date:~-8,4%_%date:~-14,2%_%date:~-11,2%
adb shell rm -r /sdcard/Download/Temp

Saved that in a .bat file and set as target for the created task, and then saved it. Finally set it as hidden and "Run whether user is logged in or not" so that it won't pop up a command prompt window and runs silently in the background. Seems to be working out well so far, copies all those folders over silently each time the device is plugged in, so if something goes wrong should have a decent amount of backups. Pretty much serves the same purpose as what the old KoboUtilities plugin did so I am content for now. Actually KoboUtilities tended to take a pretty long time to fetch all that data when I plugged my device in so in that regard this is better, happening in the background and only taking a second or two.

If you can't or don't want to enable USB debugging but have some experience with Python I know there's a few libraries there such as PyMTP that allow you to copy files over from a MTP device, that might work out for you. Also if your device doesn't use MTP and just mounts on a normal drive letter then all that is irrelevant and you can just have your batch file copy the folders over as normal.

[pazos]

As ilovejedd said the info is stored on a sidecar dir, they will be stored in koreader/history if the path is read only.

KOReader was hacked on top of a filemanager and lacks the concept of "library".The best way to emulate one is to parse all metadata. This will give you info for any book that was opened at least once.

https://github.com/noembryo/KoHighlights is a standalone tool that does that very well (iterate over a path looking for metadata).

[NiLuJe]

On a tangentially related note, there's also https://github.com/koreader/koreader/issues/3814 which might warrant another poke to see if @onde2rock got stuff back in working order.

[OrangeFlavored]

Experimented around with Calibre's database API as mentioned here and managed to set up a little CLI plugin to pull read percentage and last read time from a folder full of KOreader metadata into Calibre.

import os
import datetime
from dateutil.tz import tzlocal
from calibre.customize import Plugin
from calibre.library import db

class KoreaderPercentSync(Plugin):
	name                = 'KOreader Percent Sync'
	description         = 'Sync KOReader Percentage from command line'
	supported_platforms = ['windows', 'osx', 'linux']
	author              = 'Orange'
	version             = (0, 0, 3)
	minimum_calibre_version = (0, 7, 53)
		
	def cli_main(self, args):
		target = args[1]
		database = db(args[2]).new_api
		to_update_percent, to_update_date = {}, {}
		for root, dirs, files in os.walk(target):
			for target_folder in dirs:
				book = int(target_folder[:-4].rpartition(" - ")[2])
				calibre_percent = database.field_for("#perc_read", book)
				if calibre_percent != 1:
					percent, date = parse_metadata(os.path.join(root, target_folder, "metadata.epub.lua"))
					if not calibre_percent or abs(calibre_percent - percent) > 0.00001:
						to_update_percent[book] = percent
						to_update_date[book] = date
		database.set_field("#perc_read", to_update_percent)
		database.set_field("#last_read", to_update_date)
				
def parse_metadata(meta_file):
	with open(meta_file, "r") as metadata:
		percent = 0
		date = datetime.datetime.fromtimestamp(os.path.getmtime(meta_file), tzlocal())
		for line in metadata:
			read_line = line.strip()
			if read_line.startswith("[\"percent_finished\"]"):
				percent = read_line[23:-1]
			elif read_line == "[\"status\"] = \"complete\"":
				return (1, date)
		return (float(percent), date)

Works as a command line plugin taking two arguments for KOReader files & Calibre library location. For now expects files are named with the calibre ID at the end of the file/folder name separated by " - ". .txt files with timestamp no longer needed since I finally figured a way to move the files to my PC from my e-reader without messing with the timestamps on the files themselves...

[NiLuJe]

Can't you just stat() the metadata file to get the mtime?

[Frenzie]

At a quick glance that looks fairly cross-platform btw?

[OrangeFlavored]

Originally Posted by NiLuJe

Can't you just stat() the metadata file to get the mtime?

Problem is I don't have a way of getting the metadata files from my ereader to my PC without messing up the timestamp. Text files are just a workaround for that, otherwise yeah I'd do something like that. ADB Pull has a switch to maintain timestamps, but unless I pull the files individually I need a way to move them around on the Android filesystem while maintaining timestamps as well, and I kept running into permission errors when trying to do this through various methods... Writing the timestamps back into the files and deleting the text files after moving them to my PC should be possible but would just make the process take even longer so I just left them in for now.

Originally Posted by Frenzie

At a quick glance that looks fairly cross-platform btw?

Yeah, it probably would work on other platforms too, though I only tried it on windows.

[NiLuJe]

Oh, right, I was assuming you were working on a partition exported over USBMS. But Android, of course, means that, nope, not going to happen ;p.

[OrangeFlavored]

Turns out timestamps can be preserved if I tar everything before pulling it, so doing that now, seems faster this way anyway since I only have to pull a single file rather than hundreds of individual ones. Though I do have to be careful not to let the path get over 100 characters or it gives an error. Ran into that with a few files but for now just shortened everything enough that it's not a problem anymore.

Also changed my send to device template to append the book id to the end of each book so there's no need to search the database for each book based on title/author anymore, just uses the ID from the end of the folder name to lookup the book directly. I considered trying to mess with upload_books() to add this info to the metadata.epub.lua instead, but it seemed like generating this file for every book in my library at upload time might not be desirable for the moment so I put that idea off for now. Using a plugboard and writing the calibre db ID into the ebook's metadata could also be a potential option I guess, though depends on whether you consider it less intrusive to mess with the ebook's metadata vs sticking something in its filename. Personally I think the filename thing is fine for now since it also makes it less likely I run into an unintentional filename collision on my device or something.