Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > KOReader

Notices

Reply
 
Thread Tools Search this Thread
Old 03-12-2020, 03:30 PM   #1
OrangeFlavored
Enthusiast
OrangeFlavored doesn't litterOrangeFlavored doesn't litter
 
Posts: 32
Karma: 100
Join Date: Oct 2014
Device: Likebook Mars
Is there any way to backup/restore reading progress to Calibre?

I know I asked this a year or two ago and the answer was basically no, but curious if something has changed since then. Back when I was using my old Kobo H2O I always liked being able to automatically backup all my reading progress to Calibre each time I plugged it in using KoboUtilities plugin, or push it all back to the device when necessary. Even integrated with goodreads so it would update my finished status on there when it did so... Kinda miss the functionality since switching to my Likebook Mars and using KOReader as the main reader app.

Experimented around a bit with the auto-sync feature built into KOReader which could maybe potentially serve a similar purpose but I think it's limited to the specific book you're reading? Or is there a way to push reading progress for your entire library at once? I usually leave wifi off since it has a large impact on battery life for my device so updates are kind of sporadic and mostly happen when I specifically go out of my way to let it sync.

Also doesn't necessarily have to go through Calibre (though that would be preferred) but if someone knows of another good quick way to backup/restore your reading positions and started/finished/etc status for your entire library at once that might be good enough for me for now, since I could probably just use KOReader on whatever new device I get in the future anyway and import that data.

Edit: Oh, actually looking at the contents of my SD card it looks like that data is stored in a pretty easily readable form in the same folder as the epub files... I guess backing that up should be easy enough.

Last edited by OrangeFlavored; 03-12-2020 at 03:36 PM.
OrangeFlavored is offline   Reply With Quote
Old 03-12-2020, 03:52 PM   #2
ilovejedd
hopeless n00b
ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.ilovejedd ought to be getting tired of karma fortunes by now.
 
ilovejedd's Avatar
 
Posts: 4,640
Karma: 13510762
Join Date: Jan 2009
Location: in the middle of nowhere
Device: PW4, PW3, Libra H2O, iPad 9.7, iPad 10.5, iPad 12.9
The <book_filename>.sdr folders (stored in the same directory as the books) should contain that info.
ilovejedd is offline   Reply With Quote
Advert
Old 03-12-2020, 06:36 PM   #3
OrangeFlavored
Enthusiast
OrangeFlavored doesn't litterOrangeFlavored doesn't litter
 
Posts: 32
Karma: 100
Join Date: Oct 2014
Device: Likebook Mars
Yeah for now just automated the backup process using Windows Task Scheduler, figured I'd share my process if anyone else is interested in doing so:

First open up Event Viewer and go to Applications and Services -> Microsoft -> Windows -> DriverFrameworks-UserMode and right click Operational, Enable Log. Then plug your device in, refresh the list, find a bunch of Verbose items that popped up, go to Details, copy down the InstanceId and paste it into this:

Code:
<QueryList>
  <Query Id="0" Path="Microsoft-Windows-DriverFrameworks-UserMode/Operational">
    <Select Path="Microsoft-Windows-DriverFrameworks-UserMode/Operational">*[System[Provider[@Name='Microsoft-Windows-DriverFrameworks-UserMode'] and (Level=5) and Task = 33 and (EventID=2004)]]
	and
	*[UserData[UMDFHostAddDeviceBegin[(InstanceId='PASTE_YOUR_INSTANCE_ID_HERE')]]]</Select>
  </Query>
</QueryList>
Also make sure to replace any & with &amp; since they'll screw up your filter. Then you go to Task Scheduler and make a new task with the trigger set to "Custom event filter" and click Custom, Edit Event Filter, XML, check "Edit query manually" and paste that in there. From here you can have your task do whatever is appropriate for your particular e-reader, in my case I couldn't figure a good way to copy files from a MTP device on the command prompt so I just enabled USB debugging and did it through the ADB like so:

Code:
adb shell mkdir /sdcard/Download/Temp
adb shell cp -r /sdcard/Books/*.sdr/ /sdcard/Download/Temp
adb pull /sdcard/Download/Temp G:\Backup\KOReader\%date:~-8,4%_%date:~-14,2%_%date:~-11,2%
adb shell rm -r /sdcard/Download/Temp
Saved that in a .bat file and set as target for the created task, and then saved it. Finally set it as hidden and "Run whether user is logged in or not" so that it won't pop up a command prompt window and runs silently in the background. Seems to be working out well so far, copies all those folders over silently each time the device is plugged in, so if something goes wrong should have a decent amount of backups. Pretty much serves the same purpose as what the old KoboUtilities plugin did so I am content for now. Actually KoboUtilities tended to take a pretty long time to fetch all that data when I plugged my device in so in that regard this is better, happening in the background and only taking a second or two.

If you can't or don't want to enable USB debugging but have some experience with Python I know there's a few libraries there such as PyMTP that allow you to copy files over from a MTP device, that might work out for you. Also if your device doesn't use MTP and just mounts on a normal drive letter then all that is irrelevant and you can just have your batch file copy the folders over as normal.

Last edited by OrangeFlavored; 03-12-2020 at 07:09 PM.
OrangeFlavored is offline   Reply With Quote
Old 03-12-2020, 07:39 PM   #4
pazos
cosiñeiro
pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.pazos ought to be getting tired of karma fortunes by now.
 
Posts: 454
Karma: 435255
Join Date: Apr 2014
Device: BQ Cervantes 4
As ilovejedd said the info is stored on a sidecar dir, they will be stored in koreader/history if the path is read only.

KOReader was hacked on top of a filemanager and lacks the concept of "library".The best way to emulate one is to parse all metadata. This will give you info for any book that was opened at least once.

https://github.com/noembryo/KoHighlights is a standalone tool that does that very well (iterate over a path looking for metadata).
pazos is offline   Reply With Quote
Old 03-12-2020, 08:04 PM   #5
NiLuJe
BLAM!
NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.
 
NiLuJe's Avatar
 
Posts: 10,302
Karma: 17931787
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, PW & PW2; Kobo H2O & Forma
On a tangentially related note, there's also https://github.com/koreader/koreader/issues/3814 which might warrant another poke to see if @onde2rock got stuff back in working order.
NiLuJe is offline   Reply With Quote
Advert
Old 05-06-2020, 01:04 AM   #6
OrangeFlavored
Enthusiast
OrangeFlavored doesn't litterOrangeFlavored doesn't litter
 
Posts: 32
Karma: 100
Join Date: Oct 2014
Device: Likebook Mars
Experimented around with Calibre's database API as mentioned here and managed to set up a little CLI plugin to pull read percentage and last read time from a folder full of KOreader metadata into Calibre.

Code:
import os
import datetime
from dateutil.tz import tzlocal
from calibre.customize import Plugin
from calibre.library import db

class KoreaderPercentSync(Plugin):
	name                = 'KOreader Percent Sync'
	description         = 'Sync KOReader Percentage from command line'
	supported_platforms = ['windows', 'osx', 'linux']
	author              = 'Orange'
	version             = (0, 0, 3)
	minimum_calibre_version = (0, 7, 53)
		
	def cli_main(self, args):
		target = args[1]
		database = db(args[2]).new_api
		to_update_percent, to_update_date = {}, {}
		for root, dirs, files in os.walk(target):
			for target_folder in dirs:
				book = int(target_folder[:-4].rpartition(" - ")[2])
				calibre_percent = database.field_for("#perc_read", book)
				if calibre_percent != 1:
					percent, date = parse_metadata(os.path.join(root, target_folder, "metadata.epub.lua"))
					if not calibre_percent or abs(calibre_percent - percent) > 0.00001:
						to_update_percent[book] = percent
						to_update_date[book] = date
		database.set_field("#perc_read", to_update_percent)
		database.set_field("#last_read", to_update_date)
				
def parse_metadata(meta_file):
	with open(meta_file, "r") as metadata:
		percent = 0
		date = datetime.datetime.fromtimestamp(os.path.getmtime(meta_file), tzlocal())
		for line in metadata:
			read_line = line.strip()
			if read_line.startswith("[\"percent_finished\"]"):
				percent = read_line[23:-1]
			elif read_line == "[\"status\"] = \"complete\"":
				return (1, date)
		return (float(percent), date)
Works as a command line plugin taking two arguments for KOReader files & Calibre library location. For now expects files are named with the calibre ID at the end of the file/folder name separated by " - ". .txt files with timestamp no longer needed since I finally figured a way to move the files to my PC from my e-reader without messing with the timestamps on the files themselves...

Last edited by OrangeFlavored; Yesterday at 03:12 PM.
OrangeFlavored is offline   Reply With Quote
Old 05-06-2020, 11:20 AM   #7
NiLuJe
BLAM!
NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.
 
NiLuJe's Avatar
 
Posts: 10,302
Karma: 17931787
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, PW & PW2; Kobo H2O & Forma
Can't you just stat() the metadata file to get the mtime?
NiLuJe is offline   Reply With Quote
Old 05-06-2020, 01:22 PM   #8
Frenzie
Guru
Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.Frenzie ought to be getting tired of karma fortunes by now.
 
Posts: 978
Karma: 330014
Join Date: Oct 2014
Location: Antwerp
Device: Kobo Aura H2O
At a quick glance that looks fairly cross-platform btw?
Frenzie is offline   Reply With Quote
Old 05-06-2020, 03:44 PM   #9
OrangeFlavored
Enthusiast
OrangeFlavored doesn't litterOrangeFlavored doesn't litter
 
Posts: 32
Karma: 100
Join Date: Oct 2014
Device: Likebook Mars
Quote:
Originally Posted by NiLuJe View Post
Can't you just stat() the metadata file to get the mtime?
Problem is I don't have a way of getting the metadata files from my ereader to my PC without messing up the timestamp. Text files are just a workaround for that, otherwise yeah I'd do something like that. ADB Pull has a switch to maintain timestamps, but unless I pull the files individually I need a way to move them around on the Android filesystem while maintaining timestamps as well, and I kept running into permission errors when trying to do this through various methods... Writing the timestamps back into the files and deleting the text files after moving them to my PC should be possible but would just make the process take even longer so I just left them in for now.

Quote:
Originally Posted by Frenzie View Post
At a quick glance that looks fairly cross-platform btw?
Yeah, it probably would work on other platforms too, though I only tried it on windows.

Last edited by OrangeFlavored; 05-06-2020 at 03:49 PM.
OrangeFlavored is offline   Reply With Quote
Old 05-06-2020, 03:55 PM   #10
NiLuJe
BLAM!
NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.NiLuJe ought to be getting tired of karma fortunes by now.
 
NiLuJe's Avatar
 
Posts: 10,302
Karma: 17931787
Join Date: Jun 2010
Location: Paris, France
Device: Kindle 2i, 3g, 4, 5w, PW & PW2; Kobo H2O & Forma
Oh, right, I was assuming you were working on a partition exported over USBMS. But Android, of course, means that, nope, not going to happen ;p.
NiLuJe is offline   Reply With Quote
Old 05-07-2020, 11:33 PM   #11
OrangeFlavored
Enthusiast
OrangeFlavored doesn't litterOrangeFlavored doesn't litter
 
Posts: 32
Karma: 100
Join Date: Oct 2014
Device: Likebook Mars
Turns out timestamps can be preserved if I tar everything before pulling it, so doing that now, seems faster this way anyway since I only have to pull a single file rather than hundreds of individual ones. Though I do have to be careful not to let the path get over 100 characters or it gives an error. Ran into that with a few files but for now just shortened everything enough that it's not a problem anymore.

Also changed my send to device template to append the book id to the end of each book so there's no need to search the database for each book based on title/author anymore, just uses the ID from the end of the folder name to lookup the book directly. I considered trying to mess with upload_books() to add this info to the metadata.epub.lua instead, but it seemed like generating this file for every book in my library at upload time might not be desirable for the moment so I put that idea off for now. Using a plugboard and writing the calibre db ID into the ebook's metadata could also be a potential option I guess, though depends on whether you consider it less intrusive to mess with the ebook's metadata vs sticking something in its filename. Personally I think the filename thing is fine for now since it also makes it less likely I run into an unintentional filename collision on my device or something.

Last edited by OrangeFlavored; 05-08-2020 at 05:16 AM.
OrangeFlavored is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[Kobo] Reading progress not updated in calibre anymore 69carat Devices 2 01-16-2020 11:01 PM
Backup Kindle reading position to Calibre library MagratG Calibre 3 04-11-2018 12:30 PM
[Old Thread] Calibre Backup and Restore Timber Calibre 21 06-01-2011 07:31 PM
Backup Restore Application WzeroMN Nook Developer's Corner 6 05-20-2011 12:28 AM
iLiad Backup / Restore ? ce3po iRex Developer's Corner 3 11-18-2007 05:53 PM


All times are GMT -4. The time now is 03:47 PM.


MobileRead.com is a privately owned, operated and funded community.