Wallabag (read it later app) automated synchronization

[anarcat]

Originally Posted by ajkessel

This is now done. Thanks!
Is there any way to trigger a sync from koreader or do you need to exit out to the main interface first?

Not directly no. I was hoping to add some hook in koreader to fire up wallabako, but they ended up writing their own wallabag client instead (!). What I do is i mark the wallabako-run shell script as a "favorite" and long-press it in the favorites list, then hit "execute shell script". The problem with that is that koreader just hangs completely while the script runs, so you need to make sure the script has a & in it...

But that's a lot of tap tap, i just suspend and resume, which triggers the udev stuff anyways.

Originally Posted by ajkessel

It is syncing fine but does not appear to reflect "finished" status back on Wallabag when marked as done with the native reader. I've pasted a log below.

I did just discover if I mark it finished in koreader, it is syncing back to Wallabag properly, though.
[...]

2024/10/23 16:34:27 config after commandline parsing: main.wallabakoConfig{WallabagConfig:wallabago.WallabagConfig{REDACTED}, Debug:true, Delete:true, LogFile:"", Database:"/mnt/onboard/.kobo/KoboReader.sqlite", Concurrency:2, Count:-1, Exec:"", OutputDir:"/mnt/onboard/wallabako/", PidFile:"", RetryMax:4, Tags:"kobo", PlatoConfig:main.PlatoConfig{LibraryPath:""}, Fbink:false, FbinkInteractive:false}
2024/10/23 16:34:27 loaded configuration from /etc/wallabako.js
2024/10/23 16:34:27 trying lockfile path /var/run/wallabako.pid
2024/10/23 16:34:27 logging in to https://my-wallabag-domain-redacted
2024/10/23 16:34:27 username: REDACTED, password: REDACTED
2024/10/23 16:34:28 CSRF token found: 200 OK
2024/10/23 16:34:28 logged in successful: 302 Found
2024/10/23 16:34:59 found 26 unread entries, will check -1
2024/10/23 16:34:59 dispatching 110
2024/10/23 16:34:59 dispatching 109
2024/10/23 16:34:59 dispatching 105
2024/10/23 16:34:59 URL https://my-wallabag-domain-redacted/export/110.epub older than local file /mnt/onboard/wallabako/110.epub, skipped (2024-10-23 16:28:30 -0400 EDT > 2024-10-23 20:20:02 +0000 +0000)
2024/10/23 16:34:59 dispatching 108
2024/10/23 16:34:59 URL https://my-wallabag-domain-redacted/export/105.epub older than local file /mnt/onboard/wallabako/105.epub, skipped (2024-10-21 23:05:52 -0400 EDT > 2024-10-22 01:20:11 +0000 +0000)
2024/10/23 16:34:59 dispatching 107
2024/10/23 16:34:59 URL https://my-wallabag-domain-redacted/export/108.epub older than local file /mnt/onboard/wallabako/108.epub, skipped (2024-10-22 03:05:52 -0400 EDT > 2024-10-22 01:19:39 +0000 +0000)
2024/10/23 16:34:59 dispatching 106
2024/10/23 16:34:59 URL https://my-wallabag-domain-redacted/export/107.epub older than local file /mnt/onboard/wallabako/107.epub, skipped (2024-10-22 03:05:56 -0400 EDT > 2024-10-22 01:17:24 +0000 +0000)
2024/10/23 16:34:59 dispatching 103
2024/10/23 16:34:59 URL https://my-wallabag-domain-redacted/export/106.epub older than local file /mnt/onboard/wallabako/106.epub, skipped (2024-10-22 08:28:38 -0400 EDT > 2024-10-22 01:08:12 +0000 +0000)
2024/10/23 16:34:59 skipping 102 (not in Tags)
2024/10/23 16:34:59 skipping 101 (not in Tags)
2024/10/23 16:34:59 skipping 99 (not in Tags)
2024/10/23 16:34:59 skipping 97 (not in Tags)
2024/10/23 16:34:59 skipping 96 (not in Tags)
2024/10/23 16:34:59 skipping 88 (not in Tags)
2024/10/23 16:34:59 skipping 94 (not in Tags)
2024/10/23 16:34:59 skipping 95 (not in Tags)
2024/10/23 16:34:59 skipping 87 (not in Tags)
2024/10/23 16:34:59 skipping 85 (not in Tags)
2024/10/23 16:34:59 skipping 80 (not in Tags)
2024/10/23 16:34:59 skipping 73 (not in Tags)
2024/10/23 16:34:59 skipping 72 (not in Tags)
2024/10/23 16:34:59 skipping 68 (not in Tags)
2024/10/23 16:34:59 skipping 67 (not in Tags)
2024/10/23 16:34:59 skipping 66 (not in Tags)
2024/10/23 16:34:59 skipping 64 (not in Tags)
2024/10/23 16:34:59 skipping 63 (not in Tags)
2024/10/23 16:34:59 skipping 62 (not in Tags)
2024/10/23 16:34:59 URL https://my-wallabag-domain-redacted/export/103.epub older than local file /mnt/onboard/wallabako/103.epub, skipped (2024-10-22 08:28:40 -0400 EDT > 2024-10-21 20:30:45 +0000 +0000)
2024/10/23 16:34:59 URL https://my-wallabag-domain-redacted/export/109.epub older than local file /mnt/onboard/wallabako/109.epub, skipped (2024-10-22 08:28:38 -0400 EDT > 2024-10-22 01:26:05 +0000 +0000)
2024/10/23 16:34:59 local files to inspect: [/mnt/onboard/wallabako/103.epub /mnt/onboard/wallabako/105.epub /mnt/onboard/wallabako/106.epub /mnt/onboard/wallabako/107.epub /mnt/onboard/wallabako/108.epub /mnt/onboard/wallabako/109.epub /mnt/onboard/wallabako/110.epub] /mnt/onboard/wallabako/*.epub
2024/10/23 16:34:59 parsing Plato metadata file /mnt/onboard/.metadata.json and /mnt/onboard/.reading-states/
2024/10/23 16:34:59 could not load Plato metadata: open /mnt/onboard/.metadata.json: no such file or directory
2024/10/23 16:34:59 cannot read metadata file: /mnt/onboard/wallabako/103.sdr/metadata.epub.lua
2024/10/23 16:34:59 nickel book 103 status: 0
2024/10/23 16:34:59 path /mnt/onboard/wallabako/105.epub: fingerprint 2720593106042078941, mtime 1729566352, mtime tz Local, size 247517, diff 9223372036
2024/10/23 16:34:59 found koreader percent 0.518519 in /mnt/onboard/wallabako/105.sdr/metadata.epub.lua
2024/10/23 16:34:59 koread book /mnt/onboard/wallabako/105.sdr/metadata.epub.lua not complete: "reading"
2024/10/23 16:34:59 koreader status of /mnt/onboard/wallabako/105.sdr/metadata.epub.lua is 1
2024/10/23 16:34:59 koreader book 105 status not unread: 1
2024/10/23 16:34:59 path /mnt/onboard/wallabako/106.epub: fingerprint 2720593106041898045, mtime 1729600118, mtime tz Local, size 66621, diff 9223372036
2024/10/23 16:34:59 cannot read metadata file: /mnt/onboard/wallabako/106.sdr/metadata.epub.lua
2024/10/23 16:34:59 nickel book 106 status: 0
2024/10/23 16:34:59 path /mnt/onboard/wallabako/107.epub: fingerprint 2720593106042118421, mtime 1729580756, mtime tz Local, size 286997, diff 9223372036
2024/10/23 16:34:59 cannot read metadata file: /mnt/onboard/wallabako/107.sdr/metadata.epub.lua
2024/10/23 16:34:59 nickel book 107 status: 0
2024/10/23 16:34:59 path /mnt/onboard/wallabako/108.epub: fingerprint 2720593106041837818, mtime 1729580752, mtime tz Local, size 6394, diff 9223372036
2024/10/23 16:34:59 cannot read metadata file: /mnt/onboard/wallabako/108.sdr/metadata.epub.lua
2024/10/23 16:34:59 nickel book 108 status: 0
2024/10/23 16:34:59 path /mnt/onboard/wallabako/109.epub: fingerprint 2720593106042265426, mtime 1729600118, mtime tz Local, size 434002, diff 9223372036
2024/10/23 16:34:59 cannot read metadata file: /mnt/onboard/wallabako/109.sdr/metadata.epub.lua
2024/10/23 16:34:59 nickel book 109 status: 0
2024/10/23 16:34:59 path /mnt/onboard/wallabako/110.epub: fingerprint 2720593106041905474, mtime 1729715310, mtime tz Local, size 74050, diff 9223372036
2024/10/23 16:34:59 cannot read metadata file: /mnt/onboard/wallabako/110.sdr/metadata.epub.lua
2024/10/23 16:34:59 nickel book 110 status: 0
2024/10/23 16:34:59 14 open file descriptors: [/proc/self/fd/0 /proc/self/fd/1 /proc/self/fd/2 /proc/self/fd/3 /proc/self/fd/4 /proc/self/fd/5 /proc/self/fd/6 /dev/pts/0 /dev/pts/0 /dev/pts/0 socket:[17780] anon_inode:[eventpoll] anon_inode:[eventfd] readlink /proc/self/fd/6: no such file or directory]
2024/10/23 16:34:59 version 1.5.1-30-g4d92dfb completed in 32.655s, processed: 7, downloaded: 0, size: 0 B, deleted: 0, read: 0, unread: 26

I think you might be in a situation where the same "books" (wallabag articles, really) have metadata in both systems, and wallabag gets confused. The current logic for this is that the first book that returns a "not unread" status wins, and that status is used to determine the book status.

In this case, we're probably talking about article 105:

2024/10/23 16:34:59 path /mnt/onboard/wallabako/105.epub: fingerprint 2720593106042078941, mtime 1729566352, mtime tz Local, size 247517, diff 9223372036
2024/10/23 16:34:59 found koreader percent 0.518519 in /mnt/onboard/wallabako/105.sdr/metadata.epub.lua
2024/10/23 16:34:59 koread book /mnt/onboard/wallabako/105.sdr/metadata.epub.lua not complete: "reading"
2024/10/23 16:34:59 koreader status of /mnt/onboard/wallabako/105.sdr/metadata.epub.lua is 1
2024/10/23 16:34:59 koreader book 105 status not unread: 1

There, you can see that wallabako sees that book with the status "reading", and then skips the rest of the processing. It doesn't check Nickel at all. This is important! If, instead of treating this book as "touched" it would look in nickel, it could mark the book as "read" and *delete it locally*.

My workflow here is that I sometimes have articles that I have annotations into that I want to keep, and i *don't* mark them as read in koreader to avoid them being deleted.

I understand this might be confusing though, if you tried to mark it as read in the Nickel interface. I'm open to suggestions on how to deal with such conflicts... But is it what's happening here?