Any progress with rclone?

[ajkessel]

The Kobo rclone wrapper appears defunct with no commits or email threads since 2018. Has anyone made any progress since then? rclone would have a big benefit over kobocloud in that it has bidirectional syncing, so the user could remove a book from the kobo and have that removed from the cloud, and also annotations would be updated in cloud storage. I have rclone running on the kobo but don't see a simple way to cause the synced folders to appear in my library. Anyone working on this?

[JSWolf]

Originally Posted by ajkessel

The Kobo rclone wrapper appears defunct with no commits or email threads since 2018. Has anyone made any progress since then? rclone would have a big benefit over kobocloud in that it has bidirectional syncing, so the user could remove a book from the kobo and have that removed from the cloud, and also annotations would be updated in cloud storage. I have rclone running on the kobo but don't see a simple way to cause the synced folders to appear in my library. Anyone working on this?

The problem with bidirectional syncing can be that if something gets messed up on the cloud storage, it gets synced back to your Reader to also be messed up. Unless you know for sure that there is nothing wrong on the cloud srorage, I would not want bidirectional syncing.

[ajkessel]

Originally Posted by JSWolf

The problem with bidirectional syncing can be that if something gets messed up on the cloud storage, it gets synced back to your Reader to also be messed up. Unless you know for sure that there is nothing wrong on the cloud srorage, I would not want bidirectional syncing.

I have a pretty simple queue system for my synced cloud folder, and I would really like to be able to remove items on my Kobo once read and not have them re-appear at the next sync. Bidirectional syncing seems like the only way to accomplish this. I would keep a separate backup snapshot of the cloud folder that could be restored in the event things go awry.

[JSWolf]

Originally Posted by ajkessel

I have a pretty simple queue system for my synced cloud folder, and I would really like to be able to remove items on my Kobo once read and not have them re-appear at the next sync. Bidirectional syncing seems like the only way to accomplish this. I would keep a separate backup snapshot of the cloud folder that could be restored in the event things go awry.

You read the eBook, remove it, and because of the bidirectional sync, it's also gone from the cloud. You have to have one-way syncing that you can control the direction.

[ajkessel]

Originally Posted by JSWolf

You read the eBook, remove it, and because of the bidirectional sync, it's also gone from the cloud. You have to have one-way syncing that you can control the direction.

Right, this is exactly what I want. I don't need the cloud to be an archive -- I just want it to be a queue that disappears as items are read (essentially like Pocket).

The other way to do this would be to somehow cause Kobocloud (or rclone) not to download something again if it had already been downloaded but then removed from the device.

I don't think any of the existing solutions allow for either of these approaches.

[NiLuJe]

Just write a minimal shell script that uses rclone/rsync for the bidir sync, and NickelDBus/NickelMenu for the library refresh?

[ajkessel]

Originally Posted by NiLuJe

Just write a minimal shell script that uses rclone/rsync for the bidir sync, and NickelDBus/NickelMenu for the library refresh?

You're right, that works. I just wasn't putting the synced books in the right place. The bidir sync seems reliable enough for my purposes, and I created a separate Google Drive account just for this purpose so any snafu won't infect any other files.

[JSWolf]

Originally Posted by ajkessel

You're right, that works. I just wasn't putting the synced books in the right place. The bidir sync seems reliable enough for my purposes, and I created a separate Google Drive account just for this purpose so any snafu won't infect any other files.

Don't use Google Drive. It can corrupt things.

Use something else such as Dropbox of OneDrive.

[PeterT]

Originally Posted by JSWolf

Don't use Google Drive. It can corrupt things.



Use something else such as Dropbox of OneDrive.

Jon; google drive is great as a file storage / retrieval tool; just not a good location for a live calibre library (but nothing wrong as a one way clone of a calibre library).

Sent from my Pixel 4a using Tapatalk

[JSWolf]

Originally Posted by PeterT

Jon; google drive is great as a file storage / retrieval tool; just not a good location for a live calibre library (but nothing wrong as a one way clone of a calibre library).

Sent from my Pixel 4a using Tapatalk

Google Drive is not good as a one way clone of your library. If you need to download it, it will be corrupt. The same reason you cannot put a live library is the same reason you cannot put a copy.

[qkqw]

After KoboCloud had disappointed me a few times, mostly resulting in me loosing my library (probably due to the bind mount trick), I've decided to give rclone a go. Here's a few pointers if anybody wants to follow along:

• Get the ARMv7 - 32 Bit variant here: https://rclone.org/downloads/
• Get the Mozilla CA certificate store from the Curl website: https://curl.se/docs/caextract.html
• Put everything into /mnt/onboard/.adds/rclone
• Install NickelDBus in order to be able to refresh your library automatically: https://shermp.github.io/NickelDBus/#installation
• Create a `rclone.conf`either manually, on your computer or with rcloud config on your Kobo
• Additionally, add the following lines to your config:

[local]
encoding = Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot

[yourRemoteLibrary]
[...]

Kobo uses a VFAT filesystem, which does not like most of the special characters, hence we need to tell rclone to convert them. rclone isn't too smart about this unfortunately, thinking this is a normal Linux setup.

Next, create a script named sync.sh in the same folder:

#!/bin/sh

RCLONE="/mnt/onboard/.adds/rclone"
REMOTE="yourRemoteLibrary:"
LIBRARY="/mnt/onboard/library"

echo "waiting for internet connection"
r=1;i=0
while [ $r != 0 ]; do
  if [ $i -gt 60 ]; then
    echo "error! no network available"...
    exit 1
  fi
  ping -c 1 -w 3 aws.amazon.com >/dev/null 2>&1
  r=$?
  if [ $r != 0 ]; then sleep 1; fi
  i=$(($i + 1))
done

$RCLONE/rclone \
  --workdir $RCLONE/.cache \
  --ca-cert $RCLONE/cacert.pem \
  --log-file "$RCLONE/rclone.log" \
  --modify-window 3s --ignore-checksum --no-update-modtime \
  --verbose \
  bisync $REMOTE $LIBRARY &&
  qndb -m pfmRescanBooks

A few notes:

• I've set the workdir to the same folder, in order to not pollute anything else
• There's a log file called rclone.log which you can use when things go wrong
• I've had some issues with synchronization and file modification times, as VFAT only seems to support a granularity of up to 2 seconds, hence the modify-window
• Ignoring the checksum is probably unsafer (only relying on mod-time and file size), but way faster
• I've decided to not update the file modification times locally, as this led to some interesting problems when rebooting the kobo and having to deal with different timezones

Attention! Before running the script the first time, you need to append a `--resync` flag in order to get bisync working. See the rclone docs: https://rclone.org/bisync/

In case you want to run this script automatically, you can just add a udev rule (e.g. /etc/udev/rules.d/97-rcloud.rules) like this:

KERNEL=="eth*", ACTION=="add", RUN+="/mnt/onboard/.adds/rclone/sync.sh"
KERNEL=="wlan*", ACTION=="add", RUN+="/mnt/onboard/.adds/rclone/sync.sh"

Thanks @ajkessel who helped me to get started with this.

Ps. You can also just use "rclone sync", if you don't need bidirectional syncing.

[NiLuJe]

Originally Posted by qkqw

I've decided to not update the file modification times locally, as this led to some interesting problems when rebooting the kobo and having to deal with different timezones

Yeah, FAT32 is... not really easy on bidirectional sync frameworks. Besides the timestamp granularity issue you mentioned, it also happens to store timestamps in... local time >_<".
So, yeah, UTC vs. local vs. DST; plus weird Linux quirks regarding timezones at boot make this a power keg for stupid timestamp discrepancies .

[qkqw]

Originally Posted by NiLuJe

Yeah, FAT32 is... not really easy on bidirectional sync frameworks. Besides the timestamp granularity issue you mentioned, it also happens to store timestamps in... local time >_<".

For now everything seems to work nicely, but just in case, there's a --localtime flag for the bisync command which does exactly that: https://rclone.org/bisync/

[jmacindoe]

Thanks for the guide @qkqw! A couple of small things I'd add

1. On my Clara 2E the network device is mlan0 so I needed this rule

KERNEL=="mlan*", ACTION=="add", RUN+="..."

2. For some reason the command didn't run unless I wrapped it with /bin/sh and made it run async

KERNEL=="eth*", ACTION=="add", RUN+="/bin/sh -c '/bin/pull &'"

3. I use "rclone copy --ignore-existing" which will ignore both checksum and modification time, only checking if the file exists. Since I'm mostly syncing ebooks that are never modified, there's no need for any modification checks at all, and as you say the modification time is super buggy. But this won't work if you want to bisync .sdr's of course, since they change.

4. I like to output the log to the screen so I can check it worked. So

rclone copy --args-etc 2>&1 | /usr/local/kfmon/bin/fbink

Logging the errors 2>&1 was really useful for debugging.

5. For some reason rclone used a different config file location when running from SSH and koreader vs when running from udev. Can be fixed by specifying with the --config flag.

[qkqw]

Originally Posted by jmacindoe

1. On my Clara 2E the network device is mlan0 so I needed this rule

KERNEL=="mlan*", ACTION=="add", RUN+="..."

Interesting. Most examples list a eth* device, never understood why the Kobo would have one.

Originally Posted by jmacindoe

2. For some reason the command didn't run unless I wrapped it with /bin/sh and made it run async

KERNEL=="eth*", ACTION=="add", RUN+="/bin/sh -c '/bin/pull &'"

Might be because it needs an executable bit? The root system is an ext filesystem.

Originally Posted by jmacindoe

4. I like to output the log to the screen so I can check it worked. So

rclone copy --args-etc 2>&1 | /usr/local/kfmon/bin/fbink

Logging the errors 2>&1 was really useful for debugging.

You could also use qndb for that as it can trigger popups and dialogs.

Originally Posted by jmacindoe

5. For some reason rclone used a different config file location when running from SSH and koreader vs when running from udev. Can be fixed by specifying with the --config flag.

Should be enough to put the config in the same directory as rclone?