[WIP] kobo-rclone - Kobo wrapper for Rclone

[sherman]

Hi folks

Inspired by the recent gdrive thread (https://www.mobileread.com/forums/sh...d.php?t=309337), I decided to try and implement similar functionality for different 'cloud' or sync solutions.

That led me to the program Rclone, which is a CLI tool to sync files from a wide variety of different providers, including cloud providers, and more traditional hosting methods such as FTP, SFTP, HTTP, WebDAV etc.

Not only did I want to sync ebook files however, I also wanted to populate
series information without having to plug in my Kobo.

I'm at the point where I've got something "working", but I probably now need help to test it, and provide suggestions/contributions as to how to make the program safer, especially the metadata aspect.

Also, this is my very first Go project (I knew nothing about Go until a few days ago!), so I welcome any feedback from more experienced Go developers.


So, the project is called kobo-rclone, and it is available at https://github.com/shermp/kobo-rclone. Setup and install instructions are available in the readme. I am currently not going to distribute any binaries until I have confidence that there is a low possibility of corrupting the DB or partition.

This project is still very much a Work In Progress, so use it at your own risk etc.

[NiLuJe]

I *like* the sneak-attack approach of remounting behind Nickel's back! .

I don't know if there's a way to bypass having top tap the "Connect" button, but if there's isn't, a couple remarks:

We tend to ID the devices the same way as the FW update check, instead of the S/N (c.f., https://geek1011.github.io/KoboStuff/ & https://github.com/NiLuJe/FBInk/blob...id.c#L211-L263).

It might be doable to compute the coordinates of where you need to tap in a more generic manner (since we know the resolution of everything, barring the few sneak pixels behind bezels on the H2O & Aura), and then fake the tap programatically (c.f., https://github.com/koreader/koreader...pull/468/files). Take that with a grain of salt, the exact touch 'protocol' has a long history of being weird & messily not quite portable between Kobo models...

[frostschutz]

The "Connect" button can appear in two different places even on the same device (Beta Features, Large Print Mode) and also depends on FW version otherwise.

Best approach might still be to record the user's tap and replay it in the future (& check if the replay worked, and go back to record otherwise).

And then you still have to know the timing the dialog takes to appear... sometimes it appears late for some reason and you sent your tap too early.

It's complicated. Sometimes you just need that human helper :-) (or find a completely different way to coerce nickel into update mode)

[sherman]

Thanks for the feedback folks!

I'm still an amateur at all this Kobo hacking, so it's great to hear from more experienced developers.

I was running into an issue where I couldn't open the DB, because it was locked. I REALLY want metadata updating, so I was trying to think of ways to do so. Then I had the idea of "will Nickel freak out if I remount the filesystem while in USB connect mode?". Turns out, Nickel doesn't seem to mind.

I knew it was possible to simulate touch presses, but I didn't know how, so the current way is a quick and dirty method. I am aware that it needs some work.

[NiLuJe]

@sherman: Sometimes, you gotta get your hands dirty . (I'd trust @frostschutz on that one, he has much more experience playing with that than me, I was just throwing wild 3AM ideas around ^^).

And I definitely get the impulse about metadata, I had the same one when I started working on KFMon, until I started actually looking into it, and I realized that: wait, that's not how SQLite works .
Hence the insane bit of db_update code that's still in there, disabled by default, but sitting pretty, waiting to trap unsuspecting users into trashing their db (... or hitting a write lock timeout, most likely ).

At least the switch to WAL got rid of the *read* locks, because *that* was fun...

[sherman]

Just musing here...

I did ran strace on nickel while going through the connect/disconnect process, because I wanted to see what Nickel was doing during the time it was "connected". Not a lot, it turns out, mostly checking the battery status.

I noticed on the strace that nickel requests a full eink refresh after drawing the 'computer detected'. Maybe one could monitor that as a starting point as to when to start simulating touches...

[frostschutz]

Originally Posted by sherman

Then I had the idea of "will Nickel freak out if I remount the filesystem while in USB connect mode?". Turns out, Nickel doesn't seem to mind.

It's probably safe to do if the USB connection is not real but you tricked nickel into thinking there was one.

In a real USB connection, if you mount the filesytem, while the PC mounts it too, you'd end up with filesystem corruption.

So what I do instead (in my AutoShelf mod) is freeze nickel the moment the USB is unplugged, modify the database, unfreeze nickel. (by sending STOP / CONT kill signals)

This is dangerous too because there's a watchdog that likes to reboot the entire device when nickel stops responding, and the timing for the freeze also has to be just right (that part is left to luck)

[sherman]

Originally Posted by frostschutz

It's probably safe to do if the USB connection is not real but you tricked nickel into thinking there was one.

In a real USB connection, if you mount the filesytem, while the PC mounts it too, you'd end up with filesystem corruption.

So what I do instead (in my AutoShelf mod) is freeze nickel the moment the USB is unplugged, modify the database, unfreeze nickel. (by sending STOP / CONT kill signals)

This is dangerous too because there's a watchdog that likes to reboot the entire device when nickel stops responding, and the timing for the freeze also has to be just right (that part is left to luck)

The end goal is to launch kobo-rsync using kfmon. In that situation, it seems highly unlikely that a user would be in a position to have the filesystem mounted by the computer, as the user would not be able to launch the program from kfmon while connected to a PC.

Although, I'm not quite sure what happens if you plug it in while already "connected"... Maybe I should test that (if I'm brave enough!)

I assume users who have compiled it and are running via SSH/telnet are smart enough to avoid trying to mount the filesystem twice.
*****
Another bit of musing, could we use a bit of OCR to find the "connect" button location? Performance? Battery? I'll have a look into it.

[sherman]

Originally Posted by sherman

Another bit of musing, could we use a bit of OCR to find the "connect" button location? Performance? Battery? I'll have a look into it.

Replying to myself here, that probably wouldn't work, because languages etc. And also overkill...

[frostschutz]

I'm using "OCR" in my screensaver mod. Well not really, it just looks for a specific pixel pattern in a single line of pixels.

Described here: https://www.mobileread.com/forums/sh...35#post3165835

Didn't always work for everyone, also because of other reasons (has to be done as device goes to sleep, not much time, slow implementation, then the RGBA change etc., and just about every firmware update changed the font rendering)

[NiLuJe]

In the case of the Connect popup, there's an image, so it might be more stable than text to scan, but I *think* said image was updated recently... .

[sherman]

Originally Posted by NiLuJe

In the case of the Connect popup, there's an image, so it might be more stable than text to scan, but I *think* said image was updated recently... .

I've had an idea of searching for the buttons themselves, by their colour. The colour is a specific shade of grey that doesn't appear to be used elsewhere in any great quantity.

If one finds a certain number of grey pixels in a line (where the number is some
minimum percentage of the screen width), and then another line the same or similar length, it is highly likely that one has found the two buttons.

[NiLuJe]

@sherman: Yeah, that's a much more reasonable idea than the image (because, err, resolution swaps and all that).

That'd take an extremely unlucky pattern of homescreen tiles to confuse it .

[sherman]

Originally Posted by NiLuJe

@sherman: Yeah, that's a much more reasonable idea than the image (because, err, resolution swaps and all that).

That'd take an extremely unlucky pattern of homescreen tiles to confuse it .

How hard would it be to add a "get pixel colour/info" function to fbink?

[NiLuJe]

@sherman: There's already one (needed for alpha blending to handle transparency in images) .

So, err, I may have played a bit with this...

Oops. .

I haven't checked w/ large print enabled, and I unfortunately have the exact same device as you (and @frostschutz), so that doesn't help validating the approach much, but, hey .

EDIT: Handles large print, too .
EDIT²: Break early on match.
EDIT³: Rotation shenanigans.
EDIT⁴: More rotation shenanigans.
EDIT⁵: Even more rotation shenanigans.
EDIT⁶: Shenanigans relating to rotation .
EDIT⁷: Button pusher...
EDIT⁸: Tweaked thresholds a bit...
EDIT⁹: Bull's eye!