Prevent Kobo from importing dotfiles/folders

geek1011 · 09-18-2019, 04:41 PM

I've made a small hack to prevent Kobo from importing dotfiles/folders. It uses LD_PRELOAD to override the directory syscalls and skip over dotfiles other than .kobo* and .adobe-digital-editions. It only applies to the nickel process. This hack was made to solve the issue mentioned in the firmware thread.

This will survive through firmware upgrades. To remove it, you'll need to wait until I make a script for that, or you can delete /etc/ld.so.preload and /usr/local/geek1011/libhidedir_kobo.so (you can also factory reset). I've done some preliminary tests, and everything seems to work well.

See here for the code and more details: https://github.com/geek1011/kobo-mod...o-dotfile-hack .

Update for 4.17.13694+: There's now a official config option which can do basically the same thing. If you want, you can still use this hack (it will still work). See near the end of this thread for how to use the config option. The disadvantage of the config option is that it won't persist after a sign out and that it will delete stuff if you don't get it right the first time, but otherwise, after the initial setup, they're pretty much identical.

geek1011 · 09-18-2019, 04:42 PM

@NiLuJe, can you do a quick review of this?

NiLuJe · 09-18-2019, 09:57 PM

That's a rather large hammer to hit an annoying nail with, but I haven't got any better idea ^^.

That looks sound at first glance, but I'll take a closer look at it tomorrow

.

----

EDIT: Huh. I was wondering about the whole fd -> path mapping, turns out getting the path from a (dir) fd is a fun game...
Short of checking what /proc/self/fd/%d points to, nothing really comes to mind... Which is somewhat iffy, so, eh, building a cache from opendir looks like a solid approach

.

geek1011 · 09-18-2019, 10:12 PM

Quote:

Originally Posted by NiLuJe

That's a rather large hammer to hit an annoying nail with, but I haven't got any better idea ^^.

If you mean that literally, you can reduce the code to about 1/3rd of the size if you remove the config options and support for running on a non-kobo. If you mean the figuratively, I did look into using a patch (see the readme), using bind-mount trickery (too fragile, racy), using live syscall hooks (racy), shimming libnickel itself (would need to be linked with Qt, could break on updates, could miss things and cause unusual bugs), namespaces (not supported by the kernel version, would also require changing the boot script), a kernel module like GoboHide (it would be to dependent on the firmware), and this seemed to be the best option.

The only compatibility issues I see with this approach is if Kobo ever decides to use fdopendir (then we won't be able to see the full path to the dirents, only the names), and if Kobo needs to list another dotfile/dir under /mnt in a future update (but I think this is unlikely) (the dir won't appear, but kobo will still be able to list and access the files under it).

The advantages of this approach are that it persists (no chance of an update scanning those files by accident), it doesn't touch any pre-existing files (an update won't override it, it won't break anything else), it can limit itself to nickel only, it has complete control over what it shows/hides, it doesn't need to be updated with nickel, and it won't break anything if it a part of it is deleted (the config file or the library). The only time it'll break something enough to require a manual factory reset (it'll never require a sd reimage, unlike a kernel mod) is if there is a serious bug in the directory filtering code. This also turned out to be less brittle than I originally expected it to be, as I could limit it to nickel, it didn't need to pre-process all the dirents every time a dir was opened, and it didn't need to touch any existing files.

Quote:

That looks sound at first glance, but I'll take a closer look at it tomorrow

.

Yep. Specifically, I'm not 100% certain I freed all allocated memory in all possible cases (specifically errors from the original functions), and I want a second pair of eyes to look at the dir matching and process checking logic (and also see if it covers everything and can't be made simpler). As for the dlsym and wrapper stuff, I'm pretty sure I did it right, although the readdir_r one was a bit unusual.

Also, if you have any ideas for the uninstall script, that'd be helpful (I already know about the self-deleting udev loop hook one, but I'm wondering if there's anything simpler).

P.S. I'm probably going to convert the gist into a full repo so I can do automatic builds and code comments on it.
P.P.S. FYI, the relevant libnickel symbol is SyncFileSystemCommand::findFilesToParse(QString const&, QStringList&)@0x575308, and the relevant Qt4 class is QDir.

Update: I've moved the code from the gist to a repo: https://github.com/geek1011/kobo-mod...o-dotfile-hack.

frostschutz · 09-19-2019, 02:59 AM

I tested it on my H2O and it works, so thanks for that.

Not a huge fan of LD_PRELOAD (avoided using it in any of my mods so far). Much less a system-wide one. Ugh.

Kobo, what the heck. Why you cracking down on the modder community now. This change is so nonsensical. Hope they will reconsider...

geek1011 · 09-19-2019, 07:36 AM

Quote:

Originally Posted by frostschutz

I tested it on my H2O and it works, so thanks for that.

No problem!

Quote:

Not a huge fan of LD_PRELOAD (avoided using it in any of my mods so far). Much less a system-wide one. Ugh.

I wasn't either, but it ended up being the cleanest option of what I tried. But, that's why I added the check to ensure it's running in nickel.

Also, I just had an idea. Do you think it might be better if instead of checking /proc/self/exe, I checked if libnickel was loaded in the current process (the only disadvantage I see is a potential race condition, but the advantage is is will certainly run on anything which cares about book locations, and nothing else)?

Quote:

Kobo, what the heck. Why you cracking down on the modder community now. This change is so nonsensical. Hope they will reconsider...

I doubt Kobo is cracking down on the mods, but maybe "Kobo recommends Windows" (like the computer manufacturers)?

It breaks Trash on Linux and macOS.

P.S. I've also decided to split the Makefile into a.Kobo and native one, as the current Makefile looks horrible.

geek1011 · 09-19-2019, 07:53 AM

Quote:

Originally Posted by NiLuJe

EDIT: Huh. I was wondering about the whole fd -> path mapping, turns out getting the path from a (dir) fd is a fun game...
Short of checking what /proc/self/fd/%d points to, nothing really comes to mind... Which is somewhat iffy, so, eh, building a cache from opendir looks like a solid approach

.

Yes, but unfortunately, that's where the hackiest part IMO, and the largest performance impact comes from (from the strdup allocation, in microseconds, but that's multiplied by every directory opened, so I'd estimate around a millisecond for a few tens of thousand dirs). It's also a source of potential segfaults if I make a mistake. Also, I'm not too familiar with how arrays are optimized, but at worst, that array takes a whole ~200kb of memory when empty (4b per pointer). I wish there was a native hashmap in C (I don't want to double the code to write an efficient one).

frostschutz · 09-19-2019, 07:13 PM

So, this is ridiculous.

Kobo imports book from hidden .folder even though I don't want it to.

Kobo does NOT import book from NOT hidden folder. Thank you for nothing.

In libnickel there's some hardcoded stuff:

Code:

.Trash
.Trashes
.Trash-\d+

So the last one ... wait, what? Is that a regexp? Isn't . "any character" then?

So yup... it imports books from ".NotTrash/" but ignores books from "xTrash-123" (x instead . so not hidden in any way).

The whole notion of maintaining a list of folder names that should actually be treated like hidden folders is silly.

Old system was so much clearer and simpler. Starts with dot? Ignore that.

frostschutz · 09-19-2019, 07:24 PM

they ignore .add (which nobod uses?) but not .adds or .addons

but they also ignore (not hidden) xadd, madd, ladd, ...

regexp 101, I guess

NiLuJe · 09-19-2019, 07:26 PM

TL;DR:

Code:

ExcludeSyncFolders=\\.(?!kobo|adobe).*?

(In the FeatureSettings section)

The thing to remember is that this is sandwiched inside a

Code:

^/mnt/(sd|onboard)/%1$

so, can't use ^ or $, and that QSettings probably wants some stuff escaped, hence \\. for \. (i.e., match a dot, instead of any character, to avoid what @frostschutz just described with the default patterns

).

@frostschutz: I think some DEs do use numbered trashes, for some reason, so \.Trash-\d+ would make sense

. Or just all-in-one: \.Trash.*?

EDIT: Except that both backslash would need to be escaped here

.

NiLuJe · 09-19-2019, 07:32 PM

That yields a nice:

Code:

Sep 20 01:25:29 nickel: (  2223.266 @ 0x16d0b30 / ui.debug) void SyncFileSystemCommand::findFilesToParse(const QString&, QStringList&) skipping "/mnt/onboard/.adds" , matches "\.(?!kobo|adobe).*?" 
Sep 20 01:25:29 nickel: (  2223.280 @ 0x16d0b30 / ui.debug) void SyncFileSystemCommand::findFilesToParse(const QString&, QStringList&) skipping "/mnt/onboard/.foo" , matches "\.(?!kobo|adobe).*?" 
Sep 20 01:25:29 nickel: (  2223.282 @ 0x16d0b30 / ui.debug) void SyncFileSystemCommand::findFilesToParse(const QString&, QStringList&) skipping "/mnt/onboard/.fseventsd" , matches "\.(?!kobo|adobe).*?" 
Sep 20 01:25:29 nickel: (  2223.296 @ 0x16d0b30 / ui.debug) void SyncFileSystemCommand::findFilesToParse(const QString&, QStringList&) skipping "/mnt/onboard/.kobo/certificates" , matches "(.kobo|koboExtStorage)/certificates" 
Sep 20 01:25:29 nickel: (  2223.297 @ 0x16d0b30 / ui.debug) void SyncFileSystemCommand::findFilesToParse(const QString&, QStringList&) skipping "/mnt/onboard/.kobo/dict" , matches "(.kobo|koboExtStorage)/dict" 
Sep 20 01:25:29 nickel: (  2223.306 @ 0x16d0b30 / ui.debug) void SyncFileSystemCommand::findFilesToParse(const QString&, QStringList&) skipping "/mnt/onboard/.kobo/kepub" , matches "(.kobo|koboExtStorage)/kepub" 
Sep 20 01:25:29 nickel: (  2223.317 @ 0x16d0b30 / ui.debug) void SyncFileSystemCommand::findFilesToParse(const QString&, QStringList&) skipping "/mnt/onboard/.niluje" , matches "\.(?!kobo|adobe).*?" 
Sep 20 01:25:29 nickel: (  2223.320 @ 0x16d0b30 / ui.debug) void SyncFileSystemCommand::findFilesToParse(const QString&, QStringList&) skipping "/mnt/onboard/.Spotlight-V100" , matches "\.(?!kobo|adobe).*?" 
Sep 20 01:25:29 nickel: (  2223.322 @ 0x16d0b30 / ui.debug) void SyncFileSystemCommand::findFilesToParse(const QString&, QStringList&) skipping "/mnt/onboard/.Trashes" , matches "\.(?!kobo|adobe).*?"

In case (not) ignoring kobo & ade happens to be meaningless, \\..*? is enough:

Code:

Sep 20 01:12:26 nickel: (  1440.373 @ 0x3585888 / ui.debug) void SyncFileSystemCommand::findFilesToParse(const QString&, QStringList&) skipping "/mnt/onboard/.adds" , matches "\..*?" 
Sep 20 01:12:26 nickel: (  1440.375 @ 0x3585888 / ui.debug) void SyncFileSystemCommand::findFilesToParse(const QString&, QStringList&) skipping "/mnt/onboard/.adobe-digital-editions" , matches "\..*?" 
Sep 20 01:12:26 nickel: (  1440.380 @ 0x3585888 / ui.debug) void SyncFileSystemCommand::findFilesToParse(const QString&, QStringList&) skipping "/mnt/onboard/.foo" , matches "\..*?" 
Sep 20 01:12:26 nickel: (  1440.382 @ 0x3585888 / ui.debug) void SyncFileSystemCommand::findFilesToParse(const QString&, QStringList&) skipping "/mnt/onboard/.fseventsd" , matches "\..*?" 
Sep 20 01:12:26 nickel: (  1440.383 @ 0x3585888 / ui.debug) void SyncFileSystemCommand::findFilesToParse(const QString&, QStringList&) skipping "/mnt/onboard/.kobo" , matches "\..*?" 
Sep 20 01:12:26 nickel: (  1440.385 @ 0x3585888 / ui.debug) void SyncFileSystemCommand::findFilesToParse(const QString&, QStringList&) skipping "/mnt/onboard/.niluje" , matches "\..*?" 
Sep 20 01:12:26 nickel: (  1440.387 @ 0x3585888 / ui.debug) void SyncFileSystemCommand::findFilesToParse(const QString&, QStringList&) skipping "/mnt/onboard/.Spotlight-V100" , matches "\..*?" 
Sep 20 01:12:26 nickel: (  1440.389 @ 0x3585888 / ui.debug) void SyncFileSystemCommand::findFilesToParse(const QString&, QStringList&) skipping "/mnt/onboard/.Trashes" , matches "\..*?"

ProTip: DO NOT use a simple ".*" or variations thereof, that'll of course ignore ALL THE THINGS

. (Which will then in turn proceed to let nickel DELETE ALL THE THINGS ;p).
I suddenly have a lot more free space on my H2O

.

frostschutz · 09-19-2019, 07:39 PM

Quote:

Originally Posted by NiLuJe

ProTip: DO NOT use a simple ".*" or variations thereof, that'll of course ignore /mnt/onboard/

.

That's a nice feature actually. That way you can use nickel for the webbrowser or whatever and not even have to hide the books for koreader, plato, and the like

NiLuJe · 09-19-2019, 07:46 PM

Quote:

Originally Posted by frostschutz

That's a nice feature actually. That way you can use nickel for the webbrowser or whatever and not even have to hide the books for koreader, plato, and the like

It does lead to the subsequent deletion of the content (at least if it happened to be in the db previously). So that'd potentially work for *new* content only

.

NiLuJe · 09-19-2019, 09:26 PM

Case in point: https://github.com/NiLuJe/kfmon/comm...25df9a553cf857

.

(The good news is, tomorrow's OCP packages will handle this automagically. Just need to check the Windows script on an actual Windows and not just PowerShell on Linux...).

rtiangha · 09-19-2019, 10:02 PM

Quote:

Originally Posted by NiLuJe

Case in point: https://github.com/NiLuJe/kfmon/comm...25df9a553cf857

.

(The good news is, tomorrow's OCP packages will handle this automagically. Just need to check the Windows script on an actual Windows and not just PowerShell on Linux...).

So based on this commit, in your TL;DR: post above, if we're editing the file on our own, should we be inputting 2 slashes or 3? I'm a little confused.

09-18-2019, 09:57 PM	#3
NiLuJe BLAM! Posts: 13,477 Karma: 26012494 Join Date: Jun 2010 Location: Paris, France Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E	That's a rather large hammer to hit an annoying nail with, but I haven't got any better idea ^^. That looks sound at first glance, but I'll take a closer look at it tomorrow . ---- EDIT: Huh. I was wondering about the whole fd -> path mapping, turns out getting the path from a (dir) fd is a fun game... Short of checking what /proc/self/fd/%d points to, nothing really comes to mind... Which is somewhat iffy, so, eh, building a cache from opendir looks like a solid approach . Last edited by NiLuJe; 09-18-2019 at 10:17 PM.

09-19-2019, 07:13 PM	#8
frostschutz Linux User Posts: 2,279 Karma: 6123806 Join Date: Sep 2010 Location: Heidelberg, Germany Device: none	So, this is ridiculous. Kobo imports book from hidden .folder even though I don't want it to. Kobo does NOT import book from NOT hidden folder. Thank you for nothing. In libnickel there's some hardcoded stuff: Code: .Trash .Trashes .Trash-\d+ So the last one ... wait, what? Is that a regexp? Isn't . "any character" then? So yup... it imports books from ".NotTrash/" but ignores books from "xTrash-123" (x instead . so not hidden in any way). The whole notion of maintaining a list of folder names that should actually be treated like hidden folders is silly. Old system was so much clearer and simpler. Starts with dot? Ignore that.

09-19-2019, 07:26 PM	#10
NiLuJe BLAM! Posts: 13,477 Karma: 26012494 Join Date: Jun 2010 Location: Paris, France Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E	TL;DR: Code: ExcludeSyncFolders=\\.(?!kobo\|adobe).? (In the FeatureSettings section) The thing to remember is that this is sandwiched inside a Code: ^/mnt/(sd\|onboard)/%1$ so, can't use ^ or $, and that QSettings probably wants some stuff escaped, hence \\. for \. (i.e., match a dot, instead of any character, to avoid what @frostschutz just described with the default patterns ). @frostschutz: I think some DEs do use numbered trashes, for some reason, so \.Trash-\d+ would make sense . Or just all-in-one: \.Trash.? EDIT: Except that both backslash would need to be escaped here . Last edited by NiLuJe; 09-20-2019 at 12:44 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
How to prevent replacement of cover.jpg in kepubs when sending to Kobo e-reader?	JeanPierre	Kobo Reader	2	01-20-2019 04:57 AM
Kobo should be incapable to prevent the publishing of a stolen epub?	bathop	General Discussions	36	06-04-2016 05:49 PM
Arc Importing to Kobo Arc	Roadkill	Kobo Tablets	2	08-04-2013 07:18 AM
Dragging and Dropping Folders to Kobo	david_e	Kobo Reader	2	10-24-2012 06:20 PM

09-18-2019, 04:42 PM	#2
geek1011 Wizard Posts: 2,735 Karma: 6990705 Join Date: May 2016 Location: Ontario, Canada Device: Kobo Mini, Aura Edition 2 v1, Clara HD	@NiLuJe, can you do a quick review of this?

09-19-2019, 02:59 AM	#5
frostschutz Linux User Posts: 2,279 Karma: 6123806 Join Date: Sep 2010 Location: Heidelberg, Germany Device: none	I tested it on my H2O and it works, so thanks for that. Not a huge fan of LD_PRELOAD (avoided using it in any of my mods so far). Much less a system-wide one. Ugh. Kobo, what the heck. Why you cracking down on the modder community now. This change is so nonsensical. Hope they will reconsider...

09-19-2019, 07:24 PM	#9
frostschutz Linux User Posts: 2,279 Karma: 6123806 Join Date: Sep 2010 Location: Heidelberg, Germany Device: none	they ignore .add (which nobod uses?) but not .adds or .addons but they also ignore (not hidden) xadd, madd, ladd, ... regexp 101, I guess

09-19-2019, 09:26 PM	#14
NiLuJe BLAM! Posts: 13,477 Karma: 26012494 Join Date: Jun 2010 Location: Paris, France Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E	Case in point: https://github.com/NiLuJe/kfmon/comm...25df9a553cf857 . (The good news is, tomorrow's OCP packages will handle this automagically. Just need to check the Windows script on an actual Windows and not just PowerShell on Linux...).

Advert

Advert