Prevent Kobo from importing dotfiles/folders

[NiLuJe]

@rtiangha: 2 backslashes.

That commit was simply meant as a reference to me messing up the regex twice today ^^. And to illustrate the fact that nickel could happily purge previously indexed content if you mess it up badly enough .

The content of my earlier post is correct, and is what most users should care about, not me losing my mind in commit logs .

EDIT: The live version of this block of code has been fixed to use less confusing escaping shenanigans .

[geek1011]

Originally Posted by NiLuJe

@rtiangha: 2 backslashes.

That commit was simply meant as a reference to me messing up the regex twice today ^^. And to illustrate the fact that nickel could happily purge previously indexed content if you mess it up badly enough .

The content of my earlier post is correct, and is what most users should care about, not me losing my mind in commit logs .

EDIT: The live version of this block of code has been fixed to use less confusing escaping shenanigans .

There's one trick I sometimes do when multiple layers of escaping are needed. If you replace \. with [.], it essentially does the same thing, but without backslashes.

[NiLuJe]

That works when you happen to not need to escape square brackets, yeah .

[dmapr]

Originally Posted by frostschutz

they ignore .add (which nobod uses?) but not .adds or .addons

but they also ignore (not hidden) xadd, madd, ladd, ...

regexp 101, I guess

Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.

Source

[frostschutz]

Originally Posted by NiLuJe

It does lead to the subsequent deletion of the content (at least if it happened to be in the db previously). So that'd potentially work for *new* content only .

...

So we're deleting user files without asking now?

What were they trying to fix and what's the point of it all... it's a mystery to me.

[davidfor]

Originally Posted by frostschutz

...

So we're deleting user files without asking now?

What were they trying to fix and what's the point of it all... it's a mystery to me.

No, they are deleting details from the database for files that the import process can no longer find. Which is basically what has always been done. But, there is a new reason for not finding the file. Before, the file had to be deleted, renamed or moved. Now, we also have this filter changed to not be able to search where it is.

[NiLuJe]

Yup, it basically trips the "oh no, file is gone" check, and it turns out that, besides purging the database, it also purges the actual file (which I assume is done as part of the process of deleting the associated sidecar/thumbnails), something which we had no really easy way to know/trigger before, as @davidfor just detailed the most common ways to trip that check, and they all involved the file being gone in the first place .

TL;DR: One could argue that borking the regex *was* in fact "asking for it" .

[frostschutz]

Okay, I guess it makes sense to delete files (generated thumbnails and the like) in addition to database purging. It's just weird if they delete the non-generated stuff too... oh well.

[baskerville]

Originally Posted by NiLuJe

In case (not) ignoring kobo & adobe happens to be meaningless, \\..*? is enough.

Why would .kobo and .adobe need to be included?

Have you found the default value for ExcludeSyncFolders?

I'd rather ignore hidden directories/files at any depth:

[FeatureSettings]
ExcludeSyncFolders=([^.][^/]*/)*\\..+

[geek1011]

Originally Posted by baskerville

Why would .kobo and .adobe need to be included?

Have you found the default value for ExcludeSyncFolders?

I'd rather ignore hidden directories/files at any depth:

([^.][^/]+/)*\\..+

NiLuJe was using a negative lookahead for Kobo and Adobe. Regarding the default value(s), have a look at the attached files here (at 0x5771E2): https://www.mobileread.com/forums/sh....php?p=3892386 .

[baskerville]

Originally Posted by geek1011

NiLuJe was using a negative lookahead for Kobo and Adobe.

I understood that. .kobo doesn't match the regexp. But why would we want that?

Originally Posted by geek1011

Regarding the default value(s), have a look at the attached files here (at 0x5771E2): https://www.mobileread.com/forums/sh....php?p=3892386.

Thanks, that's impressive! So the default regexp looks like this:

(.kobo|koboExtStorage)/(articles|images|kepub|screensaver|slideshow|dict|certificates|webcache|webstorage)|.fseventsd|.Spotlight-V100|.Trash|.Trashes|.Trash-\\d+|\\$RECYCLE.BIN

?

[frostschutz]

Originally Posted by NiLuJe

The thing to remember is that this is sandwiched inside a

^/mnt/(sd|onboard)/%1$

so, can't use ^ or $, and that QSettings probably wants some stuff escaped, hence \\. for \.

Sep 21 17:08:42 nickel: (    22.221 @ 0x12dd9c0 / ui.debug) void SyncFileSystemCommand::findFilesToParse(const QString&, QStringList&) skipping "/mnt/onboard/.adobe-digital-editions" , matches "\x00|/\.|\x00"

this breaks out of the sandwich (making sure it doesn't match by requiring binary \0 character) and then should go on to ignore hidden folders at any depth

[NiLuJe]

@frostschutz: You'll have to walk me through that one... o_O .

EDIT: Thanks! Ooh, that's sneaky xD.

EDIT²: Thankfully, as we found out with the preload hack, blacklisting a toplevel hidden folder is enough to make it entirely non-traversable. But this could still be useful for more esoteric directory structures ...

@baskerville: Yeah, the idea was to let the default patterns do their stuff for adobe/kobo folders. Again, not 100% convinced it makes any difference in practice (at least for my use cases), in which case, (EDIT: a top-level only variant of) what you proposed (i.e., \..*? [-> \\..*? in the config] once you account for the full pattern) works just fine.

[baskerville]

Originally Posted by NiLuJe

Again, not 100% convinced it makes any difference in practice (at least for my use cases), in which case, what you proposed (i.e., \..*? [-> \\..*? in the config] once you account for the full pattern) works just fine.

Thanks. Please note that this isn't the regexp I've suggested: I also want to ignore the hidden directories inside the visible directories.

[NiLuJe]

@baskerville: Yep, I should have made that clearer. Would your example work, given the "sandwich", or would that need some trickery like @frostschutz did?

(I'm not completely terrible at regexps, but I'm no wizard either, so, err, small words, please ;p).

EDIT: If I put my brain in "regexp mode", I think it looks fine for that purpose, but I haven't tested it . (i.e., accepts zero or more "classic" directories in front of an hidden one).

EDIT²: The first group might fail to match for a single-character folder name, though, right?