Rooting Tab Ultra

[ottischwenk]

I have no problems with devices up to and including Android 10 opens within 2-3 seconds.
Android 11 from the Leaf 2 is still halfway (about 40 seconds) for opening this folder, but Tab Ultra!

[Renate]

@ottischwenk You can do that alpha test too, please.

If listing a directory takes minutes (or even tens of seconds), what does a regular ls -l look like?
Does it pause at the start? Does it spit out a file every second? Does it slow down as it progresses? What's going on?

[rexbanner]

I've started your test and will update it with more results later (unfortunately it's too late for breakfast here), however for the first four thousand files:

   
1001    8002   51898    
    0m02.90s real     0m00.17s user     0m00.41s system
2001   16002  103788
    0m07.02s real     0m00.12s user     0m01.09s system
3001   24002  155678
    0m13.13s real     0m00.32s user     0m01.49s system
4001   32002  207568
    0m19.76s real     0m00.51s user     0m01.72s system

So it seems much slower from the start and it seems to get worse with every additional 1000 files

For me "ls -l" always hangs at the beginning and then outputs everything bascially at once after some time.

[rexbanner]

I've posted a first run of the script but the post did not appear yet as it's being checked apparently.

Since then, I had to cancel and restart the script, and now had it running for up to 8'000

BOOX:/sdcard/Documents/tmp2 $ /data/local/tmp/alpha.sh
   1001    8002   59901
    0m13.54s real     0m00.07s user     0m00.62s system
   2001   16002  119794
    0m28.97s real     0m00.35s user     0m01.11s system
   3001   24002  179687
    0m44.88s real     0m00.34s user     0m01.68s system
   4001   32002  239580
    1m02.23s real     0m00.43s user     0m02.27s system
   5001   40002  299473
    1m20.48s real     0m00.42s user     0m02.92s system
   6001   48002  359366
    1m39.06s real     0m00.61s user     0m02.92s system
   7001   56002  419259
    1m59.83s real     0m00.70s user     0m03.86s system
   8001   64002  479152
    2m21.60s real     0m00.84s user     0m04.26s system

It seems to be roughly linear in that it takes around 15-20 seconds per thousand additional files. Which also is more ore less in line with the almost 6 minutes it took for 20'000 files in one of my last posts.

When I run a standard ls command in a folder, usually it hangs first without displaying anything and then after some time, the complete listing is shown

[ottischwenk]

Originally Posted by Renate

@ottischwenk You can do that alpha test too, please.

If listing a directory takes minutes (or even tens of seconds), what does a regular ls -l look like?
Does it pause at the start? Does it spit out a file every second? Does it slow down as it progresses? What's going on?

On Tab Ultra:
it waits app 47 seconds, an then it runs - the first time!
The second try needed 4 minutes before start
On Leaf 2:
It waits app 7 seconds and then it runs. Second and other try - 15 seconds

[Renate]

This is really weird stuff. Whoever wants to report it to Onyx is welcome to include my script.

This is interesting:

SDCardFS is deprecated on devices that launch with Android 11...

SDCardFS replacement functionality
The replacement for SDCardFS uses several Linux kernel filesystem features to achieve similar results. Case insensitivity is handled directly by the filesystem; this results in lookup times nearly identical in case-sensitive and case-insensitive folders, removing the large folder slowdown from SDCardFS.

See: https://source.android.com/docs/core...rdfs-deprecate
My Poke3 (Android 10) uses sdcardfs and it works fine.
Could somebody do a "mount | grep emulated" command and show what the "emulated" directories show?

[ottischwenk]

Originally Posted by Renate

This is really weird stuff. Whoever wants to report it to Onyx is welcome to include my script.

Funny time:

Microsoft Windows [Version 10.0.22621.963]
(c) Microsoft Corporation. Alle Rechte vorbehalten.

d:\_Reader-Smartphone\ADB>adb shell
BOOX:/ $ cd /storage/emulated/0/Calibre_Companion
BOOX:/storage/emulated/0/Calibre_Companion $ time ls -Rl | wc
  16560  240891 1747723
    0m47.04s real     0m00.52s user     0m00.94s system
BOOX:/storage/emulated/0/Calibre_Companion $ time ls -Rl | wc
  16560  240891 1747723
    4m08.24s real     0m02.06s user     0m08.95s system
BOOX:/storage/emulated/0/Calibre_Companion $ time ls -Rl | wc
  16560  240891 1747723
    4m09.04s real     0m01.88s user     0m09.79s system
BOOX:/storage/emulated/0/Calibre_Companion $
d:\_Reader-Smartphone\ADB>

47 seconds vs more than 4 minutes

[Renate]

Originally Posted by ottischwenk

47 seconds vs more than 4 minutes

Well, clearly you're tiring it out.

Also note that the bulk of the time is neither user or system time.
One can only presume that it's I/O time.

You do realize that the whole basic problem comes about by trying to layer a case-insensitive filesystem on a case-sensitive filesystem?

I was just looking how to get rid of the whole problem. Of course, this will might break MTP and UMS, neither of which I care about.

I just unmounted and mounted a bunch of stuff. I can mount "sdcard" so that it's a direct mount off of /dev/block/dm-1. I can create foo, Foo and FOO.

Poke3:/mnt/runtime/full/emulated/0 # cat foo
I am entirely lower-cased.
Poke3:/mnt/runtime/full/emulated/0 # cat Foo
I am entirely mixed-cased.
Poke3:/mnt/runtime/full/emulated/0 # cat FOO
I am entirely upper-cased.

Ok, this might make some problems on Windows, but:
Do you care? Just don't make files that are case-synonyms.
Do you use Linux?

Full disclosure: Philosophically, I am inclined to case-insensitive, but not when it causes so much trouble.

[ottischwenk]

Originally Posted by Renate

Ok, this might make some problems on Windows, but:
Do you care? Just don't make files that are case-synonyms.
Do you use Linux?

No - but the time problem exist too, when using the file manager of Tab Ultra.
Loading this folder now (after some app installations, but after reboot) more than 10 minutes.

[Renate]

Originally Posted by ottischwenk

No

Ok, so give me the output:

$ mount|grep emulated

[ottischwenk]

Originally Posted by Renate

My Poke3 (Android 10) uses sdcardfs and it works fine.
Could somebody do a "mount | grep emulated" command and show what the "emulated" directories show?

Hmmm, Diskinfo (app) shows: Data - f2fs, r/w, 4.00 KiB

[ottischwenk]

Originally Posted by Renate

Ok, so give me the output:

$ mount|grep emulated

"No" is my answer regarding Linux use.
But:

/data/media on /mnt/runtime/default/emulated type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=1015,multiuser,mask=6,derive_gid,default_normal,reserved=50MB,unshared_obb)
/data/media on /mnt/runtime/read/emulated type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=9997,multiuser,mask=23,derive_gid,default_normal,reserved=50MB,unshared_obb)
/data/media on /mnt/runtime/write/emulated type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=9997,multiuser,mask=7,derive_gid,default_normal,reserved=50MB,unshared_obb)
/data/media on /mnt/runtime/full/emulated type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=9997,multiuser,mask=7,derive_gid,default_normal,reserved=50MB,unshared_obb)
/dev/fuse on /mnt/user/0/emulated type fuse (rw,lazytime,nosuid,nodev,noexec,noatime,user_id=0,group_id=0,allow_other)
/dev/fuse on /mnt/installer/0/emulated type fuse (rw,lazytime,nosuid,nodev,noexec,noatime,user_id=0,group_id=0,allow_other)
/dev/fuse on /mnt/androidwritable/0/emulated type fuse (rw,lazytime,nosuid,nodev,noexec,noatime,user_id=0,group_id=0,allow_other)
/dev/fuse on /storage/emulated type fuse (rw,lazytime,nosuid,nodev,noexec,noatime,user_id=0,group_id=0,allow_other)
/data/media on /mnt/pass_through/0/emulated type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=9997,multiuser,mask=7,derive_gid,default_normal,reserved=50MB,unshared_obb)
/data/media on /mnt/user/0/emulated/0/Android/data type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=1015,multiuser,mask=6,derive_gid,default_normal,reserved=50MB,unshared_obb)
/data/media on /storage/emulated/0/Android/data type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=1015,multiuser,mask=6,derive_gid,default_normal,reserved=50MB,unshared_obb)
/data/media on /mnt/androidwritable/0/emulated/0/Android/data type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=1015,multiuser,mask=6,derive_gid,default_normal,reserved=50MB,unshared_obb)
/data/media on /mnt/installer/0/emulated/0/Android/data type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=1015,multiuser,mask=6,derive_gid,default_normal,reserved=50MB,unshared_obb)
/data/media on /mnt/user/0/emulated/0/Android/obb type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=1015,multiuser,mask=6,derive_gid,default_normal,reserved=50MB,unshared_obb)
/data/media on /storage/emulated/0/Android/obb type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=1015,multiuser,mask=6,derive_gid,default_normal,reserved=50MB,unshared_obb)
/data/media on /mnt/androidwritable/0/emulated/0/Android/obb type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=1015,multiuser,mask=6,derive_gid,default_normal,reserved=50MB,unshared_obb)
/data/media on /mnt/installer/0/emulated/0/Android/obb type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=9997,multiuser,mask=7,derive_gid,default_normal,reserved=50MB,unshared_obb)

And Tab Ultra has Kernel-Version: 4.19.157-perf-gee3b7 .....

[Renate]

Isn't that thing Android 11?
Weren't you paying any attention?
sdcardfs is deprecated in Android 11.

Just kidding.

I was looking into this stuff. I can't see that anybody has ever done this yet.
All the mounting with sdcardfs comes from the /system/bin/sdcard in init.rc
You'd need to get rid of that.
You might be able to replace it with some more lines in the fstab.
Still, there are a lot of options and I'm not sure.
I'll try to get it working on my Poke3.

It's still worth pursuing with Onyx.
(You could also break your humongous directory down to smaller ones.)

[ottischwenk]

Originally Posted by Renate

(You could also break your humongous directory down to smaller ones.)

There is a problem - the files=epub come from Calibre via wireless connection from the PC and Calibre Companion puts them file by file into this directory - so I can directly transfer all 16,500 books for the first time; only when a new connection is made does a timeout occur.
However, when I tell Calibre Companion to put the books in authorname subfolders, it times out even on first contact.

Calibre Companion is a book transfer and library management system with a wide variety of classification criteria that makes book selection easier.

OK - Tab Ultra isn't actually a reader, but I'd like to be able to use it for something like that.

[Renate]

Well, that timeout seems to be something in the way it's transferring files.
Is this "Companion" a server with an open port or are you using some ftp or ssh or something?

I tried my idea. Yes, I can unmount all the sdcardfs stuff and bind remount it directly. I can even pull FOO, Foo, foo separately to my desktop over ADB. Unfortunately, my apps can't read /sdcard/ Also, this does not allowed the fine-grained permissions that get mapped by sdcardfs.

It's a pretty deep subject and if you have your device on WiFi and want to keep it secure you'd better get it right.