Using entire Kobo onboard storage as a Linux partition with RAID

[jmacindoe]

This guide will let us create a big Linux partition which we can use for Linux chroots. The aim is to end up with something like this

df -H
Filesystem    Size      Used Available Use% Mounted on
/dev/md0      10.0G     1.0G      9.0G  10% /mnt/data

A 10GB ext2 partition.

The Kobo onboard storage is formatted as FAT32 which has a lot of limitations, especially since it doesn't support Linux file permissions. It would be great if we could format it as ext2 but I expect this would cause problems with Nickel. The normal workaround is to create an image file which we can mount

dd bs=1M count=2000 if=/dev/zero of=/mnt/onboard/linux.img
mke2fs -F /mnt/onboard/linux.img
mount /mnt/onboard/linux.img /mnt/data

This solves the problem, except you need to know the size in advance. Since I'm mostly using my Kobo for development/hacking, I'd happily give almost all the SD space to Linux. But FAT32 only supports files up to 4GB so we can't create a single huge image.

Enter RAID.

It sounds a bit crazy but we can create a bunch of 4GB image files and combine them all together using RAID. Then we can put everything Linux related inside (e.g. I have an Alpine chroot and a Debian chroot both inside this partition). Here's how to do it.

1. Download RAID kernel modules from here https://github.com/jmacindoe/kobo-kernel-modules/. I've only compiled them for the latest two devices. If you get this working on older devices please add the modules to the repo in a PR!

insmod md-mod.ko
insmod linear.ko

2. Create the images. Normally in RAID0 all the images need to be the same size, but we'll use Linear mode, so you can make the images any sizes.

dd bs=1M count=4000 if=/dev/zero of=/mnt/onboard/data1.img
dd bs=1M count=4000 if=/dev/zero of=/mnt/onboard/data2.img
dd bs=1M count=2000 if=/dev/zero of=/mnt/onboard/data3.img

losetup /dev/loop1 /mnt/onboard/data1.img
losetup /dev/loop2 /mnt/onboard/data2.img
losetup /dev/loop3 /mnt/onboard/data3.img

3. Set up an Alpine Linux chroot if you haven't already https://www.mobileread.com/forums/sh...d.php?t=336175. This is needed to bootstrap the RAID array, so we'll need to keep this even once we have our big partition. So probably set it up to only take up a very small amount of space. Now in the chroot run

pkg add mdadm
mdadm --create /dev/md0 --level=linear --raid-devices=3 /dev/loop1 /dev/loop2 /dev/loop3

4. Now back outside the chroot run

mke2fs -F /dev/md0
mkdir /mnt/data
mount /dev/md0 /mnt/data

5. Boom! It's a gigantic ext2 partition. Next time you reboot you can get it back with these commands

insmod md-mod.ko
insmod linear.ko

losetup /dev/loop1 /mnt/onboard/data1.img
losetup /dev/loop2 /mnt/onboard/data2.img
losetup /dev/loop3 /mnt/onboard/data3.img

# Update this command with your Alpine chroot directory
chroot /mnt/user/ mdadm --assemble --scan

mount /dev/md0 /mnt/data

[frostschutz]

this might sound crazy, but you could just shrink mmcblk0p3 and make a new mmcblk0p4 for your ext4 /mnt/data

loop device on /mnt/onboard/data.img files will prevent the filesystem from umounting, so when you connect USB, you will get filesystem corruption issues since it's still mounted by Kobo

----

if you don't want to repartition, you could create a data.img file which is unfragmented, find out where it is allocated (filefrag) then use losetup offset sizelimit options with /dev/mmcblk0 as a source (add mmcblk0p3 start to the offset), thus creating a virtual partition which does not block mmcblk0p3 from getting unmounted (same principle as swap files)

but your filesystem will corrupt if you get the offsets wrong or if anything touches changes the data.img file.

edit: oh right, there's the filesize issue in vfat... so you'd need multiple .img files that are allocated back to back. not sure if it's possible

edit2: it's possible. it does take some fiddling though.

# mkfs.vfat -F 32 foobar.img
# mount -o loop foobar.img
# for i in {1..8}; do
       yes $(printf "%15s" test_$i) | dd iflag=fullblock bs=1G count=1 of=test$i.img
   done
# sync
# filefrag -v -e test*.img
Filesystem type is: 4d44
File size of test1.img is 1073741824 (2097152 blocks of 512 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0.. 2097151:      32816..   2129967: 2097152:             merged,eof
test1.img: 1 extent found
File size of test2.img is 1073741824 (2097152 blocks of 512 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0.. 2097151:    2129968..   4227119: 2097152:             merged,eof
test2.img: 1 extent found
File size of test3.img is 1073741824 (2097152 blocks of 512 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0.. 2097151:    4227120..   6324271: 2097152:             merged,eof
test3.img: 1 extent found
File size of test4.img is 1073741824 (2097152 blocks of 512 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0.. 2097151:    6324272..   8421423: 2097152:             merged,eof
test4.img: 1 extent found
File size of test5.img is 1073741824 (2097152 blocks of 512 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0.. 2097151:    8421424..  10518575: 2097152:             merged,eof
test5.img: 1 extent found
File size of test6.img is 1073741824 (2097152 blocks of 512 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0.. 2097151:   10518576..  12615727: 2097152:             merged,eof
test6.img: 1 extent found
File size of test7.img is 1073741824 (2097152 blocks of 512 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0.. 2097151:   12615728..  14712879: 2097152:             merged,eof
test7.img: 1 extent found
File size of test8.img is 1073741824 (2097152 blocks of 512 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0.. 2097151:   14712880..  16810031: 2097152:             merged,eof
test8.img: 1 extent found
# umount loop
# hexdump -C foobar.img
[...]
1006000  20 20 20 20 20 20 20 20  20 74 65 73 74 5f 31 0a  |         test_1.|
*
41006000  20 20 20 20 20 20 20 20  20 74 65 73 74 5f 32 0a  |         test_2.|
*
81006000  20 20 20 20 20 20 20 20  20 74 65 73 74 5f 33 0a  |         test_3.|
*
c1006000  20 20 20 20 20 20 20 20  20 74 65 73 74 5f 34 0a  |         test_4.|
*
101006000  20 20 20 20 20 20 20 20  20 74 65 73 74 5f 35 0a  |         test_5.|
*
141006000  20 20 20 20 20 20 20 20  20 74 65 73 74 5f 36 0a  |         test_6.|
*
181006000  20 20 20 20 20 20 20 20  20 74 65 73 74 5f 37 0a  |         test_7.|
*
1c1006000  20 20 20 20 20 20 20 20  20 74 65 73 74 5f 38 0a  |         test_8.|
[...]

This area is allocated by 8 separate files (1G each) but since the files are allocated back to back w/o any gaps or fragmentation, you can use it as consecutive area for a virtual partition mapped out by offset loop device.

It would not work with fragmentation in between so it's easiest to achieve if you start out with an empty filesystem...

Uhhh... so it can be done but partition is so much easier, why the pain.

----

either way doing this with raid is a bit... extreme