Repartitioning Kindle Filesystem

[EternalCyclist]

@eureka: Thank you. I will try this method on my K4 later.

When you mount the new ext3 partition you do not use the entries of the partitiontable in mmcblk0p4. This means there is a discrepancy between that partition table and the actual start of the ext3 partition?

This calculated offset could be written into the partition table on mmcblk0p4, so that mounting the ext3 partition on the host can be done without the offset option?

Why do I have to mount it via /dev/loop?
What about simply doing "mount -o offset=1075838976 /dev/mmcblk0p4 /somehwere" ?

[eureka]

Originally Posted by EternalCyclist

When you mount the new ext3 partition you do not use the entries of the partitiontable in mmcblk0p4. This means there is a discrepancy between that partition table and the actual start of the ext3 partition?

There is a disrcepancy between start of free partition (/dev/mmcblk0p4p2) and start of ext3 filesystem. Or, in other words, there is a little free space (<= 4MB) between of start of that parition and actual start of ext3 filesystem to compensate misaligning of partition.

But it's:

• unintentional (I just didn't think about aligning at writing of first post)
• not required at all (I guess without aligning at 4MB nothing really bad will happen, maybe, just pereformace will degrade)

Anyway, this free space could be eliminated via extending of /dev/mmcblk0p4p1 (userstore parition) and re-initing it. I mean, just initialize PART_SIZE in diff from my "first" post with a bit larger value (UPD: exactly 2101232). Then /dev/mmcblk0p4p2 will be aligned on 4MB.

Also, I don't use entries from parition table at /dev/mmcblk0p4, because I don't know how to do it. I believe, it's not possilble at all. /dev/mmcblk0p4 is the partition, not the separate block device. Tools just don't know anything about that "embedded" paritition table and they couldn't be pointed at it.

Originally Posted by EternalCyclist

This calculated offset could be written into the partition table on mmcblk0p4, so that mounting the ext3 partition on the host can be done without the offset option?

Explicit offset is required, because location of that new partition is unknown to OS (because it can't access parition table at /dev/mmcblk0p4), not because of some disrcepancy.

Originally Posted by EternalCyclist

Why do I have to mount it via /dev/loop?
What about simply doing "mount -o offset=1075838976 /dev/mmcblk0p4 /somehwere" ?

You could mount with your command. With losetup loop device is chosen by you, and with your command loop device is chosen by mount tool. It's the sole difference, AFAIK.

[eureka]

And here is a basic upstart script for mounting/unmounting filesystem (described earlier):

start on started system and started syslog
stop on stopping filesystems

pre-start script
  source /etc/upstart/functions

  losetup -o 1075838976 /dev/loop/1 /dev/mmcblk0p4
  mount -t ext3 /dev/loop/1 /mnt/newpart
  
  f_log I newpart mount "newpart is mounted"
end script

post-stop script
  source /etc/upstart/functions

  umount /mnt/newpart
  losetup -d /dev/loop/1
  
  f_log I newpart umount "newpart is unmounted"
end script

It's really basic, if you'll use it as-is, expect some (rare?) ignored edge cases (can't imagine any example right now, though).

[geekmaster]

Originally Posted by EternalCyclist

...
Why do I have to mount it via /dev/loop?
What about simply doing "mount -o offset=1075838976 /dev/mmcblk0p4 /somehwere" ?

I was thinking this as well. In my mntusb.params (from my Select Boot thread) I run /mnt/us/RUNME.sh at startup after I directly mount /mnt/us using an offset and no loop mount. It works fine. As far as I know, no (automatic) loop mount is used (unless I am misunderstanding something here).

Originally Posted by eureka

...
You could mount with your command. With losetup loop device is chosen by you, and with your command loop device is chosen by mount tool. It's the sole difference, AFAIK.

But his command did not include "mount -o loop", and without the "loop", are you sure it is the same as losetup?

I thought that loop mounting was for mounting an image file as a block device, rather than raw partitions. If I am wrong in my thinking here, please enlighten me. Thanks...

EDIT: WikiPedia seems to agree with me:
http://en.wikipedia.org/wiki/Loop_device

In Unix-like operating systems, a loop device, vnd (vnode disk), or lofi (loopback file interface) is a pseudo-device that makes a file accessible as a block device.

And as I see it, the whole point of your exercise is to mount a raw partition that you created instead of mounting an image file, to avoid problems experienced with out-of-memory errors while using loop mounts. Remind me again, WHY are we loop mounting here?

[EternalCyclist]

Originally Posted by eureka

Also, I don't use entries from parition table at /dev/mmcblk0p4, because I don't know how to do it.

fdisk -l /dev/mmcblk0p4

sfdisk --force -l /dev/mmcblk0p4

When USB drive is exported to the PC:

fdisk -l /dev/sdc

also shows the ''secondary'' partition table.

Adding the offset to the start value in the partition table avoids the need to remember this offset.

[NiLuJe]

Just a quick note to point out that (according to the PW's cramfs mount scripts), loop/1 is historically reserved for the SDCard (while loop/0 is the userstore). (And the PW uses a couple of loop devices for cramfs, too).

[knc1]

Notes:

There is (should be, don't know the behavior of your Busybox build) a: "find next free" loop device option.
No need to hard code which one to use, lots of reasons to not do so.

Yes, the loop driver can operate as a device-device layer, not just as a file-device layer.
In fact, if I recall correctly, the author of the driver recommends that it be used device-device rather than the more common usage of turning a file into a device.

Somewhere in this thread someone posted that the large offset was to the **second** FAT, it isn't, its the offset to the **first** FAT.

You can see that in the layout of the eMMC device itself, which has a DOS disk label in the first sector and a 32Mbyte offset to the start of the first partition's file system.

And yes, you can have a "DOS disk label" at the first sector of each partition, further dividing that partition.
Its in the MS specs. (please don't ask for the link, I lost it years ago).
But when you find it, you will find that it is a recursive definition.

Not many disk partitioning and formating applications handle that aspect of the "DOS disk label".
Back in the day that I needed it for a commercial project, I had to write my own.

[twobob]

well. I have very little to add other than when the process is 100% solid I will certainly be giving this a go. Thanks everyone for the info

[eureka]

Originally Posted by geekmaster

As far as I know, no (automatic) loop mount is used (unless I am misunderstanding something here).
But his command did not include "mount -o loop", and without the "loop", are you sure it is the same as losetup?

I executed exactly that command (just with -o <offset>, without -o loop) on KT and there was new mounted loop device in mount output.

Originally Posted by geekmaster

And as I see it, the whole point of your exercise is to mount a raw partition that you created instead of mounting an image file, to avoid problems experienced with out-of-memory errors while using loop mounts. Remind me again, WHY are we loop mounting here?

Explicit offset (and loop mounting) is required, because location of that new partition is unknown to OS (because it can't access parition table at /dev/mmcblk0p4), not because of my preference.

Originally Posted by EternalCyclist

fdisk -l /dev/mmcblk0p4

sfdisk --force -l /dev/mmcblk0p4

When USB drive is exported to the PC:

fdisk -l /dev/sdc

also shows the ''secondary'' partition table.

Adding the offset to the start value in the partition table avoids the need to remember this offset.

I missed the point of your demonstration, sorry. I know how to look into that parition table. I'm calculating offset with looking into parition table. I'm using number just because this calculation could be done once and this number could be then remembered, written into some user file (upstart script, for example), etc.

What I don't know is how to mount with, say mount /dev/mmcblk0p4p2 instead of using losetup.

Originally Posted by NiLuJe

Just a quick note to point out that (according to the PW's cramfs mount scripts), loop/1 is historically reserved for the SDCard (while loop/0 is the userstore). (And the PW uses a couple of loop devices for cramfs, too).

Thanks. Good note.

Originally Posted by knc1

Notes:

There is (should be, don't know the behavior of your Busybox build) a: "find next free" loop device option.
No need to hard code which one to use, lots of reasons to not do so.

There is losetup -f, but it always displays /dev/loop0:

[root@kindle root]# mount
rootfs on / type rootfs (rw)
/dev/root on / type ext3 (ro,noatime,nodiratime,data=writeback)
none on /proc type proc (rw,nosuid,nodev,noexec,relatime)
none on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,relatime,mode=755)
tmpfs on /dev/shm type tmpfs (rw,relatime)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620)
tmpfs on /var type tmpfs (rw,relatime,size=32768k)
/dev/loop/1 on /mnt/newpart type ext3 (rw,relatime,errors=continue,data=writeback)
/dev/mmcblk0p3 on /var/local type ext3 (rw,relatime,errors=continue,data=writeback)
fsp on /mnt/us type fuse.fsp (rw,nosuid,nodev,noatime,user_id=0,group_id=0)
/dev/loop/0 on /mnt/base-us type vfat (rw,noexec,noatime,nodiratime,fmask=0022,dmask=0022,codepage=cp437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro)

[root@kindle root]# losetup -h
losetup: invalid option -- 'h'
BusyBox v1.17.1 (2012-07-17 16:29:54 PDT) multi-call binary.

Usage: losetup [-o OFS] LOOPDEV FILE - associate loop devices
	losetup -d LOOPDEV - disassociate
	losetup [-f] - show

Options:
	-o OFS	Start OFS bytes into FILE
	-f	Show first free loop device

[root@kindle root]# losetup -f
/dev/loop0

[EternalCyclist]

Originally Posted by eureka

What I don't know is how to mount with, say mount /dev/mmcblk0p4p2 instead of using losetup.

Sorry, I misunderstood you. I also don't know how to mount it directly on Kindle.
But on the connected PC I could mount it directly with the mount command.
- On the other hand, why would someone want to mount the ext3 partition on the PC via usbdrive. I am prefering sshfs.

So: The offset could be written to the partition table. But it is practically no use. I agree.

[eureka]

Originally Posted by NiLuJe

Just a quick note to point out that (according to the PW's cramfs mount scripts), loop/1 is historically reserved for the SDCard (while loop/0 is the userstore). (And the PW uses a couple of loop devices for cramfs, too).

Indeed, on PW loops from 2 to 7 are already used. I've also found mentioning of /dev/loop/1 in some system scripts (/usr/sbin/wfm_mount, /usr/sbin/wfm_erase). So it's safer to create /dev/loop/8 and use it, as in this changed upstart script:

Spoiler:

start on started system and started syslog
stop on stopping filesystems

pre-start script
  source /etc/upstart/functions

  [ -e /dev/loop/8 ] || mknod /dev/loop/8 b 7 8
  losetup -o 1075838976 /dev/loop/8 /dev/mmcblk0p4
  mount -t ext3 /dev/loop/8 /mnt/newpart
  
  f_log I newpart mount "newpart is mounted"
end script

post-stop script
  source /etc/upstart/functions

  umount /mnt/newpart
  losetup -d /dev/loop/8
  [ -e /dev/loop/8 ] && rm /dev/loop/8
  
  f_log I newpart umount "newpart is unmounted"
end script

Originally Posted by EternalCyclist

Sorry, I misunderstood you. I also don't know how to mount it directly on Kindle.
But on the connected PC I could mount it directly with the mount command.
- On the other hand, why would someone want to mount the ext3 partition on the PC via usbdrive. I am prefering sshfs.

So: The offset could be written to the partition table. But it is practically no use. I agree.

OMG, I've just understood what you've already knew. This new parition will be seen by computer when KT will be connected by USB.

While it generally doesn't bring any harm, sometimes Linux system automounts all paritions on removable device (such as Kindle) and then this parition could be used by heedless user simultaneously from computer side and Kindle side. So there is some value in skipping of creating of new parition (i.e. skipping of making a new record in parition table) and then just using free space after shrinked userstore parition as-is.

Here is diff of userstore from Kindle Touch 5.1.2 with just shrinking of userstore parition:

Spoiler:

PHP Code:

--- /etc/upstart/userstore
+++ /var/tmp/root/userstore
@@ -29,7 +29,7 @@
     PART_START=${_TRACKSIZE}
 
     # try to read back the existing partition size
-    PART_SIZE=`sfdisk --force -l ${MNTUS_DEV} | awk '$1 ~ '/${_BASE_MNTUS_DEV}p1/\ '{ print $5 }'`
+    PART_SIZE=`expr 1024 \* 1024 \* 1024 \/ 512`
 
     if [ -z "${PART_SIZE}" ]; then
         _CREATE=1
@@ -41,7 +41,7 @@
 
         # create the volume partition
         sfdisk -q --force -uS ${MNTUS_DEV} <<EOI
-${PART_START},,b
+${PART_START},${PART_SIZE},b
 EOI
 
         # read back the created partition size
@@ -237,8 +237,6 @@
 # returned status is that of the called routine
 
 case "$1" in
-  start) start_us ;;
-  stop) stop_us ;;
-  init) init_us ;;
-  *) echo "usage: $0 [start|stop|init]" ; exit 1 ;;
+  repartition) init_us "create";; 
+  *) echo "usage: $0 repartition" ; exit 1 ;; 
 esac 


And here is command for calculating absolute offset of new free space (in bytes, from start of eMMC device):

[root@kindle root]# expr 512 \* \( \
>   $(sfdisk -uS -l --force /dev/mmcblk0p4 | awk '/mmcblk0p4p1/ {print $3 + 1}') \
>   \+ $(cat /sys/class/block/mmcblk0p4/start) \
> \)

[eureka]

Right before intializing of userstore parition (i.e. before creating of parition table and formatting of new parition), usestore script clears first cylinder of /dev/mmcblk0p4 with the following command (some values are defined with variables, but I've expanded them here just for convenience):

# clear the first cylinder
dd if=/dev/zero of=/dev/mmcblk0p4 bs=64 count=1

Does anybody have any hypothesis about point of this command?

[knc1]

Yeah, right!
As if a flash device had "cylinders".
Do not presume the author of what you read in Consumer Linux devices knew what they where doing.

[twobob]

Originally Posted by knc1

Yeah, right!
As if a flash device had "cylinders".
Do not presume the author of what you read in Consumer Linux devices knew what they where doing.

or... more to the point... what "you" are doing

[geekmaster]

Windows XP likes to "think" that FAT devices have 53 sectors per track, which makes a cylinder a multiple of 53 (NOT a power of two). Because all partitions start on a cylinder boundary, this causes misalignment between disk clusters and mmc erase block, causing wasteful read/modify/write operations across mismatched boundaries (which has the side effect of flashing each erase block TWICE). This was remedied in Windows 7, by using SSD-friendly 1MB alignment.

So, from an XP point of view, the device IS treated as having "cylinders", much to the detriment of performance (and extra write wear). And older linux drivers use "XP-style" FAT partition alignment...