Development Rebuilding the filesystem to increase app storage

[pablob]

Hi Everyone!

I have managed to hack a new UBI image (based on the Dingo golden update one) with a larger space for applications (taken from the storage volume, since that can go into the SD card).

This is *NOT* user-friendly, and I am not sure it works properly. I have found the following issues after flashing with the new image:

• The modified Dingo seems to work, but it is impossible to install applications using the Package Manager (it complains that there is not enough free space ). However, installing APKs widh ADB works. Go figure...
• Sometimes, you can't even download files from the browser!
• You can upgrade to Ermine (I went to Ermine 1.02), and there you can install applications from the web or files apparently without problems.
• In Ermine, the Calendar shortcut in the home screen did not work, but I just erased and re-added it and it works.
• In Ermine, also, the "x" for applications does not slide anymore, but the applications only appear when you tap on it (that might be normal AFAIK, I never ran the normal Ermine ).
• I cannot guarantee everything works; the system might be f****d-up a thousand different ways which I don't recognize from my lack of experience.

On the other hand, if things go bad the device should be recoverable using the Dingo golden update (I lost count of the number of times I have flashed that ).

In order to make run the script that makes the new image you need:

• A GNU/Linux system, with a recent kernel (3.0 works, 2.6.32 does not) with nandsim and ubifs compiled as modules (ubifs can be probably compiled in).
• A recent version of mtd-utils.
• The android_ubi.img.crc and update.sh from the Dingo golden update.
• sed
• Root access in the computer, not the PE (you need to run this as root).

Other technical notes:

• The image needs to be flashed with ubiformat 1.5 or higher. The one in the Dingo maint ramdisk does not work (it seems it is a known bug), I have used the one in the Ermine maint ramdisk and added it to the tools.zip file.
• For some reason, the CRC of the flashed image fails verification, even though the image is properly flashed (I think it has to do with the newer ubiformat you have to use but I don't know for sure). That means that you need to hack update.sh so that it ignores the CRC check.
• I've done the flashing over a Dingo image, it might work over Ermine but I don't know for sure.

The script I have used follows. Again, *DO NOT USE* unless you know exactly what you are doing!

#! /bin/sh
# This script grabs the UBI image from the Pocket Edge Dingo golden update and
# creates a modified UBI image where the "userdata" (where apps are installed)
# and the "storage" (where the library resides) can be custom-sized (with some
# restraints).
#
# Requires:
#  * A recent enough kernel with nandsim and ubifs.
#  * mtd-utils (probably a recent version)
#  * sed
#  * The android_ubi.img.crc file from the Dingo Golden update.
#  * The update.sh file from the Dingo Golden update.
#  * It needs to be run as root.
# 
# Creates new-android_ubi.img.crc and new-update.sh. Put these two files into 
# the original update.zip and update the system in your Pocket Edge.

# New size for the storage volume and userdata volume in logical eraseblocks 
# (LEBs); each LEB is 504Kb long. The sum of these must be equal or less to
# 6357 (if it's less it will work, but you will be wasting space).
# NOTE: Keep in mind that the actual usable space will be at least ~70 Mib 
#       smaller because of filesystem structures.
STO_SIZE=1200    # ~590 MiB
USD_SIZE=5157	# ~2.5 GiB

# Temporary directory. BE CAREFUL, it will be wiped out and created each time!
TMPDIR=/tmp/muckrom

# It should not be necessary to edit anything below this.

# File names.
CRC_IMAGE=android_ubi.img.crc
IMAGE=android_ubi.img

# Check for an image without CRC. If there is none, and the CRC image is
# present, remove it. 
if [ ! -r $IMAGE ]
then
  echo "Eliminating the CRC from the image file $CRC_IMAGE."
  dd if=$CRC_IMAGE of=$IMAGE bs=512k count=327
fi

# First, load up the NAND simulator, trying to reproduce the layout in the PE.
# The first partition will be unused, and the second one is the one with the
# UBI volumes (the total NAND size is 8192 LEBs, and the second partition size
# is 7864 LEBs. 
echo "Loading the NAND simulator."
modprobe ubifs
modprobe nandsim first_id_byte=0xec second_id_byte=0xd7 third_id_byte=0x00 fourth_id_byte=0x36 parts=328,7864 cache_file=/tmp/nand.img

# This will be the MTD device to use.
MDEV=/dev/mtd1

# Copy (bit-for-bit) the image into the simulated NAND partition.
echo "Uploading image into simulated NAND."
dd if=$IMAGE of=$MDEV bs=512k count=327

# Verify the copy, to make sure nothing went wrong.
echo "Verifying the image copied to the simulated NAND."
nanddump -o -b -l `echo 327*512*1024 | bc` -s 0 -f verify.img $MDEV
cmp $IMAGE verify.img
if [ $? -ne 0 ]
then
  echo "Problem copying the image to the simulated NAND!"
  rm verify.img
  modprobe -r nandsim
  rm -f /tmp/nand.img
  exit 1
fi
echo "Dump verified OK!"
rm verify.img

# Attach the UBI volumes (force UBI not to use subpages, for some reason the 
# NAND simulator reports 1Kb subpagesm while the PE hardware does not seem to
# support them).
echo "Attaching UBI device."
ubiattach -p $MDEV -O 4096

# Create the temporary directory.
echo "Creating the temporary directory $TMPDIR."
rm -r $TMPDIR
mkdir -p $TMPDIR

# Create the directories and mount the UBI images.
echo "Mounting original UBI images."
for vol in system userdata cache storage 
do
  mkdir $TMPDIR/$vol
  mount -t ubifs ubi:$vol $TMPDIR/$vol
done

# Make the UBI filesystem images.
echo "Making the new UBI filesystem images..."
echo "   ...system"
mkfs.ubifs -v -r $TMPDIR/system -m 4096 -e 516096 -c 712 -j 10584KiB -R 0 -x favor_lzo -X 1 new-system.img 
echo "   ...userdata"
mkfs.ubifs -v -r $TMPDIR/userdata -m 4096 -e 516096 -c $USD_SIZE -j 10584KiB -R 0 -x favor_lzo -X 1 new-userdata.img
echo "   ...cache"
mkfs.ubifs -v -r $TMPDIR/cache -m 4096 -e 516096 -c 712 -j 17640KiB -R 4536KiB -x favor_lzo -X 1 new-cache.img
echo "   ...storage"
mkfs.ubifs -v -r $TMPDIR/storage -m 4096 -e 516096 -c $STO_SIZE -j 32760KiB -R 4536KiB -x favor_lzo -X 1 new-storage.img

# Unmount the UBI images.
echo "Unmounting original UBI images."
for vol in system userdata cache storage
do
  umount $TMPDIR/$vol
done

# Then, let's detach UBI from the device.
echo "Detaching UBI device."
ubidetach -p $MDEV

# We can remove the NAND simulator now.
echo "Removing the NAND simulator."
modprobe -r nandsim
rm -f /tmp/nand.img
modprobe -r ubifs

# Make the big, ubinized, image.
cat <<EOF > new-ubinize.cfg
[system]
mode=ubi
image=./new-system.img
vol_size=358848KiB
vol_id=0
vol_type=dynamic
vol_name=system

[userdata]
mode=ubi
image=./new-userdata.img
vol_size=`echo 504*$USD_SIZE | bc`KiB
vol_id=1
vol_type=dynamic
vol_name=userdata

[cache]
mode=ubi
image=./new-cache.img
vol_size=358848KiB
vol_id=2
vol_type=dynamic
vol_name=cache

[storage]
mode=ubi
image=./new-storage.img
vol_size=`echo 504*$STO_SIZE | bc`KiB
vol_id=3
vol_type=dynamic
vol_name=storage

EOF

echo "Running ubinize."
ubinize -o new-$IMAGE -p 512KiB -m 4096 new-ubinize.cfg

# Add the CRC to the end.
# After this, the image should be ready to put in a custom "update.zip" file.
echo "Adding CRC to the modified image."
CRC=`/usr/sbin/ubicrc32 new-$IMAGE`
perl -e "print pack L, ${CRC}" > crc_file.dat
cat new-$IMAGE crc_file.dat > new-$IMAGE.crc
rm -f crc_file.dat

# NOTE: The custom "update.sh" should have VFAT_NUM_BLOCKS equal to
# $STO_SIZE*504 (the new number of blocks in the storage filesystem)
# To that, we substract the journal and root reserved space to make sure the
# image will never be larger than the available space.
NEW_BLOCKS=`echo $STO_SIZE*504-32760-4536 | bc`
sed s/VFAT_NUM_BLOCKS=2617152/VFAT_NUM_BLOCKS=${NEW_BLOCKS}/ update.sh > new-update.sh

[vicinc]

while this is good, why dont u provide your golden modified as a file?

[pablob]

Originally Posted by vicinc

while this is good, why dont u provide your golden modified as a file?

My main reason is that I am still not sure it works well (I have just finished doing the Ermine update today), and not having the file makes it certain that only people who really know what they are doing can do this.

A more trivial reason is that I don't know what would be an ideal split between app space and storage.

With the caveats above, plus the fact that this will *ERASE ALL DATA AND APPS*, here it goes:

1) Downgrade all the way to Dingo (you might need to use one of the hellbent updates for that).
2) Download the file from http://www.megaupload.com/?d=93P0FWVB.
3) Save the file as "update.zip" in the root directory of a USD drive or a MicroSD card (I had more luck with the microSD card, but YMMV).
4) Flash the PE with it. If everything goes right, it will reboot. If there is a problem, the PE will power off instead of rebooting.
4b) If something went wrong, there should be an "upd.log" file in the root directory of the medium used for the upgrade. That might give some info on what happened.
5) Upgrade to Ermine (don't bother with Dingo, it doesn't work well for some reason).
5b) You can upgrade to Allmine, if you want (I did it and it seems to work).
6) You'll probably want to move the library to SD. I don't know if that works yet because I haven't received my new card.
7) Fill it up with apps and let me know if it works!
8) Don't complain if your PE sprouts wings and flies away.

Ok, that's it. Good luck everyone!

P.S.: It seems to be working on mine, so the obsession-fueled adrenaline rush is over. Now I need to go work.

[Filark]

Very cool work you are doing, pablob! Nothing I will attempt, of course, but I'm really impressed that you can figure this out!

[pablob]

Thanks Filark!

It was a nice learning experience about NAND flash, Linux MTDs and UBI.

Now I'm thinking it should be possible to actually coalesce all the updates (Dingo, Ermine, Allmine) into one single update file (a ROM, you could say)...

The question is whether I'll have time to think about that...

[nikkie]

Excellent work. I'm personally more interested in "re-partitioning" the device so we can add a real android style recovery partition, but this is a wonderful step in the right direction. For that I'd need to shrink one of the images and create a new one. Looks like I could learn quite a bit from this, will have to dig into it tomorrow.

[vicinc]

hip hip hooray. yes. android recovery and larger apps partition is the way forward. really not interested in the internal sd space for anything else as we do have 16 gb + sd cards now. can have the library on different ext sd and no issue on removing the cards as with current method for extending app partition.

[vicinc]

can we dream for a gingerbread rom?

[Gunnerp245]

Originally Posted by vicinc

can we dream for a gingerbread rom?

Would our hardware support it?

[tarvoke]

quite possibly... a lot of gingerbread "rom"s I've seen for phones, basically keep the old kernel (mostly b/c binary blobs for certain device drivers, as we do have on the PE with eink) from the phone's existing rom, and then add as much of gingerbread framework as they can (some of which is kernel dependent, so bits get left out)

[tekknogenius]

This is fantastic! I was thinking about how to do this myself after I learned how it works from fixing my frozen PE. I was thinking about replacing the existing storage partition and using that space for the data partition since I was using the link2sd method to put stuff there.

This is awesome.

[pablob]

Originally Posted by nikkie

Excellent work. I'm personally more interested in "re-partitioning" the device so we can add a real android style recovery partition, but this is a wonderful step in the right direction. For that I'd need to shrink one of the images and create a new one. Looks like I could learn quite a bit from this, will have to dig into it tomorrow.

Thanks!

I have been thinking on ways to have recovery working, and this is what I have come up so far:

• We'd need to have a full UBI image with a decent, up-to-date system (I'd vote for Allmine) so we can reflash to that.
  
• By properly resizing the volumes we should devote ~200 MiB to this image so it can be permanently stored somewhere in the device. This should probably be another UBI image in a different MTD device, that's where "re-partitioning" will come into play.
  
• Re-partitioning might affect the system and require changes to the ramdisks...
  
• Then, we should have a program we can install in the maintenance ramdisk that will let the user choose whether to update using "update.zip" or reflash from the recovery image. I imagine that CWM does this, but we could probably have a more rudimentary program if it's easier.

Ok, I'm not saying anything new in here, but I wanted to do a "brain dump".

[vicinc]

allmine works fine, at least for me.
An allmine recovery and option to update to any .zip file is all we need, really. menu+rotate should bring up that menu.
than no need for serial and tftps from linux. we will have a failsafe recovery and everybody could experiment safely.
Agreed on no internal storage, or minimal storage and plenty of space for apps. external storage is enough really if you have a large 8/16/32 gb sd.

[pablob]

Interesting... Yesterday I turned off my PE with no problems. Today, when I turn it on, it goes all the way to the 4 dots and then it stays there, flashing from time to time. It seems the boot up got stuck somewhere, but unfortunately I have no way of knowing where (I don't have the serial port open yet). It might be random, it might be a problem with the filesystem change, who knows...

I wonder if anybody has any idea on how to get a console with the boot messages. That might require a bit of "kernel magic"...

[vicinc]

I think you may have bad sectors. Repartitioning could have brought those forward.