Hacking SD card file system - Page 3

auspex · 06-05-2013, 08:35 AM

Quote:

Originally Posted by flowoeB

thank you, found the initial mount of "onboard" at boot and the rmount after unplugging the usb. I'll simply do a mount without the vfat argument if there is an error on the original mount line... will give it a try tomorrow...

I looked at that - you can't actually "just do a mount without the vfat argument" if the mount of the internal card is like the one for the external card - it includes an option ("-o shortname=mixed") that's invalid on an ext* partition.

I suggested native linux partitions long ago, but never had the gumption to try it myself - with my 800 odd books, and too many collections, it is very slow to boot, so I just don't ever boot if I can help it!

flowoeB · 06-07-2013, 12:18 AM

there are no short filenames under ext3 - so why would you use "-o shortname=mixed" ? I just deleted the argument.

auspex · 06-07-2013, 08:43 AM

Like I said - you can't "just do a mount without the vfat argument".

flowoeB · 06-21-2013, 01:27 PM

hm, tried to mount an external ext3 sd on my aura today and it won't mount - I see it, when I plug the reader into my macs USB and can write books to it - but I can't access it with the reader itself... a messagebox is popping up that tells me to format the sd with fat32... the USB and SD files are modified as described above and I'm running 2.6.1b...

Ken Maltby · 06-21-2013, 04:13 PM

Quote:

Originally Posted by flowoeB

hm, tried to mount an external ext3 sd on my aura today and it won't mount - I see it, when I plug the reader into my macs USB and can write books to it - but I can't access it with the reader itself... a messagebox is popping up that tells me to format the sd with fat32... the USB and SD files are modified as described above and I'm running 2.6.1b...

Hmm... Did you get the internal/onboard SD working?

Luck;
Ken

flowoeB · 06-22-2013, 12:24 AM

didn' try it. if the external does not work I don't think the internal will... I'll take a deeper look into this as soon as i have more time...

frostschutz · 12-11-2014, 02:52 PM

Even without changing any of the rcS/sd/... scripts that do the mounting, it's possible to change both the internal and external storage to ext4.

I noticed that mount and dosfsck are located in /bin/, but PATH is /sbin:/usr/sbin:/bin:/usr/bin. Thus you can replace the mount and dosfsck commands with shell scripts in either /sbin/ or /usr/sbin/.

Example for /sbin/mount:

Code:

#!/bin/busybox sh

echo mount "$@" >> /mount.log

exec /bin/mount "$@"

So all this does is log the call and then execute the original binary in /bin/.

Here's the log:

Code:

mount -o remount,noatime,nodiratime /dev/mmcblk0p1 /
mount -t vfat -o noatime,nodiratime,shortname=mixed,utf8 /dev/mmcblk0p3 /mnt/onboard
mount -t devpts devpts /dev/pts
mount -r -t vfat -o noatime,nodiratime,shortname=mixed,utf8 /dev/mmcblk1p1 /mnt/sd
mount -o remount,rw /mnt/sd
mount -o remount,ro /mnt/sd
mount -o remount,rw /mnt/sd
mount -o remount,ro /mnt/sd
mount -r -t vfat -o noatime,nodiratime,shortname=mixed,utf8 /dev/mmcblk1p1 /mnt/sd
mount -o remount,rw /mnt/sd
mount -o remount,ro /mnt/sd

Note the various proc, tmpfs, etc. mounts are missing from this log, since rcS for some reason calls mount with full path for those, so the PATH trick does not catch them. If there were any such calls for the vfat, you'd have to replace /bin/mount and exec /bin/busybox mount instead.

And the dosfsck log:

Code:

dosfsck -a -w /dev/mmcblk0p3
dosfsck -a -w /dev/mmcblk1p1
dosfsck -a -w /dev/mmcblk1p1

So you can hook into those calls and mangle the parameters any way you like. And it should even survive firmware updates.

Getting those scripts onto the reader is easy enough using KoboRoot.tgz. This method seems to be lacking an uninstall facility though. My iriver Story HD was nice enough to execute a shell script for every update, but if Kobo has something like it, I haven't found it yet. Naturally you could just use the same trick with a self-destructive script, hmmm...

frostschutz · 12-15-2014, 12:12 PM

Hi,

Warning: This works fine for me, but I can't guarantee it will work for you. Use at your own risk!

this hack allows using either vfat or ext2/3/4 for both internal and external storage. It does not automatically format your filesystems, you have to do that yourself while the device is connected to USB.

Since this hack does not modify any of the original firmware files, there is a good chance it will survive firmware updates (but not factory resets). Otherwise, a firmware update would likely end in a brick or forced factory reset since the ext* filesystem would be considered corrupt by dosfsck.

Backup all your files (rsync -a or similar)
Install the attached KoboRoot.tgz (tested with H2O only)
mkfs.ext4 -L KOBOeReader /dev/sdx (for internal storage)
mkfs.ext4 /dev/sdy (for external storage)
Restore your backup to the newly created filesystems

Note that unlike vfat, ext* does not set the executable flag automatically, so this may break some other mods that put binaries there without explicitely setting them executable.

Regards
frostschutz

PS: I haven't tested what Calibre makes of the change; I do not use Calibre for syncing.

PPS: I made this to see if it could be done; not because it's actually useful for anything.

davidfor · 12-15-2014, 05:50 PM

Quote:

Originally Posted by frostschutz

PS: I haven't tested what Calibre makes of the change; I do not use Calibre for syncing.

Calibre's only requirement is that it can see the drive. If it is a mounted file system the OS can write to, then calibre can use it.

frostschutz · 12-15-2014, 06:05 PM

It's also a question of file permissions and ownerships.

PeterT · 12-15-2014, 08:01 PM

From what I remember if this is done the attaching via USB to a Windows machine will totally fail. Windows will NOT recognize the exported device as a validly formatted device.

Sent from my Nexus 7 using Forum Fiend v1.2.14.

diskette · 02-17-2016, 01:43 PM

a quick ls -l shows that the sd file was recently modified in the december 2015 update, though seemingly not in the january 2015 update, so it is possible that changing the default file system could require fiddling after updates every so often

maranium · 07-23-2016, 09:28 PM

Does frostschutz's installer work? I have a Kobo Glo, which I have upgraded with an internal microsd of 128 GB, with 3753 books on it (60% manga), and running firmware 2.5.1 (it has been the most stable for me, and works well for showing manga full screen). For a while now, I have had issues with covers not being generated. I have sort of cheated, and moved the LIBRARY_FULL covers into a separate directory, but even then ran out of space. Now books I rarely read have all their covers in another directory, but it makes browsing the library difficult, as it tries to generate covers not located in the image directory, and then cannot. Formatting to an ext version seems the only solution, but apart from this thread, I am not finding much info about it. Right now I am making an image of the internal SD, then going to back up just the files on the fat32 partition, before I attempt to run the installer, and reformat the partition to ext, leaning toward ext2. Doing all this with two different Windows computers, one a modified XP, one running 7. Hoping I have the ext drivers installed properly. Anyone have any experience with this/suggestions/comments?

davidfor · 07-23-2016, 11:59 PM

I can't really comment on changing to ext on user partition. I think I tried it as an experiment in the past, but as I didn't have a real need to do it or otherwise run the ext drivers under Windows, I haven't kept it up. If frostschutz's installer doesn't work, it will be because it was written a long time after the firmware version you are using

But, the problem you are having has been fixed in later firmware. In the version you are using, all cover images are stored in one directory one either the internal card, or the SD card, depending on where the book is. This causes a problem due to the fact the partition in use is formatted as FAT32 and the long file names. This means that covers for somewhere between 2000 and 3000 books can be stored.

With firmware version 2.9.0, Kobo changed to using a tree structure for the cover images and a hash based on the name to locate cover files. This solved the problem with using a FAT32 partition.

There are some other things you can do. Simply deleting all the cover images when the directory gets full will help a lot. They will be regenerated as needed, but it pushes the problem back for a while. And, unless you are scrolling through the whole library list, you will be removing covers images for books you don't see.

Another hack is to fiddle with the database and add a slash into the ImageID for your books. I did some experimenting before Kobo changed the cover image storage and the firmware will support sub-directories. The ImageID is in the database and is just the name of the cover image file. If it contains a slash, the firmware treats this as a directory hierarchy within the cover images directory. There was some discussion here about how to do this, but it was never implemented as Kobo solved the problem for us.

rtiangha · 08-05-2016, 09:43 AM

For what it's worth, the files in frostschutz's installer and instructions still work on my Glo HD running 3.19.5761. But I didn't run it; I cloned my SD card and while it was still plugged into my Linux machine, I copied his two scripts into rootfs's /sbin directory, and formatted my KOBOeReader partition into ext4 using the default options in gparted (remember, the Glo HD doesn't have an external sd card reader).

Calibre on Ubuntu wouldn't detect the device when connected by default, though. I had to mount the partition first before opening Calibre. And because of permissions (when I first mounted the partition through the file explorer, the permissions on the directory were set to root), I couldn't write anything to it either by default; I went into /media/user/ and changed owner and group to me, and I ran chmod 777 on the directory as well, just in case.

I'm using a 64GB sdcard, my device database is 600MB+ and I have over 3800 books and the file system change definitely made a difference for me. Boot up is so much faster and sending metadata to the device, getting the list of books on the device and backing up the database through Calibre is faster too, sometimes by a factor of two. Keep in mind though that this is through Linux; using an ext driver on Windows or OSX might perform worse than if it were still using FAT32.

Some things seem to lag on the device itself sometimes too, it seems, for example, tapping on something *sometimes* seems to take a smidge longer to display than it did using FAT32. That could be because of the journaling, or maybe it's just psychological (or maybe I just haven't used it long enough). Powering off takes a bit longer too (I assume to sync the journal first before unmounting and shutting down). I'll play around with it a bit more and if I'm feeling motivated, might try running ext2 or ext3 or maybe just ext4 without journaling enabled to see how it performs. But the increase in speed for me with just the file transfers and device operations between it and my Linux machine alone make me strongly consider keeping the device this way. It used to take me 10-15 minutes to sync my device and its large collection with my Linux machine through Calibre and that time was cut in half using ext4. So to me, it's worth it (and again, boot up from Power Off state is so much faster now).

That said, I wonder if it's possible to modify frostschutz's dosfsck script to either probe the partition to figure out what it's formatted with and run the proper utility, or to run e2fsck instead if dosfsck fails, so that way, it wouldn't matter to the device what file system that partition is formatted in (let the Kobo assume it's still FAT32) and will run the disk check whenever it needs to run a disk check under normal operations. Seems kind of silly to have a journaled file system if the device won't file check it every so often (assuming Kobo hard coded it to do that to the FAT32 partition once in a while), although to be fair, I don't know under what circumstances the Kobo would call dosfsck in the first place. Hopefully it would do it on regular intervals or whenever it detects issues with the file system.

12-11-2014, 02:52 PM	#37
frostschutz Linux User Posts: 2,279 Karma: 6123806 Join Date: Sep 2010 Location: Heidelberg, Germany Device: none	Even without changing any of the rcS/sd/... scripts that do the mounting, it's possible to change both the internal and external storage to ext4. I noticed that mount and dosfsck are located in /bin/, but PATH is /sbin:/usr/sbin:/bin:/usr/bin. Thus you can replace the mount and dosfsck commands with shell scripts in either /sbin/ or /usr/sbin/. Example for /sbin/mount: Code: #!/bin/busybox sh echo mount "$@" >> /mount.log exec /bin/mount "$@" So all this does is log the call and then execute the original binary in /bin/. Here's the log: Code: mount -o remount,noatime,nodiratime /dev/mmcblk0p1 / mount -t vfat -o noatime,nodiratime,shortname=mixed,utf8 /dev/mmcblk0p3 /mnt/onboard mount -t devpts devpts /dev/pts mount -r -t vfat -o noatime,nodiratime,shortname=mixed,utf8 /dev/mmcblk1p1 /mnt/sd mount -o remount,rw /mnt/sd mount -o remount,ro /mnt/sd mount -o remount,rw /mnt/sd mount -o remount,ro /mnt/sd mount -r -t vfat -o noatime,nodiratime,shortname=mixed,utf8 /dev/mmcblk1p1 /mnt/sd mount -o remount,rw /mnt/sd mount -o remount,ro /mnt/sd Note the various proc, tmpfs, etc. mounts are missing from this log, since rcS for some reason calls mount with full path for those, so the PATH trick does not catch them. If there were any such calls for the vfat, you'd have to replace /bin/mount and exec /bin/busybox mount instead. And the dosfsck log: Code: dosfsck -a -w /dev/mmcblk0p3 dosfsck -a -w /dev/mmcblk1p1 dosfsck -a -w /dev/mmcblk1p1 So you can hook into those calls and mangle the parameters any way you like. And it should even survive firmware updates. Getting those scripts onto the reader is easy enough using KoboRoot.tgz. This method seems to be lacking an uninstall facility though. My iriver Story HD was nice enough to execute a shell script for every update, but if Kobo has something like it, I haven't found it yet. Naturally you could just use the same trick with a self-destructive script, hmmm... Last edited by frostschutz; 12-11-2014 at 04:02 PM.

08-05-2016, 09:43 AM	#45
rtiangha Evangelist Posts: 496 Karma: 356531 Join Date: Jul 2016 Location: 'burta, Canada Device: Kobo Glo HD	For what it's worth, the files in frostschutz's installer and instructions still work on my Glo HD running 3.19.5761. But I didn't run it; I cloned my SD card and while it was still plugged into my Linux machine, I copied his two scripts into rootfs's /sbin directory, and formatted my KOBOeReader partition into ext4 using the default options in gparted (remember, the Glo HD doesn't have an external sd card reader). Calibre on Ubuntu wouldn't detect the device when connected by default, though. I had to mount the partition first before opening Calibre. And because of permissions (when I first mounted the partition through the file explorer, the permissions on the directory were set to root), I couldn't write anything to it either by default; I went into /media/user/ and changed owner and group to me, and I ran chmod 777 on the directory as well, just in case. I'm using a 64GB sdcard, my device database is 600MB+ and I have over 3800 books and the file system change definitely made a difference for me. Boot up is so much faster and sending metadata to the device, getting the list of books on the device and backing up the database through Calibre is faster too, sometimes by a factor of two. Keep in mind though that this is through Linux; using an ext driver on Windows or OSX might perform worse than if it were still using FAT32. Some things seem to lag on the device itself sometimes too, it seems, for example, tapping on something sometimes seems to take a smidge longer to display than it did using FAT32. That could be because of the journaling, or maybe it's just psychological (or maybe I just haven't used it long enough). Powering off takes a bit longer too (I assume to sync the journal first before unmounting and shutting down). I'll play around with it a bit more and if I'm feeling motivated, might try running ext2 or ext3 or maybe just ext4 without journaling enabled to see how it performs. But the increase in speed for me with just the file transfers and device operations between it and my Linux machine alone make me strongly consider keeping the device this way. It used to take me 10-15 minutes to sync my device and its large collection with my Linux machine through Calibre and that time was cut in half using ext4. So to me, it's worth it (and again, boot up from Power Off state is so much faster now). That said, I wonder if it's possible to modify frostschutz's dosfsck script to either probe the partition to figure out what it's formatted with and run the proper utility, or to run e2fsck instead if dosfsck fails, so that way, it wouldn't matter to the device what file system that partition is formatted in (let the Kobo assume it's still FAT32) and will run the disk check whenever it needs to run a disk check under normal operations. Seems kind of silly to have a journaled file system if the device won't file check it every so often (assuming Kobo hard coded it to do that to the FAT32 partition once in a while), although to be fair, I don't know under what circumstances the Kobo would call dosfsck in the first place. Hopefully it would do it on regular intervals or whenever it detects issues with the file system. Last edited by rtiangha; 08-05-2016 at 10:24 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
kindle 3 hardware hacking (micro usb & sd card)	damaru	Amazon Kindle	5	03-16-2012 08:19 AM
360 Pocketbook 360 and SD card file system error	readerreader	PocketBook	2	08-29-2011 12:10 AM
K2 system file access	dwallbaum	Amazon Kindle	2	09-08-2010 04:32 PM
File System Information	Adam B.	iRex	0	10-29-2008 08:46 AM
how to clean more disk space in root file system to upgrade system	chinaet	iRex	1	12-18-2006 03:54 PM

06-07-2013, 12:18 AM	#32
flowoeB E-Reader Sammler Posts: 134 Karma: 1000 Join Date: Feb 2009 Location: Austria Device: plenty...	there are no short filenames under ext3 - so why would you use "-o shortname=mixed" ? I just deleted the argument.

06-07-2013, 08:43 AM	#33
auspex Groupie Posts: 199 Karma: 1071756 Join Date: Sep 2012 Location: Nova Scotia Device: Kobo Aura, Nexus 5x	Like I said - you can't "just do a mount without the vfat argument".

06-21-2013, 01:27 PM	#34
flowoeB E-Reader Sammler Posts: 134 Karma: 1000 Join Date: Feb 2009 Location: Austria Device: plenty...	hm, tried to mount an external ext3 sd on my aura today and it won't mount - I see it, when I plug the reader into my macs USB and can write books to it - but I can't access it with the reader itself... a messagebox is popping up that tells me to format the sd with fat32... the USB and SD files are modified as described above and I'm running 2.6.1b...

06-22-2013, 12:24 AM	#36
flowoeB E-Reader Sammler Posts: 134 Karma: 1000 Join Date: Feb 2009 Location: Austria Device: plenty...	didn' try it. if the external does not work I don't think the internal will... I'll take a deeper look into this as soon as i have more time...

12-15-2014, 06:05 PM	#40
frostschutz Linux User Posts: 2,279 Karma: 6123806 Join Date: Sep 2010 Location: Heidelberg, Germany Device: none	It's also a question of file permissions and ownerships.

12-15-2014, 08:01 PM	#41
PeterT Grand Sorcerer Posts: 12,166 Karma: 73448616 Join Date: Nov 2007 Location: Toronto Device: Nexus 7, Clara, Touch, Tolino EPOS	From what I remember if this is done the attaching via USB to a Windows machine will totally fail. Windows will NOT recognize the exported device as a validly formatted device. Sent from my Nexus 7 using Forum Fiend v1.2.14.

02-17-2016, 01:43 PM	#42
diskette Member Posts: 11 Karma: 450 Join Date: Jan 2016 Device: kobo touch	a quick ls -l shows that the sd file was recently modified in the december 2015 update, though seemingly not in the january 2015 update, so it is possible that changing the default file system could require fiddling after updates every so often

07-23-2016, 09:28 PM	#43
maranium Member Posts: 14 Karma: 10 Join Date: Jan 2012 Device: Kobo Touch, Glo, Mini	Does frostschutz's installer work? I have a Kobo Glo, which I have upgraded with an internal microsd of 128 GB, with 3753 books on it (60% manga), and running firmware 2.5.1 (it has been the most stable for me, and works well for showing manga full screen). For a while now, I have had issues with covers not being generated. I have sort of cheated, and moved the LIBRARY_FULL covers into a separate directory, but even then ran out of space. Now books I rarely read have all their covers in another directory, but it makes browsing the library difficult, as it tries to generate covers not located in the image directory, and then cannot. Formatting to an ext version seems the only solution, but apart from this thread, I am not finding much info about it. Right now I am making an image of the internal SD, then going to back up just the files on the fat32 partition, before I attempt to run the installer, and reformat the partition to ext, leaning toward ext2. Doing all this with two different Windows computers, one a modified XP, one running 7. Hoping I have the ext drivers installed properly. Anyone have any experience with this/suggestions/comments?

07-23-2016, 11:59 PM	#44
davidfor Grand Sorcerer Posts: 24,907 Karma: 47303748 Join Date: Jul 2011 Location: Sydney, Australia Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos	I can't really comment on changing to ext on user partition. I think I tried it as an experiment in the past, but as I didn't have a real need to do it or otherwise run the ext drivers under Windows, I haven't kept it up. If frostschutz's installer doesn't work, it will be because it was written a long time after the firmware version you are using But, the problem you are having has been fixed in later firmware. In the version you are using, all cover images are stored in one directory one either the internal card, or the SD card, depending on where the book is. This causes a problem due to the fact the partition in use is formatted as FAT32 and the long file names. This means that covers for somewhere between 2000 and 3000 books can be stored. With firmware version 2.9.0, Kobo changed to using a tree structure for the cover images and a hash based on the name to locate cover files. This solved the problem with using a FAT32 partition. There are some other things you can do. Simply deleting all the cover images when the directory gets full will help a lot. They will be regenerated as needed, but it pushes the problem back for a while. And, unless you are scrolling through the whole library list, you will be removing covers images for books you don't see. Another hack is to fiddle with the database and add a slash into the ImageID for your books. I did some experimenting before Kobo changed the cover image storage and the firmware will support sub-directories. The ImageID is in the database and is just the name of the cover image file. If it contains a slash, the firmware treats this as a directory hierarchy within the cover images directory. There was some discussion here about how to do this, but it was never implemented as Kobo solved the problem for us.

Advert

Advert