I don't know which kernel the Libra is using but for the original H2O it's 2.6.x kernel, not sure if 11 years old btrfs is a good idea ;-)
If Libra is same as H2O, then it's possible with hacks (but not really worth it) to use ext2/3/4. That's the only thing it supports (and uses for root and recovery filesystem) in addition to the regular FAT32.
lzo compression won't do anything for you considering most ebook formats already have their own compression, anyway. Books can't be compressed twice.
I'd leave the kernel and its modules well alone. Without internal SD card slot, the risk of bricking is too high to even make the attempt.
----
On H2O (sorry, don't have Libra)
Code:
[root@(none) ~]# uname -a
Linux (none) 2.6.35.3-850-gbc67621+ #3032 PREEMPT Mon Jan 9 13:37:40 CST 2017 armv7l GNU/Linux
[root@(none) ~]# cat /proc/filesystems
nodev sysfs
nodev rootfs
nodev bdev
nodev proc
nodev tmpfs
nodev sockfs
nodev pipefs
nodev anon_inodefs
nodev devpts
ext2
ext3
ext4
nodev ramfs
vfat
msdos