MobileRead Forums - View Single Post - Handling a huge number of files on the SD Card

Iñigo · 06-25-2011, 09:27 PM

Hi again,

Today I've tested the patch on my DR800, so let's report these good results!

Steps I've done:
- disable automatic generation of thumbnails while indexing
- extracted covers from global.db to the file system, removed all images from db and finally vacuum db [1]
- copied the ctb built with my patch from message #44 and reboot the ereader

* BEFORE
Entries in db: 4k
Entries in db with thumbnails: 2.2k
DB Size: 29.4 MB
File system space for covers: 0
Time to open Book view after reboot (cold): 30 sec
Time to open Book view (warm): 27 sec
Time to open Recent* view after reboot (cold), 50 items: 9 sec
Time to open Recent* view (warm), 50 items: 7 sec

* AFTER THE PATCHED CTB
Entries in db: 4k
Entries in db with thumbnails: 0
DB Size: 1 MB
File system space for covers: 36.2 MB = 11 MB (small size) + 25.2 (medium size)
Time to open Book view after reboot (cold): 4-5 sec
Time to open Book view (warm): 4 sec
Time to open Recent* view after reboot (cold), 50 items: 3 sec
Time to open Recent* view (warm), 50 items: 2-3 sec

[1] the vala-based DR binary program I've written has a bug and can't be used now. I'm waiting to get a fix for a sqlite bug I've discovered.
So I've written a quick hack in python, run the script on my computer and then moved manually the extracted covers to the DR

Things to do before considering this finished:
- wait for a fix for the vala program, so we could migrate cover on the DR
- settings icons are missing
. they were stored in global.db
. which program added them?
- change other parts of the DR firmware to use file system instead of DB:
. mdbindex
. metadata extracted from UDS
- build a pretty package to the DR

Final comments:
- this is a really needed hack to be implemented in the DR, unless you have less than 1000 books or you don't use covers
- we only use ~25% more of disk space, i.e. ~7MB more than the global.db for 2000+ books, neligible considering the advantages
- combined with yuri_b's db handling improvements we could support much more books in the DR, not 150k but maybe 25k or even more

Iñigo

06-25-2011, 09:27 PM	#51
Iñigo Guru Posts: 730 Karma: 72743 Join Date: Feb 2008 Location: Here or there Device: iRex iLiad, iRex DR800S. K4NT. Kobo Aura, Aura One, Libra 2.	CTB: covers in file system instead of global.db (cont) Hi again, Today I've tested the patch on my DR800, so let's report these good results! Steps I've done: - disable automatic generation of thumbnails while indexing - extracted covers from global.db to the file system, removed all images from db and finally vacuum db [1] - copied the ctb built with my patch from message #44 and reboot the ereader * BEFORE Entries in db: 4k Entries in db with thumbnails: 2.2k DB Size: 29.4 MB File system space for covers: 0 Time to open Book view after reboot (cold): 30 sec Time to open Book view (warm): 27 sec Time to open Recent* view after reboot (cold), 50 items: 9 sec Time to open Recent* view (warm), 50 items: 7 sec * AFTER THE PATCHED CTB Entries in db: 4k Entries in db with thumbnails: 0 DB Size: 1 MB File system space for covers: 36.2 MB = 11 MB (small size) + 25.2 (medium size) Time to open Book view after reboot (cold): 4-5 sec Time to open Book view (warm): 4 sec Time to open Recent* view after reboot (cold), 50 items: 3 sec Time to open Recent* view (warm), 50 items: 2-3 sec [1] the vala-based DR binary program I've written has a bug and can't be used now. I'm waiting to get a fix for a sqlite bug I've discovered. So I've written a quick hack in python, run the script on my computer and then moved manually the extracted covers to the DR Things to do before considering this finished: - wait for a fix for the vala program, so we could migrate cover on the DR - settings icons are missing . they were stored in global.db . which program added them? - change other parts of the DR firmware to use file system instead of DB: . mdbindex . metadata extracted from UDS - build a pretty package to the DR Final comments: - this is a really needed hack to be implemented in the DR, unless you have less than 1000 books or you don't use covers - we only use ~25% more of disk space, i.e. ~7MB more than the global.db for 2000+ books, neligible considering the advantages - combined with yuri_b's db handling improvements we could support much more books in the DR, not 150k but maybe 25k or even more Iñigo Last edited by Iñigo; 06-25-2011 at 09:40 PM.