MobileRead Forums - View Single Post - Handling a huge number of files on the SD Card

Mackx · 06-20-2011, 05:50 AM

I have created an experiment where in SDCard mode files are directly extracted from SDCard and not from global.db. The good thing is that basic browsing seems to works OK (although a lot still needs to be done, like sorting). On the emulator I created some extra folders with files to make sure that global.db was not used. Opening a 'old' file also worked OK, however opening a 'new'-file (that was not in global.db) caused UDS to crash and show some error messages. Opening non-UDS files (xournal, notes) worked OK.
This problem will also occur when using a new file browser.
I was thinking of adding the files on-the-fly to global.db. A pruning mechanism (based on file_time_lastread) could then be applied to limit the size of global.db (to a few thousand entries?). The advantage would be that all views, search and current_page would still work on the most (recently) used documents.
There could even by an automatic mechanism that puts mdbindex and SD-Card view in a special mode when the SD-card contains more then a <threshold> number of files?

There are also some upgrading (R1.7.1 -> R2.0) issues that need to be addressed. The mdbindex program is also used to convert the content of the databases (global.db but also all metadata.db) to the correct version. Removing unwanted data, moving data between tables and adding new data.
Upgrading is always done with a official version of the firmware so one has to limit the number of files that are on the SDCard. Or is it easy to modify the original installation image with a modified mdbindex?

06-20-2011, 05:50 AM	#4
Mackx Guru Posts: 999 Karma: 19985 Join Date: Dec 2008 Location: Netherlands Device: iRex DR1000S	I have created an experiment where in SDCard mode files are directly extracted from SDCard and not from global.db. The good thing is that basic browsing seems to works OK (although a lot still needs to be done, like sorting). On the emulator I created some extra folders with files to make sure that global.db was not used. Opening a 'old' file also worked OK, however opening a 'new'-file (that was not in global.db) caused UDS to crash and show some error messages. Opening non-UDS files (xournal, notes) worked OK. This problem will also occur when using a new file browser. I was thinking of adding the files on-the-fly to global.db. A pruning mechanism (based on file_time_lastread) could then be applied to limit the size of global.db (to a few thousand entries?). The advantage would be that all views, search and current_page would still work on the most (recently) used documents. There could even by an automatic mechanism that puts mdbindex and SD-Card view in a special mode when the SD-card contains more then a <threshold> number of files? There are also some upgrading (R1.7.1 -> R2.0) issues that need to be addressed. The mdbindex program is also used to convert the content of the databases (global.db but also all metadata.db) to the correct version. Removing unwanted data, moving data between tables and adding new data. Upgrading is always done with a official version of the firmware so one has to limit the number of files that are on the SDCard. Or is it easy to modify the original installation image with a modified mdbindex?