View Single Post
Old 06-23-2011, 07:46 AM   #14
Mackx
Guru
Mackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to behold
 
Posts: 998
Karma: 19985
Join Date: Dec 2008
Location: Netherlands
Device: iRex DR1000S
Quote:
Originally Posted by Iņigo View Post
Today I've did some new tests removing global.db and rebuilding without images.
global.db is less than 2 MB.
Books view spends no more than 2 or 3 secs to show the files.
Similar time for Recently Added view.


These are some of my conclusions:
- main culprit of the slowness is the quantify of images in the DB
- when indexing with covers there is a moment where mdbindex stops working. I've tested removing the files the indexer was analyzing at that moment but it's the same. So I presume the problem is the lack of free memory. Would enabling a swap partition/file improve something?
- does DR DB engine cache some contents in memory?
I also had the impression that the thumbnails were the problem and more specific mainly the memory usage. The DR has around 50MB free memory after it started in Home (on my DR1000). When showing Books-view, information of all books, including thumbnails, are loaded into memory.
Quote:
Originally Posted by Iņigo View Post
A good but infeasible solution would be to use 2 different db files, one for metadata and another one for images.
Or better, not using a db for images, but a hidden directory with the cover files: System/_covers/medium/id.png, System/_covers/small/id.png.
This solution doesn't seem difficult. I'll study it.
- change code to avoid writing images to the DB when closing UDS, and write to a file
- CTB would only load the covers needed for current page

This would reduce the overall DB size in memory and the introduced slowness due to images file loading should not be noticeable...
This is indeed a good idea, I was also thinking of somehow not loading the thumbnails. Putting them in a separate files would indeed be a good solution. I am not sure how many files can be in a FAT directory, should the System/_covers/ contains a directory structure?
It is probably not that difficult to change ermetadb to store thumbnails in separate files i.s.o. in global.db. And ctb to read thumbnails from file before showing.
One problem would also be how to delete the thumbnails when the documents are removed. This could probably be done by mdbindex when removing entries from global.db.

I have some time right now, so if I can help please tell me.

I am also busy working on a mechanism to limit the number of files in global.db to help with the 150 000 files problem of Viasheslav. I think that not adding the thumbnails is not enough for that many files. Extrapolating from the 4k->2MB that would give 150k->75MB of data without thumbnails.
Mackx is online now   Reply With Quote