Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Readers > More E-Book Readers > iRex > iRex Developer's Corner

Notices

Reply
 
Thread Tools Search this Thread
Old 06-23-2011, 09:23 AM   #16
Iņigo
Guru
Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.
 
Posts: 730
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, iRex DR800S. K4NT. Kobo Aura, Aura One, Libra 2.
Quote:
Originally Posted by Mackx View Post
I am not sure how many files can be in a FAT directory, should the System/_covers/ contains a directory structure?
For speed I'd suggest a structure like:
Code:
SD_card/System/.covers
    medium/
        00
        01
        ..
        FF/
            id.png
            other_id.png
            ...
    small/
        [similar to medium]
That's the way some programs work to avoid file system limitations.
If 256 subdirectories are not enough we could use a bigger number.
In order to make file loading as fast as possible we have to think in a map function that equally distributes covers in the 256 subdirectories. I can think of "id mod_of_256" (id % 256). Btw, does % operator exist in C?
Iņigo is offline   Reply With Quote
Old 06-23-2011, 09:41 AM   #17
Iņigo
Guru
Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.
 
Posts: 730
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, iRex DR800S. K4NT. Kobo Aura, Aura One, Libra 2.
Quote:
Originally Posted by yuri_b View Post
Hi

I remember how long it tooks to view current directory in Norton Commander when number of files >500.
With number of books > 500 it's not good way to use FAT file system and trust me data base will search for file faster then file system, just because this is data base's job.

Another aspect is a limitation of FAT32:
a) The maximum possible number of clusters on a volume using the FAT32 file system is 268,435,445 http://support.microsoft.com/kb/184006
b) Cluster size for file on FAT32 for 32GB: 16 KB, so every thumbnail will cost you no less then 16K. So 150000*4 files will cost no less then 9.6 GB on FAT32.
http://support.microsoft.com/kb/192322
c) Every entry occupied 4 bytes in FAT32 table (up to 8 MB). http://www.pcguide.com/ref/hdd/file/partFAT32-c.html


Sincerely
Yura

P.S. A program, that I wrote for home use, a film catalog, uses sqlite for storing all data for a film, including several pictures, db file today about 60MB in size and it works as a sharm (Win32 )
Sorry, I don't want to sound rough, and my English is far from being perfect too, so don't take this as a personal attack.
But I think that 150k files is far from DR devices capacity, we can't design and write code to support that huge numbers. Even I don't know any other user than you with more than 5k files.

I'm not a professional developer as my daily job, it's only a hobby for me, but I've been writing code since almost 30 years ago.
One of my current (halted) projects is a book manager. There I was considering to store covers on the db (SQLite) or not. After some tests I finally I preferred to keep them in the file system. I have over 55k files and my machine is a linux AMD64x2 with 6GB RAM, file system is ext3, much powerful than the DRs.

And here, on the DRs, maybe SQLite is enough to handle large number of entries, but both CPU and RAM are really not as powerful as we would need.

I think in cases like yours is better to use a plain file system browser, not a database-based one.

Iņigo
Iņigo is offline   Reply With Quote
Old 06-23-2011, 09:58 AM   #18
yuri_b
Connoisseur
yuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enough
 
Posts: 71
Karma: 592
Join Date: Aug 2010
Device: irex dr800sg DR1000S
Quote:
Originally Posted by Iņigo View Post
Sorry, I don't want to sound rough, and my English is far from being perfect too, so don't take this as a personal attack.
It's OK

150K I took from Mack mail:
Quote:
I am also busy working on a mechanism to limit the number of files in global.db to help with the 150 000 files problem of Viasheslav.
As I already sad in SQL missed a needed indexes, that's way it works so long.
And add indexes is job for 5 minutes, a much simpler then change a way all system works Lets checks maybe it would be enough?

Secondly: Limit cbt to 2-3 page also much easy then change all

I'm not saying "Don't do this.". I say lets check simpler ways to fix our problem. Our time not flexible and the less we will spend for each problem then more we will do.

Believe me I also would like to store on 1 SD card all my library, and I have a lot of books, the problem you know

Sincerely
Yura
yuri_b is offline   Reply With Quote
Old 06-23-2011, 10:34 AM   #19
Iņigo
Guru
Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.
 
Posts: 730
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, iRex DR800S. K4NT. Kobo Aura, Aura One, Libra 2.
Quote:
Originally Posted by yuri_b View Post
It's OK


Quote:
Originally Posted by yuri_b View Post
As I already sad in SQL missed a needed indexes, that's way it works so long.
And add indexes is job for 5 minutes, a much simpler then change a way all system works Lets checks maybe it would be enough?
Ok, we agree

I'm not an expert on DB either, but I think the new indexes can be added now and force a reindex of the db.
Could you post the concrete modifications to the DB?
I can run them on my computer after copying global.db from the DR800.
So we can test the results.

As a second step I will investigate about moving covers to outside the DB.
Iņigo is offline   Reply With Quote
Old 06-23-2011, 12:10 PM   #20
yuri_b
Connoisseur
yuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enough
 
Posts: 71
Karma: 592
Join Date: Aug 2010
Device: irex dr800sg DR1000S
Quote:
Originally Posted by Iņigo View Post


I'm not an expert on DB either, but I think the new indexes can be added now and force a reindex of the db.
Could you post the concrete modifications to the DB?
I can run them on my computer after copying global.db from the DR800.
So we can test the results.

As a second step I will investigate about moving covers to outside the DB.
What I did:

1) copy global.db in my PC
2) I have sqlite3.exe already so I just execute those commands:
3) sqlite3.exe global.db
4) in opened sqlite shell enter those commands:
create index filename_index on file_metadata(filename,tag);
create index title_index on file_metadata(title,tag);
create index author_index on file_metadata(author,tag);
create index type_index on file_metadata(file_type,tag);
create index size_index on file_metadata(file_size,tag);
create index date_added_index on file_metadata(file_time_added,tag);
create index time_viewed_index on file_metadata(file_time_lastread,tag);

5) enter: .quit
to exit
6) copy global.db back to device.

Right now I don't have all books on my IREX, so find backup and restore those ~3000 files SD will take a while


Moreover, as I sad, in case missed global.db IREX copy one empty from some place, so we can just generate one empty with all indexes, and place it as template on this location.

Sincerely
Yura
yuri_b is offline   Reply With Quote
Old 06-23-2011, 12:51 PM   #21
Iņigo
Guru
Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.
 
Posts: 730
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, iRex DR800S. K4NT. Kobo Aura, Aura One, Libra 2.
Quote:
Originally Posted by yuri_b View Post
What I did:

1) copy global.db in my PC
2) I have sqlite3.exe already so I just execute those commands:
3) sqlite3.exe global.db
4) in opened sqlite shell enter those commands:
create index filename_index on file_metadata(filename,tag);
create index title_index on file_metadata(title,tag);
create index author_index on file_metadata(author,tag);
create index type_index on file_metadata(file_type,tag);
create index size_index on file_metadata(file_size,tag);
create index date_added_index on file_metadata(file_time_added,tag);
create index time_viewed_index on file_metadata(file_time_lastread,tag);

5) enter: .quit
to exit
6) copy global.db back to device.

Right now I don't have all books on my IREX, so find backup and restore those ~3000 files SD will take a while


Moreover, as I sad, in case missed global.db IREX copy one empty from some place, so we can just generate one empty with all indexes, and place it as template on this location.

Sincerely
Yura
Tested on my ~4k books global.db. Same speed, maybe even a bit slower
Iņigo is offline   Reply With Quote
Old 06-23-2011, 12:56 PM   #22
yuri_b
Connoisseur
yuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enough
 
Posts: 71
Karma: 592
Join Date: Aug 2010
Device: irex dr800sg DR1000S
What speed?
Speed of indexing? or speed of opening page in file view?
Indexing should speed up browsing dbase
yuri_b is offline   Reply With Quote
Old 06-23-2011, 01:08 PM   #23
Iņigo
Guru
Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.
 
Posts: 730
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, iRex DR800S. K4NT. Kobo Aura, Aura One, Libra 2.
Quote:
Originally Posted by yuri_b View Post
What speed?
Speed of indexing? or speed of opening page in file view?
Indexing should speed up browsing dbase
Sorry. I meant opening Book view.
Iņigo is offline   Reply With Quote
Old 06-23-2011, 01:13 PM   #24
yuri_b
Connoisseur
yuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enough
 
Posts: 71
Karma: 592
Join Date: Aug 2010
Device: irex dr800sg DR1000S
If the speed not increased I see 2 explanations:
1) sql not uses this index, we created wrong index
2) there is additional place that slows all down.
yuri_b is offline   Reply With Quote
Old 06-23-2011, 04:21 PM   #25
Mackx
Guru
Mackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to behold
 
Posts: 999
Karma: 19985
Join Date: Dec 2008
Location: Netherlands
Device: iRex DR1000S
Looking at libermetadb I see the following is used (when sorting on filename):
Code:
ORDER BY
    sort_priority DESC,
    (CASE WHEN filename IS NOT NULL THEN 0 ELSE 1 END),
    filename COLLATE IREX_CASE_INSENSITIVE,
    directory_path COLLATE IREX_CASE_INSENSITIVE
(sort_priority is used to get folder in front of files.) I am no database expert, so its just for information.
Mackx is offline   Reply With Quote
Old 06-23-2011, 04:27 PM   #26
Iņigo
Guru
Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.
 
Posts: 730
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, iRex DR800S. K4NT. Kobo Aura, Aura One, Libra 2.
no words here
Attached Thumbnails
Click image for larger version

Name:	test1.png
Views:	535
Size:	58.3 KB
ID:	73238  
Iņigo is offline   Reply With Quote
Old 06-23-2011, 04:55 PM   #27
yuri_b
Connoisseur
yuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enough
 
Posts: 71
Karma: 592
Join Date: Aug 2010
Device: irex dr800sg DR1000S


Sicerely
Yura
yuri_b is offline   Reply With Quote
Old 06-23-2011, 05:02 PM   #28
yuri_b
Connoisseur
yuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enough
 
Posts: 71
Karma: 592
Join Date: Aug 2010
Device: irex dr800sg DR1000S
Quote:
Originally Posted by Mackx View Post
Looking at libermetadb I see the following is used (when sorting on filename):
This uses index filename+directory. IMHO.
yuri_b is offline   Reply With Quote
Old 06-24-2011, 04:47 AM   #29
Iņigo
Guru
Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.
 
Posts: 730
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, iRex DR800S. K4NT. Kobo Aura, Aura One, Libra 2.
ok, it was a bug or incomplete GtkIconView functionality in erGtkIconView implementation.
Now, after 7 hours of hard work, it looks nicer ;-)
Attached Thumbnails
Click image for larger version

Name:	test2.png
Views:	555
Size:	99.9 KB
ID:	73259  
Iņigo is offline   Reply With Quote
Old 06-24-2011, 06:31 AM   #30
Viacheslav
Connoisseur
Viacheslav doesn't litterViacheslav doesn't litter
 
Posts: 59
Karma: 136
Join Date: Dec 2008
Device: iRex DR1000
I was following this thread and it was confirmed several times that FAT slows down file browsing and is suboptimal. What about switching to some other Linux FS e.g. ReiserFS? It was tooted as being fast with many small files. Besides, an Ext* formatted new empty HDD has noticeably less free space than the same HDD with Reiser, because Ext needs more space for system data.

Last edited by Viacheslav; 06-24-2011 at 06:33 AM.
Viacheslav is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Mobiperl Mobiperl - Perl tools for handling MobiPocket files tompe Kindle Formats 890 01-17-2021 06:16 PM
Large number of books on memory card pwalker8 Sony Reader 8 03-24-2009 02:20 PM
huge pdf files on iliad yolle iRex 7 08-23-2008 10:59 AM
Fictionwise Handling Fee Against Credit Card Terms of Service? Gideon News 9 08-15-2008 02:13 PM
Huge RTF files after conversion TheMadBrewer Sony Reader 2 01-01-2007 02:55 AM


All times are GMT -4. The time now is 02:24 AM.


MobileRead.com is a privately owned, operated and funded community.