Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Readers > More E-Book Readers > iRex > iRex Developer's Corner

Notices

Reply
 
Thread Tools Search this Thread
Old 06-24-2011, 07:53 AM   #31
rvs
Groupie
rvs has a complete set of Star Wars action figures.rvs has a complete set of Star Wars action figures.rvs has a complete set of Star Wars action figures.rvs has a complete set of Star Wars action figures.
 
Posts: 181
Karma: 364
Join Date: Oct 2009
Device: DR1000s, Hanlin V5
Quote:
Originally Posted by Viacheslav View Post
I was following this thread and it was confirmed several times that FAT slows down file browsing and is suboptimal. What about switching to some other Linux FS e.g. ReiserFS? It was tooted as being fast with many small files. Besides, an Ext* formatted new empty HDD has noticeably less free space than the same HDD with Reiser, because Ext needs more space for system data.
This is perhaps an interesting article on this question. According to the article ext4 is probably the fastest.

I don't think the DR supports ext4 right now, but perhaps this could be added? The SD-card would wear down faster though because of the journaling.
rvs is offline   Reply With Quote
Old 06-24-2011, 08:26 AM   #32
yuri_b
Connoisseur
yuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enough
 
Posts: 71
Karma: 592
Join Date: Aug 2010
Device: irex dr800sg DR1000S
I did some analyse for queries:
Actual query is:
SELECT filename, directory_path, is_directory, file_type, file_size, file_time_modified, title, author, thumb_data_small FROM file_metadata AS m LEFT JOIN thumbnails AS t ON m.file_id = t.file_id WHERE tag = 'book' ORDER BY sort_priority DESC, (CASE WHEN title IS NOT NULL THEN 0 ELSE 1 END), title COLLATE IREX_CASE_INSENSITIVE ASC, filename COLLATE IREX_CASE_INSENSITIVE ASC, directory_path COLLATE IREX_CASE_INSENSITIVE ASC;


result of SQL analise of this query is:

SEARCH TABLE file_metadata AS m USING AUTOMATIC COVERING INDEX (tag=?)
SEARCH TABLE thumbnails AS t USING INTEGER PRIMARY KEY (rowid=?) (~1 rows)
USE TEMP B-TREE FOR ORDER BY


What I say?
Two search was use not direct indexing and building temporary tree in memory or on disk.
Very bad.

BTW the same for indexed table . Indexes that I added was wrong !!!
yuri_b is offline   Reply With Quote
Old 06-24-2011, 08:54 AM   #33
Iņigo
Guru
Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.
 
Posts: 730
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, iRex DR800S. K4NT. Kobo Aura, Aura One, Libra 2.
Quote:
Originally Posted by Viacheslav View Post
I was following this thread and it was confirmed several times that FAT slows down file browsing and is suboptimal. What about switching to some other Linux FS e.g. ReiserFS? It was tooted as being fast with many small files. Besides, an Ext* formatted new empty HDD has noticeably less free space than the same HDD with Reiser, because Ext needs more space for system data.
reiserfs is not supported by DR linux kernel, your best bet should be ext2 or maybe ext3.
Code:
root@qemuarm:/media/mmcblk0p1# cat /etc/filesystems 
minix
fat
vfat
ext2
ext3
In my tests an 4GB ext3 formatted sd card has ~130MB less space available than with vfat.
Iņigo is offline   Reply With Quote
Old 06-24-2011, 09:31 AM   #34
rvs
Groupie
rvs has a complete set of Star Wars action figures.rvs has a complete set of Star Wars action figures.rvs has a complete set of Star Wars action figures.rvs has a complete set of Star Wars action figures.
 
Posts: 181
Karma: 364
Join Date: Oct 2009
Device: DR1000s, Hanlin V5
Quote:
Originally Posted by Iņigo View Post
reiserfs is not supported by DR linux kernel, your best bet should be ext2 or maybe ext3.
ext2 and ext3 aren't supported either (at the moment)...
rvs is offline   Reply With Quote
Old 06-24-2011, 10:10 AM   #35
Iņigo
Guru
Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.
 
Posts: 730
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, iRex DR800S. K4NT. Kobo Aura, Aura One, Libra 2.
Quote:
Originally Posted by rvs View Post
ext2 and ext3 aren't supported either (at the moment)...
aren't DR internal partitions ext2 file systems?

Code:
root@qemuarm:/media/mmcblk0p1# mount 
rootfs on / type rootfs (rw)
/dev/root on / type ext2 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
/dev/root on /dev/.static/dev type ext2 (rw)
tmpfs on /dev type tmpfs (rw)
devpts on /dev/pts type devpts (rw)
tmpfs on /var type tmpfs (rw)
tmpfs on /media/ram type tmpfs (rw)
Also if you take a look at /etc/filesystems, there appear the file systems supported by the kernel.

Iņigo
Iņigo is offline   Reply With Quote
Old 06-24-2011, 12:02 PM   #36
rvs
Groupie
rvs has a complete set of Star Wars action figures.rvs has a complete set of Star Wars action figures.rvs has a complete set of Star Wars action figures.rvs has a complete set of Star Wars action figures.
 
Posts: 181
Karma: 364
Join Date: Oct 2009
Device: DR1000s, Hanlin V5
The qemu-image uses ext2, but the DR uses JFFS2 for the internal filesystem.

If you look at the kernel config file, you can also see support for ext2/3/4 is not set. The qemu uses a different kernel than the DR.
rvs is offline   Reply With Quote
Old 06-24-2011, 12:04 PM   #37
Iņigo
Guru
Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.
 
Posts: 730
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, iRex DR800S. K4NT. Kobo Aura, Aura One, Libra 2.
Quote:
Originally Posted by rvs View Post
The qemu-image uses ext2, but the DR uses JFFS2 for the internal filesystem.

If you look at the kernel config file, you can also see support for ext2/3/4 is not set.
Didn't know it. Thanks for the correction.
Iņigo is offline   Reply With Quote
Old 06-24-2011, 01:34 PM   #38
Iņigo
Guru
Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.
 
Posts: 730
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, iRex DR800S. K4NT. Kobo Aura, Aura One, Libra 2.
It's working

There is a lot of work to be done yet, but it works.
I'm including some screenshots and some code (patch against r49 of https://bitbucket.org/inigoserna/dr800, remember to do "hg udpdate -C dr800+ -r49" after cloning the repository).

Could anyone check for memory leaks please? Lot of time since I used to code in C...

CTB: covers in file system instead of global.db
* some comments about the code
- the basic idea is to keep global.db free of book covers, they will be in the file system SD_card/System/.icons/medium/XX/asdfgdgdfh.png
- the original method store all the pixbufs (images) for current view loaded from the db in the model, here we get from file system on demand
- it should be a bit slower than original method in case of few books in the view, but this method use less memory, the difference is huge with lots of books so then it should be much faster too
- id is not stored in the model, so we can't use id % 256 to distribute the covers over the covers root directory
I'm using the first 2 chars of the md5_hex_digest on the fullpath of the book
- erGtkIconView needs to be patched in order to get the images as needed on runtime instead of retrieving from db (look previous screenshots in this thread)
- it handles "system" icons perfectly except a few (see below) => screenshots: test10, test11
- IT WORKS!!! => screenshot: test13: image showing the billiard ball is the presumed cover loaded from the file system

TODO:
* this PoC only works with iconview by now, it needs some code for contentview as well => very easy
* "fix" old code which uses model store
- remove pixbuf column and references to it
- don't query db for thumbnails
* remaining special icons not managed by now
- settings apps => screenshots: test12
- shorcuts?
* we use a cache, a hash table
- i think it stores some entries repeated -> glib's g_str_equal bug?
- clean contents when changedir, change_view
- destroy at end
* write a tool (in vala) to move images from db to fs, remove them from db and do a vacuum
* update other programs to copy new icons to fs instead of db
- mdbindex
- cover from uds
* test in the real DR with lots of books and compare

Kind regard,
Iņigo
Attached Thumbnails
Click image for larger version

Name:	test10.png
Views:	516
Size:	101.2 KB
ID:	73277   Click image for larger version

Name:	test11.png
Views:	499
Size:	39.0 KB
ID:	73278   Click image for larger version

Name:	test12.png
Views:	498
Size:	65.6 KB
ID:	73279   Click image for larger version

Name:	test13.png
Views:	503
Size:	60.6 KB
ID:	73280  
Attached Files
File Type: gz covers_in_fs.patch.gz (9.1 KB, 430 views)
Iņigo is offline   Reply With Quote
Old 06-24-2011, 03:13 PM   #39
Mackx
Guru
Mackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to behold
 
Posts: 999
Karma: 19985
Join Date: Dec 2008
Location: Netherlands
Device: iRex DR1000S
Hi Iņigo,

Why didn't you change filemodel.c: load_items_in_model()? The
for loop:
Code:
    for (row=g_item_offset; row < (g_item_offset + items_per_page) && row < g_total_items ; row++)
only fills the gtk_list_store with one-page of icons. The line
Code:
        GdkPixbuf *thumbnail = get_thumbnail(cell);
should then only be changed to load the thumbnail from file?

If there is no image in global.db you can check if there is one in the filesystem. This would then work for the settings/global.db also I guess?

I haven't looked at this in much detail so I might be missing some details? Do the Delete-mode overlays work?
Mackx is offline   Reply With Quote
Old 06-24-2011, 03:38 PM   #40
Mackx
Guru
Mackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to behold
 
Posts: 999
Karma: 19985
Join Date: Dec 2008
Location: Netherlands
Device: iRex DR1000S
Quote:
Originally Posted by Iņigo View Post
* update other programs to copy new icons to fs instead of db
- mdbindex
- cover from uds
The function set_file_metadata_impl() in ermetadb_file.c seems to be a good candidate. If I remember well from some past logging UDS will also use this function to update thumbnails (mdbindex used UDS to update thumbnails). The function could be changed to not execute the sql_t-query, but use the data to write images into the filesystem.
Mackx is offline   Reply With Quote
Old 06-24-2011, 05:12 PM   #41
Iņigo
Guru
Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.
 
Posts: 730
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, iRex DR800S. K4NT. Kobo Aura, Aura One, Libra 2.
Quote:
Originally Posted by Mackx View Post
The
for loop:
Code:
    for (row=g_item_offset; row < (g_item_offset + items_per_page) && row < g_total_items ; row++)
only fills the gtk_list_store with one-page of icons.
Ouchhhh!!! silly me... I hadn't noticed this part
I was studying load_items_in_model just now. I thought ctb populated the model with all the results for the view... but not... so no need to get the pixbufs on runtime... so no need to change so much code, even no need to patch erGtkViewIcon.

Quote:
Originally Posted by Mackx View Post
Do the Delete-mode overlays work?
CTB code was almost finished except ovelays and shortcuts that needed some modifications to work with the new code.

Over 20 hours of hard work...

Ok, I'll create a branch with these code as a monument/example of my silliness.

Anyway, needed changes will be much easier.
Iņigo is offline   Reply With Quote
Old 06-24-2011, 05:45 PM   #42
Mackx
Guru
Mackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to behold
 
Posts: 999
Karma: 19985
Join Date: Dec 2008
Location: Netherlands
Device: iRex DR1000S
I will be resetting my brain tomorrow, glad I told you today...
Mackx is offline   Reply With Quote
Old 06-24-2011, 06:04 PM   #43
Mackx
Guru
Mackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to behold
 
Posts: 999
Karma: 19985
Join Date: Dec 2008
Location: Netherlands
Device: iRex DR1000S
Quote:
Originally Posted by Iņigo View Post
- don't query db for thumbnails
If keeping thumbnail in the query is not generating too much extra delay it might be better to keep it in, to handle the global.db from settings. Or maybe it is possible to leave thumbnail from the query in filemodel.c:create_query() in case it is not the SETTINGS_VIEW? Although the delay might be caused more by the JOIN in ermetadb_file.c:global_select_common() or the JOIN is removed if the SQL-query is optimised before executing... my database knowledge is not good enough for that.
Mackx is offline   Reply With Quote
Old 06-24-2011, 07:23 PM   #44
Iņigo
Guru
Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.
 
Posts: 730
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, iRex DR800S. K4NT. Kobo Aura, Aura One, Libra 2.
Quote:
Originally Posted by Mackx View Post
If keeping thumbnail in the query is not generating too much extra delay it might be better to keep it in, to handle the global.db from settings. Or maybe it is possible to leave thumbnail from the query in filemodel.c:create_query() in case it is not the SETTINGS_VIEW? Although the delay might be caused more by the JOIN in ermetadb_file.c:global_select_common() or the JOIN is removed if the SQL-query is optimised before executing... my database knowledge is not good enough for that.
I don't think it has much sense to keep thumbnail in the query as the benefits will come from not having any cover in global.db

Patch is quite simple now. See attached file.
It works without problems in the emulator.

Next steps:
1. write a tool (in vala) to copy images from db to fs, remove them from db and do a vacuum
2. test in the real DR with lots of books and compare
3. update other programs to copy new icons to fs instead of db
- mdbindex
- cover from uds

Btw, have a nice party tomorrow
I have a cooking day
Attached Files
File Type: gz covers_in_fs.patch.gz (2.9 KB, 422 views)

Last edited by Iņigo; 06-24-2011 at 07:32 PM.
Iņigo is offline   Reply With Quote
Old 06-25-2011, 02:00 AM   #45
Mackx
Guru
Mackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to behold
 
Posts: 999
Karma: 19985
Join Date: Dec 2008
Location: Netherlands
Device: iRex DR1000S
Quote:
Originally Posted by Iņigo View Post
I don't think it has much sense to keep thumbnail in the query as the benefits will come from not having any cover in global.db
I thought it might make handling settings and icons for (application) shortcuts and folders easier. But only if the penalty of leaving thumbnail in the query is not too high.

(Also have a lot of fun today.)
Mackx is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Mobiperl Mobiperl - Perl tools for handling MobiPocket files tompe Kindle Formats 890 01-17-2021 06:16 PM
Large number of books on memory card pwalker8 Sony Reader 8 03-24-2009 02:20 PM
huge pdf files on iliad yolle iRex 7 08-23-2008 10:59 AM
Fictionwise Handling Fee Against Credit Card Terms of Service? Gideon News 9 08-15-2008 02:13 PM
Huge RTF files after conversion TheMadBrewer Sony Reader 2 01-01-2007 02:55 AM


All times are GMT -4. The time now is 11:44 PM.


MobileRead.com is a privately owned, operated and funded community.