Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Readers > More E-Book Readers > iRex > iRex Developer's Corner

Notices

Reply
 
Thread Tools Search this Thread
Old 04-03-2011, 04:22 PM   #16
Mackx
Guru
Mackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to behold
 
Posts: 998
Karma: 19985
Join Date: Dec 2008
Location: Netherlands
Device: iRex DR1000S
I did some quick testing with the emulator (creating 3000 links to the same document). Some measurements results on the emulator and some preliminary conclusions.

I used the Book-View, sorted on title.
Book-View: 22 books
load_dir_from_metadb() ~75ms
load_items_in_model() ~200ms

Book-View: 3022 books
load_dir_from_metadb() ~2000ms
load_items_in_model() ~250ms

Using a different sorting method does not change the time significantly (size, date-added, type).

The Recently-Read View uses a LIMIT(38) in the SQL query.
Recently-Read View: 3022 books
load_dir_from_metadb() ~1000ms
load_items_in_model() ~200ms

Favourite-View( dr1000-hackx): 3022 books (of which only 2 are favourite)
load_dir_from_metadb() <100ms
load_items_in_model() <100ms


Some early conclusions:
- The database access consumes most of the time.
- Even when using LIMIT the database access will still use a lot of time. Although time is halved, which might be enough to make it usable for 3000 books.
- Using tags to limit the number of books reduces the time considerably.
- It seems that most time is consumed in sorting the documents (or copying from the database ) and not in selecting the documents in the database.

TODO:
- Check influence of (many) thumbnails in the database
- Removing sorting from SQL-command to check if it is sorting or copying 'selected' data from database to memory that is consuming the time.
- Using OFFSET/LIMIT as a test to check impact.

@Iņigo: How long does it normally take to show the Book-view on your DR?


Marcel.
Mackx is offline   Reply With Quote
Old 04-03-2011, 07:38 PM   #17
Iņigo
Guru
Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.
 
Posts: 712
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, in love with iRex DR800S. Also a K4NT. Now a Kobo Aura
Hi Mackx and everyone,

More news: I've added some improvements to popupmenu, in reverse
commit code on BitBucket repository (i.e. chronological):
- Mackx's tasks-list menu
- sysinfo menu entry (requires sysinfo binary installed on /usr/bin)
- only show tasks-list group menu entry if more than 1 active task

Finally, I'm thinking not to create a new menu entry for Programs.
I'm going to write a vala-based installer (instead of using zenity) and publish a new release of DR800+.
Then, when I have free time, I'll study Mackx's hacks and decide what do do next.

Btw, vala-0.12 was released today. It works ok

Iņigo
Iņigo is offline   Reply With Quote
Old 04-05-2011, 03:47 PM   #18
Mackx
Guru
Mackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to behold
 
Posts: 998
Karma: 19985
Join Date: Dec 2008
Location: Netherlands
Device: iRex DR1000S
Quote:
Originally Posted by Mackx View Post
TODO:
- Check influence of (many) thumbnails in the database
- Removing sorting from SQL-command to check if it is sorting or copying 'selected' data from database to memory that is consuming the time.
- Using OFFSET/LIMIT as a test to check impact.
I did some extra measurements (also on the emulator)

I removed the ORDER BY lines from the query
Book-View-no sort: 3022 books
load_dir_from_metadb() ~650ms
load_items_in_model() ~250ms

I added 'LIMIT 19 OFFSET 500' (to the original query with sorting)
Book-View-limit/offset: 3022 books
load_dir_from_metadb() ~1500ms
load_items_in_model() ~75ms

So as I already suspected the sorting takes most of the time. Using LIMIT/OFFSET does not give the speed gain hoped.
So I am afraid that 'simple' changes will not make the DR more responsive when using a lot of books.
Mackx is offline   Reply With Quote
Old 04-05-2011, 06:51 PM   #19
dima_tr
Groupie
dima_tr has learned how to read e-booksdima_tr has learned how to read e-booksdima_tr has learned how to read e-booksdima_tr has learned how to read e-booksdima_tr has learned how to read e-booksdima_tr has learned how to read e-booksdima_tr has learned how to read e-books
 
Posts: 175
Karma: 863
Join Date: Oct 2009
Location: Germany, Karlsruhe
Device: Iliad v2, DR800S (inlove)
Marcel, can you please check the performance change when using the libsqlite3.so.0 attached? The QEMU works OK with it.

SQLite3, Version 3.7.5, http://www.sqlite.org/sqlite-amalgamation-3070500.zip Amalgamation (gives about 5-10% better performance) + built with some optimization flags. See the build script.
Attached Files
File Type: zip libsqlite.zip (412.7 KB, 65 views)
dima_tr is offline   Reply With Quote
Old 04-06-2011, 07:06 AM   #20
Iņigo
Guru
Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.
 
Posts: 712
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, in love with iRex DR800S. Also a K4NT. Now a Kobo Aura
Quote:
Originally Posted by dima_tr View Post
Marcel, can you please check the performance change when using the libsqlite3.so.0 attached? The QEMU works OK with it.

SQLite3, Version 3.7.5, http://www.sqlite.org/sqlite-amalgamation-3070500.zip Amalgamation (gives about 5-10% better performance) + built with some optimization flags. See the build script.
why is the library so big? Analyzing the DR800 2.0RC3 library it looks it's version 3.5.2:
libsqlite3.so.0.8.6 -> 375544 vs. libsqlite3.so.0 -> 857224 bytes

Anyway, that performance improvement is interesting.
/me thinking if upgrading glib could be useful too...


I've spent (well, wasted) some hours trying ty compile the recently released v1.0 of pcc (a compiler, like gcc, [1]) for my DR devel environment, in order to check if the resulting DR binaries could run faster that the ones with gcc.
pcc is not supported for linux + ARM processors, but it is for NetBSD + ARM so I thought it wouldn't be so hard... I was mistaken...


DR800 and DR1000 share the same processor ([2], [3]): FreeScale i.MX31 (ARM1136JF-S, [4]).
@dima_tr: could you repeat your tests with these new arguments to gcc:
-march=arm1136jf-s -mtune=arm1136jf-s


Btw, Mackx, I think you work as C developer for embeded devices so maybe you can explain us if it's worth to build a newer gcc or pcc cross-compilers?
would we get any speed difference in the binaries? Devel env includes gcc v4.3.0. but gcc v4.4 and v4.5 changes notes ([5], [6]) list some performance improvements for ARM processors.

Iņigo

[1] http://pcc.ludd.ltu.se/
[2] http://wiki.mobileread.com/wiki/DR800SG
[3] http://wiki.mobileread.com/wiki/IRex_Digital_Reader
[4] http://www.freescale.com/webapp/sps/...sp?code=i.MX31
[5] http://gcc.gnu.org/gcc-4.4/changes.html
[6] http://gcc.gnu.org/gcc-4.5/changes.html
Iņigo is offline   Reply With Quote
Old 04-06-2011, 09:07 AM   #21
Mackx
Guru
Mackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to behold
 
Posts: 998
Karma: 19985
Join Date: Dec 2008
Location: Netherlands
Device: iRex DR1000S
Quote:
Originally Posted by Iņigo View Post
Btw, Mackx, I think you work as C developer for embeded devices so maybe you can explain us if it's worth to build a newer gcc or pcc cross-compilers?
would we get any speed difference in the binaries? Devel env includes gcc v4.3.0. but gcc v4.4 and v4.5 changes notes ([5], [6]) list some performance improvements for ARM processors.
That is a good question, I was never involved in selecting a compiler version. Unfortunately my company is downsizing, so the knowledge is now somewhere in India, with collegues that I do not know.
Are there any measurements available on the internet comparing the gcc versions for an arm11 or comparing pcc against gcc (is pcc faster in compiling or does it generate faster code?). Is it a 5-10% improvement or a much more?
Mackx is offline   Reply With Quote
Old 04-06-2011, 10:59 AM   #22
dima_tr
Groupie
dima_tr has learned how to read e-booksdima_tr has learned how to read e-booksdima_tr has learned how to read e-booksdima_tr has learned how to read e-booksdima_tr has learned how to read e-booksdima_tr has learned how to read e-booksdima_tr has learned how to read e-books
 
Posts: 175
Karma: 863
Join Date: Oct 2009
Location: Germany, Karlsruhe
Device: Iliad v2, DR800S (inlove)
Guys I have not done any performance testing, just asked Marcel to do ones; it will be faster as he has the setup for this. 10% improvements came from the explanation on what Amalgamation is (SQLite page). The reasoning makes sense.

Marcel, can you please add the mentioned flags, compile libsqlite3.so.0 and make the performance test? Pleeease

-march=arm1136jf-s -mtune=arm1136jf-s -O3 -funroll-loops
dima_tr is offline   Reply With Quote
Old 04-06-2011, 11:29 AM   #23
Mackx
Guru
Mackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to behold
 
Posts: 998
Karma: 19985
Join Date: Dec 2008
Location: Netherlands
Device: iRex DR1000S
I would first like to know more about the problem that we are looking at. Until now I only heared that the Book view is unusable when having 3000 books. But what does that mean, you need to wait for 2 seconds or for 20 seconds?
Next is if it makes sense to save 10% (maximum) of this time. I have seen memory sticks taking a lot of time when 'switching banks' currently I only measured on the emulator. The real DR might show that most time is lost in SD-card access.
Also things like the size of the database (global.db) might play a role, since thumbnails are also stored there.
Iņigo: you reported the issue with the unresponsive Book-view, can you tell a little more about how the Book-view mis-behaves and the size of your global.db file?
Mackx is offline   Reply With Quote
Old 04-06-2011, 11:55 AM   #24
Iņigo
Guru
Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.
 
Posts: 712
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, in love with iRex DR800S. Also a K4NT. Now a Kobo Aura
Quote:
Originally Posted by Mackx
Are there any measurements available on the internet comparing the gcc versions for an arm11 or comparing pcc against gcc (is pcc faster in compiling or does it generate faster code?). Is it a 5-10% improvement or a much more?
I surely agree that the main factor on the usability is SD access, but an increment of 5% on the libs and apps could be useful too.
Evenmore if the only work for us is to add some new flags to the compiler or build a new version of it.

Also note that as I mentioned before, pcc doesn't compile for linux+arm+elf nowdays.


Quote:
Originally Posted by Mackx View Post
I would first like to know more about the problem that we are looking at. Until now I only heared that the Book view is unusable when having 3000 books. But what does that mean, you need to wait for 2 seconds or for 20 seconds?
Next is if it makes sense to save 10% (maximum) of this time. I have seen memory sticks taking a lot of time when 'switching banks' currently I only measured on the emulator. The real DR might show that most time is lost in SD-card access.
Also things like the size of the database (global.db) might play a role, since thumbnails are also stored there.
Iņigo: you reported the issue with the unresponsive Book-view, can you tell a little more about how the Book-view mis-behaves and the size of your global.db file?
I can't provide any real data now because my DR800 halts when I access Books atm.
But if I remember correctly, I have ~3000 books, global.db size is ~28MB and it lasted ~20 or 30 secs in showing me the first page.

EDIT: 3680 rows in global.db, 31 seconds to show Book folder, 27.82 MB

I'm moving my DR devel env from a VM to my real desktop and I'm building a new cross-compiler now too. So I'll try to come back with some numbers early.

Iņigo

Last edited by Iņigo; 04-06-2011 at 01:35 PM.
Iņigo is offline   Reply With Quote
Old 04-06-2011, 11:57 AM   #25
dima_tr
Groupie
dima_tr has learned how to read e-booksdima_tr has learned how to read e-booksdima_tr has learned how to read e-booksdima_tr has learned how to read e-booksdima_tr has learned how to read e-booksdima_tr has learned how to read e-booksdima_tr has learned how to read e-books
 
Posts: 175
Karma: 863
Join Date: Oct 2009
Location: Germany, Karlsruhe
Device: Iliad v2, DR800S (inlove)
My expectation is that it is more than 10% speedup.
dima_tr is offline   Reply With Quote
Old 04-07-2011, 04:39 PM   #26
Iņigo
Guru
Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.
 
Posts: 712
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, in love with iRex DR800S. Also a K4NT. Now a Kobo Aura
Quote:
Originally Posted by dima_tr View Post
Guys I have not done any performance testing, just asked Marcel to do ones; it will be faster as he has the setup for this. 10% improvements came from the explanation on what Amalgamation is (SQLite page). The reasoning makes sense.

Marcel, can you please add the mentioned flags, compile libsqlite3.so.0 and make the performance test? Pleeease

-march=arm1136jf-s -mtune=arm1136jf-s -O3 -funroll-loops
That "-march" is incorrect there, it should be "-mcpu" i.e.:
Code:
-mcpu=arm1136jf-s -mtune=arm1136jf-s -O3 -funroll-loops
Anyway, I've recompiled last version of FBReader with and without these flags.

My tests:

2 books:
  • Book A: .fb2, 1.6MB, 699 pages (small font size)
  • Book B: .fb2, 430KB, 192 pages (small font size)
3 different tests, one run atfer the previous one
  • (1) open book A from reboot
  • (2) open book A after closing 1
  • (3) open book B after closing 2

Between classic FBReader tests and new FBReader tests I've rebooted the DR800S.

Times (in secs):
Code:
          (1)  (2)  (3)  
Classic:  13    7    11
New:       8    5    9
Of course, my measurements are far from the scientific method, but it looks there are some useful speed improvements.
Also real overall use feels faster, for example, changing pages.


I've attached both FBReader versions (note new one is bigger: 3.5 vs 2.3 MB).
Could you repeat the tests and comment, please?


If we can confirm these speed improvements, I really think we could get significant speed enhancements recompiling some DR apps/libs.

Iņigo

PS: I presume size increment comes from the "-funroll-loops" flags


Off-topic:
- My experiments building a new cross-compiler from gcc v4.5.2 sources failed.
I think I need to create a sysroot... it requires too much time I don't currently have
- I've also tried installing new versions of the prepackaged poky platform [1].
. v4 and v5 are too modern
. v3.2 and v3.3.1 compilers can be used with some minor hacks and links, but qemu doesn't run.
So I've returned to IREX supplied poky version.


[1] http://pokylinux.org/


EDIT: I've recompiled all components of DR800+ and the vala programs... system feels faster. Don't know if it's my own imagination, but feels faster.

EDIT 2: I've changed original sqlite library for a compilation of sqlite amalgamation with these flags. System and Books folder opening looks faster as well.
Attached Files
File Type: gz FBReader.tests.tar.gz (1.96 MB, 60 views)

Last edited by Iņigo; 04-08-2011 at 06:32 PM.
Iņigo is offline   Reply With Quote
Old 04-08-2011, 06:49 AM   #27
Mackx
Guru
Mackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to behold
 
Posts: 998
Karma: 19985
Join Date: Dec 2008
Location: Netherlands
Device: iRex DR1000S
Interesting! I will also use the new flags to do some experiments.
Mackx is offline   Reply With Quote
Old 04-14-2011, 03:20 AM   #28
Mackx
Guru
Mackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to behold
 
Posts: 998
Karma: 19985
Join Date: Dec 2008
Location: Netherlands
Device: iRex DR1000S
Quote:
Originally Posted by Iņigo View Post
EDIT: 3680 rows in global.db, 31 seconds to show Book folder, 27.82 MB
Something related:
I recently examined a global.db file from someone else. It seems that in the past a lot of documents where put onto it and removed again. When I used the VACUUM-command (using the free SqLite2009Pro program) the size of global.db was reduced from ~2.5MB to ~0.5MB

Maybe reducing the size also improves the speed?
Mackx is offline   Reply With Quote
Old 04-15-2011, 10:04 AM   #29
dima_tr
Groupie
dima_tr has learned how to read e-booksdima_tr has learned how to read e-booksdima_tr has learned how to read e-booksdima_tr has learned how to read e-booksdima_tr has learned how to read e-booksdima_tr has learned how to read e-booksdima_tr has learned how to read e-books
 
Posts: 175
Karma: 863
Join Date: Oct 2009
Location: Germany, Karlsruhe
Device: Iliad v2, DR800S (inlove)
Hehe, it is possible to add autovacuuming using the approach described here http://www.friday.com/bbum/2007/03/0...#comment-81574

Seems to be easily done by operating global.db on local Linux machine.

Another way is to recompile libsqlite to have autovacuuming on by default (now it is off). http://www.sqlite.org/compile.html#default_autovacuum
dima_tr is offline   Reply With Quote
Old 04-15-2011, 10:52 AM   #30
Mackx
Guru
Mackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to behold
 
Posts: 998
Karma: 19985
Join Date: Dec 2008
Location: Netherlands
Device: iRex DR1000S
Quote:
Originally Posted by dima_tr View Post
Hehe, it is possible to add autovacuuming using the approach described here http://www.friday.com/bbum/2007/03/0...#comment-81574

Seems to be easily done by operating global.db on local Linux machine.

Another way is to recompile libsqlite to have autovacuuming on by default (now it is off). http://www.sqlite.org/compile.html#default_autovacuum
I am not sure how long vacuuming takes, I think even auto vacuum, rewrites the complete file, so it could cause unpredictable delays. I am not sure if this works well with the idle-mechanism of the DR.
Maybe an extra item in Settings (or a simple application or script) to maintain global.db (and all the metadata.db) files. Allowing to do an integrity check and a vacuum?
Mackx is offline   Reply With Quote
Reply

Tags
secret, suprise, testing

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Strange things happening w/PDF ebook Writernan Astak EZReader 5 04-05-2010 07:36 PM
Best firmware for Hanlin V5 aka Bebook Mini aka Papyre 5.1? javimm HanLin eBook 1 01-11-2010 06:37 PM
Strange things happening at Fictionwise... captaingeorges Bookeen 5 03-08-2009 06:20 AM
The Secret of Lost Things Taylor514ce Reading Recommendations 7 05-15-2008 10:06 AM
Strange things happening after Kindle update tsgreer Amazon Kindle 15 02-22-2008 06:34 PM


All times are GMT -4. The time now is 06:24 AM.


MobileRead.com is a privately owned, operated and funded community.