Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Readers > More E-Book Readers > iRex > iRex Developer's Corner

Notices

Reply
 
Thread Tools Search this Thread
Old 06-25-2011, 11:32 AM   #46
yuri_b
Connoisseur
yuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enough
 
Posts: 71
Karma: 592
Join Date: Aug 2010
Device: irex dr800sg DR1000S
Smile new design of ctb

Hi
I did some research on sql and how it used in IREX device in file viewer,
and spend several hours to implement new design of queuing for files in ctb

In short new design:
1) SELECT from data base will return minimum info, in order minimize memory load.
2) All QUERIES done in such manner that SQL use only 1 index to retrive data.
3) All data for each item will be loaded when it needed. On preparing view.
4) When ctb starts it will automatically create needed indixes in not exists.
5) It would be nice implement caching for items in page, Next time.

So selecting all items from data base will be only while changing sort order or directory. When changing view only visible items are loaded from db.

In my tests I get time about 3-10 shorter then old version.

For example:

First loading: 22415 <=> 6381
Change to icon view: 54536 <=> 2435 (no loading done here)
Sort by author: 14079 <=> 7311.


Changed programs: ctb and libermetadb.so.0.0.0

Attached version for test.


Hav a nice day

Yura
Attached Files
File Type: zip ctb800sg2.zip (73.2 KB, 53 views)

Last edited by yuri_b; 06-25-2011 at 12:12 PM.
yuri_b is offline   Reply With Quote
Old 06-25-2011, 12:29 PM   #47
Iņigo
Guru
Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.
 
Posts: 712
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, in love with iRex DR800S. Also a K4NT. Now a Kobo Aura
Quote:
Originally Posted by yuri_b View Post
Hi
I did some research on sql and how it used in IREX device in file viewer,
and spend several hours to implement new design of queuing for files in ctb

In short new design:
1) SELECT from data base will return minimum info, in order minimize memory load.
2) All QUERIES done in such manner that SQL use only 1 index to retrive data.
3) All data for each item will be loaded when it needed. On preparing view.
4) When ctb starts it will automatically create needed indixes in not exists.
5) It would be nice implement caching for items in page, Next time.

So selecting all items from data base will be only while changing sort order or directory. When changing view only visible items are loaded from db.

In my tests I get time about 3-10 shorter then old version.

For example:

First loading: 22415 <=> 6381
Change to icon view: 54536 <=> 2435 (no loading done here)
Sort by author: 14079 <=> 7311.


Changed programs: ctb and libermetadb.so.0.0.0

Attached version for test.


Hav a nice day

Yura
Looks impressive.
How many books/entries do you have in your global.db file?

Could you share the sources?
Iņigo is online now   Reply With Quote
Old 06-25-2011, 12:52 PM   #48
yuri_b
Connoisseur
yuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enough
 
Posts: 71
Karma: 592
Join Date: Aug 2010
Device: irex dr800sg DR1000S
1920 entries.
No problem it based on your last version ctb + my dinamic bar. Lets check if if all OK and if this changes are acceptable, after you could insert if we will decide this, to version control.

Yura
yuri_b is offline   Reply With Quote
Old 06-25-2011, 01:46 PM   #49
Iņigo
Guru
Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.
 
Posts: 712
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, in love with iRex DR800S. Also a K4NT. Now a Kobo Aura
Quote:
Originally Posted by yuri_b View Post
1920 entries.
No problem it based on your last version ctb + my dinamic bar. Lets check if if all OK and if this changes are acceptable, after you could insert if we will decide this, to version control.

Yura

Have you adapted and added the 2nd language bar already? perfect! I hadn't started yet.
Today I'm resting a bit and fighting the strange problem with popupmenu and newspaperdirect app.

The idea is to branch from r49 and then merge to trunk.
I'm working on some things in trunk on my local repository, not yet on bitbucket.

Could you attach a diff here please? I can say I'm eager to see it ;-)

Iņigo
Iņigo is online now   Reply With Quote
Old 06-25-2011, 06:09 PM   #50
fekhner
Addict
fekhner is kind to children and small, furry animalsfekhner is kind to children and small, furry animalsfekhner is kind to children and small, furry animalsfekhner is kind to children and small, furry animalsfekhner is kind to children and small, furry animalsfekhner is kind to children and small, furry animalsfekhner is kind to children and small, furry animalsfekhner is kind to children and small, furry animalsfekhner is kind to children and small, furry animalsfekhner is kind to children and small, furry animalsfekhner is kind to children and small, furry animals
 
Posts: 326
Karma: 6908
Join Date: Sep 2010
Device: DRs, EE, ipad/iphone, Helix
Quote:
Originally Posted by Mackx View Post
I will be resetting my brain tomorrow, glad I told you today...
have fun in Belgium !

you guys are really working hard here ...

Last edited by fekhner; 06-25-2011 at 06:11 PM.
fekhner is offline   Reply With Quote
Old 06-25-2011, 10:27 PM   #51
Iņigo
Guru
Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.
 
Posts: 712
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, in love with iRex DR800S. Also a K4NT. Now a Kobo Aura
CTB: covers in file system instead of global.db (cont)

Hi again,

Today I've tested the patch on my DR800, so let's report these good results!

Steps I've done:
- disable automatic generation of thumbnails while indexing
- extracted covers from global.db to the file system, removed all images from db and finally vacuum db [1]
- copied the ctb built with my patch from message #44 and reboot the ereader


* BEFORE
Entries in db: 4k
Entries in db with thumbnails: 2.2k
DB Size: 29.4 MB
File system space for covers: 0
Time to open Book view after reboot (cold): 30 sec
Time to open Book view (warm): 27 sec
Time to open Recent* view after reboot (cold), 50 items: 9 sec
Time to open Recent* view (warm), 50 items: 7 sec

* AFTER THE PATCHED CTB
Entries in db: 4k
Entries in db with thumbnails: 0
DB Size: 1 MB
File system space for covers: 36.2 MB = 11 MB (small size) + 25.2 (medium size)
Time to open Book view after reboot (cold): 4-5 sec
Time to open Book view (warm): 4 sec
Time to open Recent* view after reboot (cold), 50 items: 3 sec
Time to open Recent* view (warm), 50 items: 2-3 sec


[1] the vala-based DR binary program I've written has a bug and can't be used now. I'm waiting to get a fix for a sqlite bug I've discovered.
So I've written a quick hack in python, run the script on my computer and then moved manually the extracted covers to the DR


Things to do before considering this finished:
- wait for a fix for the vala program, so we could migrate cover on the DR
- settings icons are missing
. they were stored in global.db
. which program added them?
- change other parts of the DR firmware to use file system instead of DB:
. mdbindex
. metadata extracted from UDS
- build a pretty package to the DR


Final comments:
- this is a really needed hack to be implemented in the DR, unless you have less than 1000 books or you don't use covers
- we only use ~25% more of disk space, i.e. ~7MB more than the global.db for 2000+ books, neligible considering the advantages
- combined with yuri_b's db handling improvements we could support much more books in the DR, not 150k but maybe 25k or even more


Iņigo

Last edited by Iņigo; 06-25-2011 at 10:40 PM.
Iņigo is online now   Reply With Quote
Old 06-26-2011, 03:58 AM   #52
yuri_b
Connoisseur
yuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enoughyuri_b will become famous soon enough
 
Posts: 71
Karma: 592
Join Date: Aug 2010
Device: irex dr800sg DR1000S
Quote:
Originally Posted by Iņigo View Post
Could you attach a diff here please? I can say I'm eager to see it ;-)
Iņigo
Attached.

But wait, it isn't last and checked version, maybe I will change it.

Yura


P.S. As you can see I query only for a few items: file_id, filename, is_directory, author and title, NO thumbnails are loaded, and the most important: the SQL uses index to find all items. It loads an index from DB and uses it to find and fetch all data, so thumbnails shouldn't affect to performance.
Ideally it will load only file_id's for first 10 pages, so there is will be no restrictions for book count. The problem as I wrote already how to fill alpha bar. I still look for it's solution.
Attached Files
File Type: zip diff_ctb.zip (14.6 KB, 53 views)

Last edited by yuri_b; 06-26-2011 at 04:06 AM.
yuri_b is offline   Reply With Quote
Old 06-26-2011, 04:49 AM   #53
Iņigo
Guru
Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.
 
Posts: 712
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, in love with iRex DR800S. Also a K4NT. Now a Kobo Aura
Quote:
Originally Posted by yuri_b View Post
Attached.

But wait, it isn't last and checked version, maybe I will change it.

Yura


P.S. As you can see I query only for a few items: file_id, filename, is_directory, author and title, NO thumbnails are loaded, and the most important: the SQL uses index to find all items. It loads an index from DB and uses it to find and fetch all data, so thumbnails shouldn't affect to performance.
Ideally it will load only file_id's for first 10 pages, so there is will be no restrictions for book count. The problem as I wrote already how to fill alpha bar. I still look for it's solution.
a diff against r46 or r49 should be ok.
r50, last one as now, is a different branch were I put my first initial - and silly - patch to use covers in file system instead of in global.db. Note r50 branch will not be used anymore, work will continue in r49.
Iņigo is online now   Reply With Quote
Old 06-26-2011, 08:03 AM   #54
Mackx
Guru
Mackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to behold
 
Posts: 998
Karma: 19985
Join Date: Dec 2008
Location: Netherlands
Device: iRex DR1000S
You (Iņigo and Yura) have been working very hard yesterday(, while I had all the fun:-).

Quote:
Originally Posted by Iņigo View Post
- settings icons are missing
. they were stored in global.db
. which program added them?
The global.db for settings is created in the build process. See sources: iRex\ctb\data\add_shortcuts_to_database.sh
Mackx is offline   Reply With Quote
Old 06-26-2011, 09:29 AM   #55
Mackx
Guru
Mackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to behold
 
Posts: 998
Karma: 19985
Join Date: Dec 2008
Location: Netherlands
Device: iRex DR1000S
Quote:
Originally Posted by Iņigo View Post
Final comments:
- combined with yuri_b's db handling improvements we could support much more books in the DR, not 150k but maybe 25k or even more
I created a patch that allows to directly browse the SD Card without using the database. It can add documents that are not yet in the database to the database. I still have to write some code to prune the database to an acceptable count. I will see how to merge this with the patches that you are creating here.
Mackx is offline   Reply With Quote
Old 06-26-2011, 10:31 AM   #56
Mackx
Guru
Mackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to beholdMackx is a splendid one to behold
 
Posts: 998
Karma: 19985
Join Date: Dec 2008
Location: Netherlands
Device: iRex DR1000S
Quote:
Originally Posted by Iņigo View Post
Things to do before considering this finished:
- change other parts of the DR firmware to use file system instead of DB:
. mdbindex
. metadata extracted from UDS
I commented before on this and did some experiments. UDS updates thumbnails (and other info) when a document is first shown. Notepad always updates the thumbnail when it is closed. All updates go via the function ermetadb_file.c:set_file_metadata_impl(). When a thumbnail column is found (look for '// find column in thumbnails' and further), all required information is available. cell_name contains 'thumb_data_medium'/'small'. cell is of type metadata_cell, so the blob info can easily be retrieved (metadata_cell.h). See notepad_thumbnail.cpp for the buffers that are supplied.

Not sure if you found this yourself, but I wanted to contribute also.
Mackx is offline   Reply With Quote
Old 06-26-2011, 12:12 PM   #57
Iņigo
Guru
Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.
 
Posts: 712
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, in love with iRex DR800S. Also a K4NT. Now a Kobo Aura
Quote:
Originally Posted by Mackx View Post
I commented before on this and did some experiments. UDS updates thumbnails (and other info) when a document is first shown. Notepad always updates the thumbnail when it is closed. All updates go via the function ermetadb_file.c:set_file_metadata_impl(). When a thumbnail column is found (look for '// find column in thumbnails' and further), all required information is available. cell_name contains 'thumb_data_medium'/'small'. cell is of type metadata_cell, so the blob info can easily be retrieved (metadata_cell.h). See notepad_thumbnail.cpp for the buffers that are supplied.

Not sure if you found this yourself, but I wanted to contribute also.
Thanks, this part is already finished (your previous comments were very helpful, thanks), but not in the repo yet.
I attach the new patch and the files. If anyone wants to try remember to remove global.db first (and make a backup).

There is a bug (not specially related to this patch) using UDS from DR1000 and DR800+. When indexing many files, indexer open the books with UDS to retrieve metadata and thumbnail. As we susbtituted original DR800 UDS binary with the one for DR1000, there will produce a crash on UDS eventually, so indexer won't continue.
I can't see a solution for this. So run indexer normally and if you want a cover for any book, open it manually and you'll get the image.


Remaining steps:
- handle files deleted => remove image files from disk if they exist
- settings icons are not recreated yet
- old notes thumb are not migrated (they need to be updated), but new notes get their thumb correctly
- finish the tool to move covers from global.db to file system [waiting due to a bug in vala's sqlite3 bindings]
- rewrite my "icons" app to copy images to file system instead of to global.db
- package
- profit


Iņigo

Btw, hope you had a good weekend, full of fun, sex and rock and roll!
Attached Files
File Type: gz covers_in_fs.patch.gz (4.8 KB, 54 views)
File Type: gz covers_in_fs.files.tar.gz (95.4 KB, 55 views)

Last edited by Iņigo; 06-26-2011 at 12:57 PM.
Iņigo is online now   Reply With Quote
Old 07-01-2011, 07:18 PM   #58
Iņigo
Guru
Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.
 
Posts: 712
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, in love with iRex DR800S. Also a K4NT. Now a Kobo Aura
Hi,

I've been some days away for work reasons.

Anyway tonight I've rewritten some parts of the patch.
Now the covers are stored as <sd_card>/System/.covers/size/xx/file_id.png
where:
size = thumbnail size: "small" or "medium"
xx = file_id % 256, in hexadecimal
file_id = book identifier index inside global.db

This way we could provide a two way conversion between "classic" format (with thumbnails inside the db) and new format.

Now I have to complete the code to handle file deletion and finish the migration tools.

Best regards,
Iņigo
Attached Files
File Type: gz covers_in_fs.patch.gz (5.1 KB, 47 views)
Iņigo is online now   Reply With Quote
Old 07-01-2011, 11:12 PM   #59
Mr. X
Connoisseur
Mr. X is on a distinguished road
 
Posts: 68
Karma: 64
Join Date: Mar 2010
Location: Philadelphia
Device: iRex DR800SG
Jumping in to this thread very late...

If you're already mucking about in erindexer could I kindly request a feature that's been on my todo list for about a year now?

I use Calibre to manage my books and add them to the DR's filesystem. Calibre's database generally has better metadata and much better cover images (pulled from Amazon etc) compared to the .epub or .pdf, but short of duplicating the indexing operation entirely from within calibre there's no way to get this data into global.db since inserting a new row with just the minimum columns (filename, directory_path, is_directory, filetype, title, author, tag) filled in didn't seem to leave the database in a happy state... And cramming pngs into the thumbnails table seemed even uglier from within python

My plan had been to modify the indexer to look for filename.metadata and filename.cover whenever it found filename.{pdf,epub,pdb} and insert that information into the database instead of opening up UDS for every book. That lets you have metadata (so sort by author works) without the super slow UDS indexing. This isn't tied to Calibre, you could generate these .metadata and .cover files on a PC from any source (including even doing what UDS does, just on a machine with a few GB of ram not a few MB...)
Mr. X is offline   Reply With Quote
Old 07-02-2011, 06:26 AM   #60
Iņigo
Guru
Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.Iņigo did not drink the Kool Aid.
 
Posts: 712
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, in love with iRex DR800S. Also a K4NT. Now a Kobo Aura
Quote:
Originally Posted by Mr. X View Post
Jumping in to this thread very late...

If you're already mucking about in erindexer could I kindly request a feature that's been on my todo list for about a year now?

I use Calibre to manage my books and add them to the DR's filesystem. Calibre's database generally has better metadata and much better cover images (pulled from Amazon etc) compared to the .epub or .pdf, but short of duplicating the indexing operation entirely from within calibre there's no way to get this data into global.db since inserting a new row with just the minimum columns (filename, directory_path, is_directory, filetype, title, author, tag) filled in didn't seem to leave the database in a happy state... And cramming pngs into the thumbnails table seemed even uglier from within python

My plan had been to modify the indexer to look for filename.metadata and filename.cover whenever it found filename.{pdf,epub,pdb} and insert that information into the database instead of opening up UDS for every book. That lets you have metadata (so sort by author works) without the super slow UDS indexing. This isn't tied to Calibre, you could generate these .metadata and .cover files on a PC from any source (including even doing what UDS does, just on a machine with a few GB of ram not a few MB...)
Hi Mr. X,

that's an interesting feature some of us have been thinking on too, but it's not so easy to implement.

For pdf and epub files, UDS will overwrite every changed field if they don't match with what it reads from the books metadata. There is a hack in Mackx's DR1000 hack to block UDS writing but it's not present in DR800+ yet.

Adding many .metadata and .cover files will pollute the file system with may new files. Also .cover files will be inserted in global.db as valid documents.

The perfect solution would be to set the proper metadata outside DR (this is not valid for covers), and then let mdbindex do its work.
Poppler library (the pdf handling library used in Linux) has an experimental branch with editing of metadata, but it hasn't released yet.

Btw, Mackx's code also includes a way to edit metadata on the DR1000.


I has been working in a ebook manager (now it's in a halted state as I don't have much free time), one of the features in the TODO list is to export books and synchronize their metadata with global.db. But it's only vaporware by now.


On the other hand, have you seen my "icons" tool. It's a tiny DR app to add covers for .desktop and .fb2 books (and also for .pdf in last release, although I don't remember if I released this last version). I wrote 2 versions, first one was in lua language, second one is in vala so as it compiles to native binay it doesn't need any external libs in the DR.
You can see the code to get some inspiration.

Finally, another crazy idea I have for the future is to convert this tool in a library and make mdbindex use it for .desktop and .fb2 files. But this is the future.

Iņigo
Iņigo is online now   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Mobiperl Mobiperl - Perl tools for handling MobiPocket files tompe Kindle Formats 871 05-30-2014 03:59 PM
Large number of books on memory card pwalker8 Sony Reader 8 03-24-2009 03:20 PM
huge pdf files on iliad yolle iRex 7 08-23-2008 11:59 AM
Fictionwise Handling Fee Against Credit Card Terms of Service? Gideon News 9 08-15-2008 03:13 PM
Huge RTF files after conversion TheMadBrewer Sony Reader 2 01-01-2007 03:55 AM


All times are GMT -4. The time now is 11:02 AM.


MobileRead.com is a privately owned, operated and funded community.