IF you could design the user interface for the next Kobo Firmware

[TechniSol]

Ken,

I think you're hung up on the whole "hierarchy of a user maintained file structure." There's no need for the user to have any reason to create, organize, or maintain an actual file or directory structure. Hell, most people have no desire to do that, or in many cases the background or desire to understand the best ways to do so. In fact, there is no way to easily change the way you'd like to browse, so you're stuck with one hierarchy once you've implemented it.

The whole point of abstraction is to allow the machine to do the work for the user and present it however the user likes without having to change the most efficient underlying physical or logical structures employed by the file system and database. Let the database on the device take on the brunt of the work. The user only need dump books into one location for side loading and let the device take on the burden of organization.

This approach eliminates the need for any thrashing around every time the system is booted beyond checking one location for new books. Simple and FAST!

All tasks related to adding or dismissing titles, or even renaming them should be handled through the database.

I'm making the case that Kobo just hasn't gone far enough with their implementation of a database to eliminate the time-wasters and log jams.

You dump your stuff into the black box and let it do the heavy lifting. It's no big deal to allow an outside library like Calibre access and it might even make sense for the outside library to impose the file structure as long as it just dumps directories into "NEW" when adding items, or has another mechanism to remove them via the database. You do away with all duplicate processing or checking by passing everything through the database.

I think Kobo Development had a good idea, they have just not implemented it completely and this leaves holes for log jams to form.

[guma]

It seems to me that the discussion here confuses two issues: (a) The ability to exclusively access content via a (real) file browser system (i.e. without reading it into a database structure first) vs. (b) the possibility to offer the user a browser-like GUI experience (after having imported everything into a database).

(a) Would get rid of the time it takes the device to build/refresh the data base after adding or changing content. (b) Would just offer a the user an alternative GUI paradigm.

While I can understand that KOBO would never consider (a), since this would basically require a complete firmware re-write from scratch, I can't see any technical reason not to implement (b). At least from a programming point of view this should be trivial.

P.S.
There is no good reason that it takes that long for the device to build/update this rather small and not overly complex sqlite data base in the first place. This lack of speed is just the result of bad/sloppy programming.

[vandafc]

Here is a proposal of how I would see the home screen organized ... Send me your comments and I will try to improve it (if I can ;-)

[speakingtohe]

Bravo on posting the pic.

Helen

[Ken Maltby]

Originally Posted by guma

It seems to me that the discussion here confuses two issues: (a) The ability to exclusively access content via a (real) file browser system (i.e. without reading it into a database structure first) vs. (b) the possibility to offer the user a browser-like GUI experience (after having imported everything into a database).

(a) Would get rid of the time it takes the device to build/refresh the data base after adding or changing content. (b) Would just offer a the user an alternative GUI paradigm.

While I can understand that KOBO would never consider (a), since this would basically require a complete firmware re-write from scratch, I can't see any technical reason not to implement (b). At least from a programming point of view this should be trivial.

P.S.
There is no good reason that it takes that long for the device to build/update this rather small and not overly complex sqlite data base in the first place. This lack of speed is just the result of bad/sloppy programming.

Exactly, on all points.

Although the parts of the firmware that would need addressing for (a) are the same parts/modules that they need to do extensive work on to get their database working properly.

Another issue might be how they are doing maintenance on the database. It appears that they do try and deal with the database all in one go. When they could just do a minimal catalog data acquisition on the file structure, at the start, and there after build data on a file as it is accessed. (Not that they should need that much from the files.)

Luck;
Ken

[speakingtohe]

Originally Posted by Ken Maltby

Exactly, on all points.

Although the parts of the firmware that would need addressing for (a) are the same parts/modules that they need to do extensive work on to get their database working properly.

Another issue might be how they are doing maintenance on the database. It appears that they do try and deal with the database all in one go. When they could just do a minimal catalog data acquisition on the file structure, at the start, and there after build data on a file as it is accessed. (Not that they should need that much from the files.)

Luck;
Ken

I would rather see all of the database upkeep, sorting, covers stored etc. at the time the books are sent to the reader, and then have it function quickly until more books are added. Other readers may take more time, or at least as much time to initialize, but then are much faster accessing books in day to day use.

Doing whatever it is they do whenever you open the shelf list for a minute or two is can be done at startup and save the interminable wait.

Helen

[TechniSol]

They could save the wait on opening shelves by re-writing the code so that they retrieve the data necessary just to build the first page of shelf view, and then display it and then continue to process whatever it is they are doing that seems to take so long. It seems like they are processing something(covers? or just searching the whole database to round up a complete list of all titles they would need if they were going to display all pages of the shelf, if so silly. What if what you want is just a few pages in? Why process all that data first?) for all the titles that will open on all the pages of the shelf prior to displaying the first one! If so, DUMB.

The other question, and it's a good one, is why are those sqlite databases so large? Based on the size reported by one poster for a given number of books they are storing 40KB on average per book? What is all that, and is it why it appears accessing the DB takes so long? That's a lot of space to store a book title, path, current page, some reading stats, and some metadata... so what else are they bogging the Db down with? If it's annotations or the like, reference them by storing a path and filename and dump them into their own text files already.

These are portable devices, they should be lean and mean and so should the databases.

[DNSB]

Originally Posted by TechniSol

The other question, and it's a good one, is why are those sqlite databases so large? Based on the size reported by one poster for a given number of books they are storing 40KB on average per book? What is all that, and is it why it appears accessing the DB takes so long? That's a lot of space to store a book title, path, current page, some reading stats, and some metadata... so what else are they bogging the Db down with? If it's annotations or the like, reference them by storing a path and filename and dump them into their own text files already.

These are portable devices, they should be lean and mean and so should the databases.

Quite a bit of the space is taken up by information on the chapters within a book.

Regards,
David

[cesarx4s]

I would add one simple feature.
An extra that would allow to take notes in text-mode.

[guma]

Originally Posted by TechniSol

The other question, and it's a good one, is why are those sqlite databases so large? Based on the size reported by one poster for a given number of books they are storing 40KB on average per book? What is all that, and is it why it appears accessing the DB takes so long? That's a lot of space to store a book title, path, current page, some reading stats, and some metadata... so what else are they bogging the Db down with? If it's annotations or the like, reference them by storing a path and filename and dump them into their own text files already.
.

There are multiple entries for each epub (one entry per html split) with a lot of properties/flags in this database - and this in two tables: tbl_content and tbl_volume_shortcovers). That could explain the DB size increase per book. Maybe this also explains the slow updating/DB-building speed: unpacking the epub files in bulk might well take some time. (But this is just speculation since I have no idea what to expect from a Linux based device with the processing power of an ebook reader in terms of file handling).

Anyway: the database, while not elegantly designed, is still small by any standard and should be easily manageable by the device - assuming reasonable programming that is ...

[guma]

Originally Posted by vandafc

Here is a proposal of how I would see the home screen organized ... Send me your comments and I will try to improve it (if I can ;-)

This sort of reminds me of the home screen layout as it used to look a few firmware versions ago ...

[davidfor]

Originally Posted by guma

There are multiple entries for each epub (one entry per html split) with a lot of properties/flags in this database - and this in two tables: tbl_content and tbl_volume_shortcovers). That could explain the DB size increase per book. Maybe this also explains the slow updating/DB-building speed: unpacking the epub files in bulk might well take some time. (But this is just speculation since I have no idea what to expect from a Linux based device with the processing power of an ebook reader in terms of file handling).

For an epub, it is one entry in the content per TOC entry plus the main row. Then the volume_shortcovers has a matching row for each of the TOC entries. For kepubs, there is an entry per TOC plus an entry for each HTML file in the manifest.

They do need to unpack the files, but they should only need to read the OPF and NCX files to get this information. The don't validate the TOC against the HTML files. I have tested this by putting an epub on the device were the TOC was nothing like the files.

Anyway: the database, while not elegantly designed, is still small by any standard and should be easily manageable by the device - assuming reasonable programming that is ...

The database also grows because of they way they use it. They don't generally update rows in the database, they do a delete and insert. Depending on the cleanup of SQLite, that can leave a lot of unused space.

[ottdmk]

Hmnn. Database programming is not my thing. I wonder if the extra entries are to help facilitate search within the book?

[davidfor]

The entries are to allow them to build the TOC and do navigation rereading the book. But, I think it is overkill now. The book is being opened to be read, so why not process the TOC at that time.

For kepubs, there is a some extra data. They store the word count for each chapter, but this is calculated the first time the book is opened. These also have the time estimates, but again, these are calculated while reading.

The only other use of the TOC rows for epubs, is recording the current position. The main row, has which of the chapters you are up to. The chapter row has where in the chapter. For kepubs, this is all stored on the main row.

I suspect these are carried over from older devices and firmware. It might have been better to do it this way with slower devices. Or they might have had an option to display the TOC without opening the book. Again, for a slower device, that might have been useful. But, apart from "don't fix what ain't broke", I don't think this is a useful mechanism now.

And while I wrote the above, a picture came into my mind: a group of Kobo developers reading my post and shaking their heads and saying "You have no idea what you are talking about".

[Terisa de morgan]

Originally Posted by davidfor

But, apart from "don't fix what ain't broke", I don't think this is a useful mechanism now.

And while I wrote the above, a picture came into my mind: a group of Kobo developers reading my post and shaking their heads and saying "You have no idea what you are talking about".

Me too...