Mobipocket vs ePub: Why worse is better in ebook formats

[kovidgoyal]

Actually, now that I think some more about it, how will knowing the parent tree be enough to support absolute positioning. For example suppose that you use absolute positioning to position a semitransparent box in the upper left corner of the document as a sort of overlay. Also suppose that the <div> representing this box i sthe last element in the document tree. Now when you are rendering the first page, how will you know that the upper left corner needs a box overlay unless you've parsed the rest of the tree?

[Jellby]

I admit I am ignorant of the details, but I wonder...

When you talk about the "parse tree", do you mean (grosso modo) the set of tags in an XHTML file without their content (text)? If the location of these tags in the XHTML file is indexed, wouldn't that be enough for the renderer in order know the right format to apply (it would know the tag context for any text location)? Wouldn't it then be enough to keep this index in memory instead of the full XHTML file? How much would this index take? I guess not too much, usually a negligible amount.

[kovidgoyal]

As I pointed out, it is certainly possible to discard the text from the parse tree in memory, but that wont change the speed limitations imposed by having to parse the full XHTML file in the first place.

[Tuna]

Originally Posted by kovidgoyal

So every time either the user or some javascript makes a change to the DOM/CSS you propose re-indexing the entire tree (at least upto the current point?) For example, calibre's EPUB viewer actaually supports a reference mode that changes the DOM and the CSS of document elements on the fly in response to user interaction.

Does the Epub specification support documents that can dynamically alter themselves? If it does, then regardless of your renderer, you would have to either re-index the entire tree up to your current point or accept that your page boundaries may be inaccurate.

If your viewer allows the user to change the DOM/CSS then it faces exactly the same issues. On a PC, that's not a problem. On a resource restricted device such as a low-end e-reader, you have to ask yourself if providing such functionality is going to provide the user with a poor experience.

If your software does supports changes to the DOM or CSS, there is no reason why it cannot work to update indexes in quite an efficient manner. Certainly any change that will affect indices can be transformed into a delta and that applied over the index (which itself is a relatively small set of structures, so has minimal performance penalty). In that case, re-indexing will be far more efficient that a full re-parse of the document - which is what you're proposing.

Originally Posted by kovidgoyal

The scheme you propose might work well for static content that the user never interacts with/modifies but not for anything else. And frankly making that trade-off (removing size restrictions but making interactivity much slower) is just wrong.

Absolutely untrue - in fact quite the reverse. If you manage document updates as deltas to the index information, then having pre-indexed documents gives faster updates than un-indexed ones that will require a complete re-parse to ensure you are displaying the correct information.

[Tuna]

Originally Posted by kovidgoyal

Actually, now that I think some more about it, how will knowing the parent tree be enough to support absolute positioning. For example suppose that you use absolute positioning to position a semitransparent box in the upper left corner of the document as a sort of overlay. Also suppose that the <div> representing this box i sthe last element in the document tree. Now when you are rendering the first page, how will you know that the upper left corner needs a box overlay unless you've parsed the rest of the tree?

I think you're searching for cases - but nothing you've come up with yet invalidates the value of indexing. If you think a little more you'll realise that absolute positioning can be handled very well by pre-indexing the document and would dramatically improve rendering performance.

[kovidgoyal]

Originally Posted by Tuna

Does the Epub specification support documents that can dynamically alter themselves?

Not yet, but it will

If it does, then regardless of your renderer, you would have to either re-index the entire tree up to your current point or accept that your page boundaries may be inaccurate.

Umm this discussion has nothing to do with page boundaries. pages are an entirely optional feature of e-books, that I for one hope will go away very quickly.

If your viewer allows the user to change the DOM/CSS then it faces exactly the same issues. On a PC, that's not a problem. On a resource restricted device such as a low-end e-reader, you have to ask yourself if providing such functionality is going to provide the user with a poor experience.

Providing a user with interactivity is far more imporatnt than providing a user with page boundaries.

Absolutely untrue - in fact quite the reverse. If you manage document updates as deltas to the index information, then having pre-indexed documents gives faster updates than un-indexed ones that will require a complete re-parse to ensure you are displaying the correct information.

Wow are you seriously saying what I think you're saying here? In order to calculate a delta to an index, you would have to re-create the entire index, calculate the delta and then apply it to the on disk index. And you really expect me to believe that is faster than altering an in-memory DOM tree?

[kovidgoyal]

Originally Posted by Tuna

I think you're searching for cases - but nothing you've come up with yet invalidates the value of indexing. If you think a little more you'll realise that absolute positioning can be handled very well by pre-indexing the document and would dramatically improve rendering performance.

That was a case that invalidates netseekers claim that parsing the parent tree is enough, it has nothing to do with indexing. Indexing is just a ghastly idea for any sort of plasticity in the document.

[Tuna]

Originally Posted by kovidgoyal

Wow are you seriously saying what I think you're saying here? In order to calculate a delta to an index, you would have to re-create the entire index, calculate the delta and then apply it to the on disk index. And you really expect me to believe that is faster than altering an in-memory DOM tree?

No, I'm not saying that. Where did you learn software engineering? Any update to the document is a calculable delta which can be applied to a parse tree or index directly. (trivial example - inserting text moves indices that occur later in the document by the size of the text inserted). You can apply those deltas trivially and quickly and still have a valid index.

As for page boundaries, these are useful in any device that shows documents in page sized lumps and where the user expects to be able to go to a given page. It's a metaphor that will hang around for a very long time, not only as the one that makes sense in paper books, but also as a comprehensible measure of your location (one that makes much more sense than being 53.4% into a book!).

[kovidgoyal]

Originally Posted by Tuna

No, I'm not saying that. Where did you learn software engineering? Any update to the document is a calculable delta which can be applied to a parse tree or index directly. (trivial example - inserting text moves indices that occur later in the document by the size of the text inserted). You can apply those deltas trivially and quickly and still have a valid index.

Unless your amazing index is really just an in memory DOM tree, there's simply no way that changing it can be faster than changing the thing that it indexes directly in memory.

Also please tell me what these browsers you worked on are, so I can go uninstall them from all my machines. Seriously, where did you learn to think?

As for page boundaries, these are useful in any device that shows documents in page sized lumps and where the user expects to be able to go to a given page. It's a metaphor that will hang around for a very long time, not only as the one that makes sense in paper books, but also as a comprehensible measure of your location (one that makes much more sense than being 53.4% into a book!).

Sigh! This debate has been rehashed over and over and over and I'm not getting into it with the latest straw man, go search mobile read for the previous threads on the subject.

[Tuna]

Originally Posted by kovidgoyal

Also please tell me what these browsers you worked on are, so I can go uninstall them from all my machines. Seriously, where did you learn to think?

Perhaps we could keep the discussion on a technical level rather than name calling?

[kovidgoyal]

Originally Posted by Tuna

Perhaps we could keep the discussion on a technical level rather than name calling?

Absolutely

[netseeker]

Oh well, i guess it's time to close the discussion on those technical details.
I have a growing interest in starting an alternative project for creating a ePub reading system. I have to mull over this a little bit and will take a look into the FBReader sources (as well as into the sources of Calibres eBook-viewer ) first. I will start a thread in the ePub section of the forum to spitball ideas then.

[jgray]

Originally Posted by netseeker

Oh well, i guess it's time to close the discussion on those technical details.
I have a growing interest in starting an alternative project for creating a ePub reading system. I have to mull over this a little bit and will take a look into the FBReader sources (as well as into the sources of Calibres eBook-viewer ) first. I will start a thread in the ePub section of the forum to spitball ideas then.

Great! We could certainly use another epub reader. All of the current programs have their own set of problems. I can't help you with the programming, but I am willing to help test and provide feedback.

Although some feathers got ruffled in the process, I think this was a useful and enlightening discussion. It never hurts to have this type of discussion from people with different backgrounds and different amounts of experience.

[Tuna]

Originally Posted by netseeker

Oh well, i guess it's time to close the discussion on those technical details.
I have a growing interest in starting an alternative project for creating a ePub reading system. I have to mull over this a little bit and will take a look into the FBReader sources (as well as into the sources of Calibres eBook-viewer ) first. I will start a thread in the ePub section of the forum to spitball ideas then.

You have a lot of choices to make if you do so - what restrictions you are going to work within as well as what features and compromises you wish to support. It'd be a fun project to undertake and could really benefit e-readers in the process. I hope Korvid'll forgive me if I join in any discussions.

[netseeker]

Originally Posted by jgray

Great! We could certainly use another epub reader. All of the current programs have their own set of problems. I can't help you with the programming, but I am willing to help test and provide feedback.

Originally Posted by Tuna

You have a lot of choices to make if you do so - what restrictions you are going to work within as well as what features and compromises you wish to support. It'd be a fun project to undertake and could really benefit e-readers in the process. I hope Korvid'll forgive me if I join in any discussions.

Thank you both for your offer to participate. I'm sure kovid will forgive us because there is just nothing to forgive: This is a discussion forum and we are just discussing different point of views. He made a great job with Calibre and Calibre could also benefit from new ideas regarding ePub rendering. Let's see if it's possible to change the "Why worse is better in ebook formats" into a future "Why ePub is just the better ebook format".

"Yes, we can".

It will take a little while before i will start a new thread on this topic because i have to pick up a lot of informations first...starting a successfull and "long-term survivable" open source project isn't that easy as i know from my own experience.

Edit:

Originally Posted by kovidgoyal

Now when you are rendering the first page, how will you know that the upper left corner needs a box overlay unless you've parsed the rest of the tree?

Hm, interesting problem. I've to admit that i don't have a solution for this problem yet.