DE upgrade ePub problems

[Steven Lyle Jordan]

Attached is a test file that (still) won't open in my DE.

[Madam Broshkina]

Originally Posted by Steve Jordan

Attached is a test file that (still) won't open in my DE.

It did not open for me either, but when I ran it through Calibre the epub file that Calibre generated opened just fine.

[JSWolf]

Steve, the problem is with WinZIP. I downloaded the ePub file and decompressed it and then recompressed it with WinRAR and it works perfectly with DE for Windows.

Also, due to the size of the HTML file, it doesn't work with Mobile DE on the 505.

[Steven Lyle Jordan]

That's very odd:

• I used the same version of WinZip to create 2 other ePub files. I haven't upgraded it.
• I checked the zip settings, and they hadn't changed.
• Also, I changed the file extensions from .epub to .zip, and back to .epub, without actually opening them in WinZip.

What the heck could have happened?

Originally Posted by JSWolf

Also, due to the size of the HTML file, it doesn't work with Mobile DE on the 505.

Hmm. Maybe the extra graphics caused that...

[wallcraft]

Originally Posted by Steve Jordan

Maybe the extra graphics caused that...

The problem is the single 759KB HTML file. ADE on the Sony has a 300KB per HTML file limit.

As Madam Broshkina says, if you run it through Calibre's any2epub it splits the HTML file into 5 pieces and also produces a correctly ZIPed epub. On the command line, use:

any2epub -o test_calibre.epub test.epub

The reason this matters is that each HTML file has an automatic page break before it in the ebook. So Calibre's automatic splitting may not be what you want.

If the items that in the TOC should be chapters (i.e. should have page breaks before them) then it is best to actually put each in a separate HTML file. Calibre can do this for you using the --chapter switch as explained in post #6 above.

[Peter Sorotokin]

Steve,

I have looked at the file and this is what I found. Your epub file is not quite proper zip file. Normal zip files should start with the sequence "PK\3\4" and your file starts with "PK00". This might be a marker that zip splitting is allowed - I am not sure, at least many zip utilities can process it.

In any case, this is not a valid epub file. OCF spec has a very strict rule on the extra headers in zip format (they are prohibited). I have looked at our code and I am pretty certain that this file could not have ever worked in Digital Editions (1.0 or 1.5). I think something changed in your process. Look carefully into WinZip options.

Peter

[kovidgoyal]

@Peter: I've noticed that ADE in general doesn't seem to believe in the philosophy of being liberal in what it accepts as input. You really should change that. For example, I've found that it refuses it display epub files that have html components that dont declare a namespace. I know that such files are not valid as per the EPUB spec, but, while being so zealous in following spec is welcome in a tool to create epub files, it is out of place in a viewer application.

And are there any plans to release a linux version sometime soon, or am I going to have to roll my own viewer?

[Peter Sorotokin]

Originally Posted by kovidgoyal

@Peter: I've noticed that ADE in general doesn't seem to believe in the philosophy of being liberal in what it accepts as input. You really should change that.

Hmm, I'd call it "sloppy", not "liberal". Why do you think it's a good idea?

For example, I've found that it refuses it display epub files that have html components that dont declare a namespace.

Yes, we either need a namespace or XHTML DTD (OPS DTD would work as well).

I know that such files are not valid as per the EPUB spec,

XHTML spec as well, I think.

but, while being so zealous in following spec is welcome in a tool to create epub files, it is out of place in a viewer application.

Unfortunately, my experience is that creation tools will make use of any sloppiness that they can get away with. And I don't think there are any disagreements that content slopiness is bad for everyone.

And are there any plans to release a linux version sometime soon, or am I going to have to roll my own viewer?

By all means, roll your own! I cannot give you any hints on what is planned when (there is a strict company-wide policy on public disclosures of that kind).

[kovidgoyal]

Originally Posted by Peter Sorotokin

Hmm, I'd call it "sloppy", not "liberal". Why do you think it's a good idea?

Because it places a lot of unneccessary burden on content creators for no real benefit. Things like requiring zip file headers be exact or that namespaces be explicitly declared are going a little beyond the level of neccessary strictness.

Yes, we either need a namespace or XHTML DTD (OPS DTD would work as well).



XHTML spec as well, I think.

Yeah but surely if DE encounters an HTML file without a namespace declaration it can assume that it is XHTML or HTML 4.01 (particularly since if I just insert a bogus namespace declaration claiming the file is XHTML (even though it is not) DE is perfetly happy (as it should be).

Unfortunately, my experience is that creation tools will make use of any sloppiness that they can get away with. And I don't think there are any disagreements that content slopiness is bad for everyone.

Ensuring correctness of files should be the job of validators like epubcheck, not viewers.

By all means, roll your own! I cannot give you any hints on what is planned when (there is a strict company-wide policy on public disclosures of that kind).

Sigh, I was hoping I wouldn't have to bother, but, I guess since once I do this, the viewer will automatically work for LIT and MOBI files as well, makes it worthwhile. In any case DE is a little too heavyweight for a simple viewer app.

[Steven Lyle Jordan]

@Jon: I tried WinRAR late last night, using it the same way I'd use WinZip. The ePub file came out fine for me, too.

I'm still at a loss to understand why WinZip worked for me once, then not now, when I know I haven't changed anything. I'll have to look into WinZip closer, to see if something changed inadvertently.

@Peter, I didn't know anything about the "PK00" sequence. But everything else about the ePub files was the same with the older and latest sets, and the older sets opened fine. As another recent example, I have an ePub of "6 of One..." on my site, which opens fine in DE, created with the same tools. (It's free, so feel free to check, compare and contrast.) How do I check the older epub documents to see which "PK" sequence is on them?

@Wallcraft, my earlier ePubs were broken up into individual HTML chapters. I didn't bother for the original test piece, but I believe the final test piece I posted was a Book Glutton ePub in one big file, shifted to .zip and back to .epub. (I've done so many tests, even I have to go back to make sure!) Anyway, any future novels will be in individual chapters.

@Kovid, I may still try to use Calibre, but I haven't gotten it to format properly yet... my initial attempts met with errors that I am still trying to work out (and which I may get back to you directly with questions, after I've exhausted the help file).

[wallcraft]

Originally Posted by Steve Jordan

I didn't know anything about the "PK00" sequence. But everything else about the ePub files was the same with the older and latest sets, and the older sets opened fine. As another recent example, I have an ePub of "6 of One...", which opens fine in DE, created with the same tools. (It's free, so feel free to check, compare and contrast.) How do I check the older epub documents to see which "PK" sequence is on them?

Use a text editor to display the first few characters.

test.epub has:

PK00PK^C^D^T^@^@^@^H^@¨*;9oa«,^V^@^@^@^T^@^@^@^H^@ ^@^@mimetypeK,(ÈÉLN

6of1.epub has:

PK^C^D
^@^@^@^@^@`«Ü6oa«,^T^@^@^@^T^@^@^@^H^@^@^@mimetype application/epub+zipPK

A "good" ePub needs the "mimetypeapplication/epub+zipPK" starting at character 31, and it needs the "PK\3\4" initial string, rather than "PK00PK\3\4" which seems to be some extended ZIP header (extended ZIP is bad here). Also test.epub has compressed the contents of the mimetype file (application/epub+zipPK) and so it has the wrong string starting at character 31 (in fact mimetype starts is at character 35, presumably because of PK00PK vs PK).

It is hard to see why any ePub viewer would care about this, since its built in unzip probably still works. It it does care, then it needs to be explicit about what the problem is.

[kovidgoyal]

Originally Posted by Steve Jordan

@Kovid, I may still try to use Calibre, but I haven't gotten it to format properly yet... my initial attempts met with errors that I am still trying to work out (and which I may get back to you directly with questions, after I've exhausted the help file).

Sure, calibre's epub support is far from mature (its only a week old) so any feedback is welcome.

[Peter Sorotokin]

Originally Posted by kovidgoyal

Because it places a lot of unneccessary burden on content creators for no real benefit.

Sticking namespace declaration is not unnecessary burden. Namespaces in XML are just as important as names. This is a slippery slope, the next thing that we will be asked to support is case-insensitive tags, then non-well-formed XML, etc.

Things like requiring zip file headers be exact or that namespaces be explicitly declared are going a little beyond the level of neccessary strictness

We do not require zip headers to be exact, but we do not want to add extra code to support zip features that the spec specifically disallowes.

Yeah but surely if DE encounters an HTML file without a namespace declaration it can assume that it is XHTML or HTML 4.01 (particularly since if I just insert a bogus namespace declaration claiming the file is XHTML (even though it is not) DE is perfetly happy (as it should be).

HTML 4.01??? Why would we do HTML? We do not even have to carry HTML parser at this point.

We do not go extra mile to weed out the broken content, but we won't go extra mile to support the broken content either. (Both are bad for the battery, BTW).

Ensuring correctness of files should be the job of validators like epubcheck, not viewers.

I agree, but neither should it be the job of the viewer to accomodate broken content. BTW, this is pretty much unanimous sentiment in IDPF, because lax viewers lead to sloppy content proliferation and sloppy content makes it hard to write a good viewer, as you have to mimic the undocumented behaviour of the existing viewers.

Sigh, I was hoping I wouldn't have to bother, but, I guess since once I do this, the viewer will automatically work for LIT and MOBI files as well, makes it worthwhile.

The only bad thing here is that it would distract you from other work that you do in this area.

In any case DE is a little too heavyweight for a simple viewer app.

[kovidgoyal]

Well, to each his own, but, I have a question for you with regards to html2epub. I find that ADE doesn't resize text if absolute font sizes are used (e.g. px or pt). So I've written code to convert absolute font sizes to relative font sizes. But I had to reverse engineer the algorithm ADE uses to calculate the final rendered font size. Can you either give me that algorithm or look at my code and tell me if it is correct?

Basically the algorithm I use is:

ptu = { # Convert to pt
                  'px' : 1.0,
                  'pt' : 1.0,
                  'pc' : 1/12.,
                  'in' : 72.,
                  'cm' : 72/2.54,
                  'mm' : 72/25.4,
                  }

val = (val * ptu[unit])/base

where val and unit are the original font size and unit respectively. This algorithm converts font sizes in px, pt, cm, mm, pc, in to a % font size.

In my testing the values in ptu above most closely replicate final rendered font sizes.

[Peter Sorotokin]

Originally Posted by kovidgoyal

I find that ADE doesn't resize text if absolute font sizes are used (e.g. px or pt). So I've written code to convert absolute font sizes to relative font sizes. But I had to reverse engineer the algorithm ADE uses to calculate the final rendered font size.

Yes, this is a common problem. Digital Editions EPUB engine is pretty much a CSS 2.1 text processor. When user changes the font size we simply feed different font-size value to the root element and then it gets inherited according to CSS rules. There is nothing special that we do.

To keep text resizable you pretty much always have to use em unit or percentage. For most content, it is sufficient to just scale the absolute value to the fraction of the default font size (which I think is somewhat what you are doing). However this does not work well if there is a block of text that uses a particular font size and a subblock in it that also sets its own font size. The problem there is that per CSS rules relative font size is given relative to the parent element, so relative font size for the inner element has to be given relative to the outer element, not relative to the default size. When size comes from CSS, this may be pretty hard to fix.

Longer term, I think we need to support CSS3 "rem" unit which is font size relative to the root element. It would work much more reliably, but we'll need to get IDPF blessing for that.