Extra metadata import from ODT

[kovidgoyal]

That's a bug. I should have used normalize() not utf8_text(). Fix committed.

[olig]

Thanks.

BTW: Do you collect sample documents? Would my ODT test cases be of interest?

[kovidgoyal]

I do collect sample documents, so your test cases will be useful. Incidentally, have you committed the use of get_metadata() to your branch? Because I tried merging and there were no changes.

[olig]

No, not yet committed, I'm still testing.

Where do I put the test file?

[kovidgoyal]

Just attach it here.

[olig]

Morning!

Still working on a new ODT Test file with some more cases than my 'real text'. While doing I found some more problems:

1) A Picture that is anchored to a page will end up in the front of the document before any text, with a page number as positioning. If you have a lot of pictures in your document anchored to pages, all of them will be the first thing in the conversion. And there is no good way to find the real place in the document. This should also produce a warning so the user knows, or perhaps I should also remove those pictures?

2) ODT sets margins double... so I will find a style like this in the input:
.P-Text_20_body {
font-size: 12pt;
margin-left: 0;
margin-bottom: 0;
margin-top: 0;
margin-right: 0;
text-indent: 1.251cm;
font-family: FreeSerif, serif;
margin: 100%;
text-align: justify
}
which becomes in 'preprocessed' (debug mode):
.P-P3 {
display: block;
font-family: FreeSerif, serif;
font-size: 1em;
margin-bottom: 100%;
margin-top: 100%;
padding-bottom: 0;
padding-left: 0;
padding-right: 0;
padding-top: 0;
text-align: justify;
text-indent: 1.251cm
}

Naturally the convert should just drop this whole 'margin: 100%' line, because it is in addition the the right margins! I suspect this is some sort of proportional setting (like margins-left-right-top-bot are X, and use 100% of this).

I will introduce a fix for this on input level, because I suspect that inside the oeb.transform fixups it's already to late. Fix will be: if there are separate margins, drop the extra margin 100% (that makes no sense at all).

I'll post a note as soon as I committed something to my branch.

[kovidgoyal]

I think it is better to have a warning and leave the pictures. Data loss is always worse than poor formatting.

IIRC, I already had code in the odt input layer to take care of the double margins, but perhaps I didn't catch all cases. bzr log on odf2xhtml.py should tell you.

[olig]

Yeah, you are right. I just saw that I called my test convert in a window that had not the devel envs set. The trunk head version just does fine.

I'll add a warning for the page anchored images in the odt.input.

[olig]

Pushed my new changes, Testfile is attached (it's odt renamed to zip, rename back to .odt to use).

A question: the OPF supports more metadata than calibre uses. Can I set those fields anyways? So for example additional contributors like editor or illustrator. By set I mean add them as fields in my ODT and then let the metadata extraction transport those fields so that they will end up in the converted output?

[kovidgoyal]

You can do that, but I doubt they will survive th conversion pipeline. IIRC the pipeline has some metadata merging code that will discard unknown metadata attributes.

[kovidgoyal]

merged.

[olig]

This is no ODT-only thing...

Now only style problems are left in my tests. Most annoying: margin and indent, which are transported 1:1 in the style. And because there are lots of styles involved (in a real world document), fixing it with extra_css is not easy.

I'm thinking about a scale_units option. Or perhaps a style_search_replace which allows regex replacements of style key/values? The second would be more flexible (but also less easy to use), because you will have a certain set of margins and so replacing would give more power than straight scaling. It would also allow replacing fixed units (cm in the ODT) with something more scalable like em. Could also be done as a flex_units option which replaces fixed margin/indent units to em by telling it what measure is 1em.

I just did a test with the last option, replacing 0.625cm with 1em (quick replace on the generated ePub stylesheet): looks very good.

Would this be an option you would accept?

[kovidgoyal]

That kind of thing strays into the territory of book editing. You have various options for that. Convert to EPUB and use calibre's tweak book feature to run search/replace on the stylesheet. Or use a dedicated tool like Sigil.

[olig]

Well, yes, one could see it that way... but isn't converting already editing? And isn't the whole 'Look and Feel' section in the manual exactly this? I see this in league with options like --remove-paragraph-spacing, --base-font-size, etc.

It should give you the power to automatically get good results. And margins that are just to big for the small screens is something I would see under the 'lets to it automatically' topic and not under eBook editing.

[kovidgoyal]

Page margins are already automatically handled. Changing margins for individual elements most definitely falls under book editing.