Adequacy of EPUB?

[Hitch]

Originally Posted by Sarmat89

That's blatantly false.

Oh, dear. You wound me, sir, by maligning my friend Toxaris unjustly. To quote Lord Peter Wimsey, in, I believe, "Have His Carcase," you drive me to the vulgarity of reminding you who I am.

I'm not, by any means, the most expert bookmaker on this forum. Far from it. Nor am I a programmer. I bow and defer to the wonks and geeks here, whose work amazes me. I'm not even the most successful person, by far. I am, however, pretty much the only one here that runs a conversion business full-time. Other people here do, of course, create eBooks professionally, but the point I'm about to make is about numbers.

My company has, in the last seven years, formatted and converted more than 3,000 books. I believe that gives me some standing to address that statement.

And I can tell you in absolutely no uncertain terms that publishers do not give two craps about metadata. For every book we've done, since about mid-2009, I've handed out our metadata sheet, to be completed by the Publisher and returned to us for inclusion in their books.

Now, ask me exactly how many have come back finished? How many have come back, at all? How many have come back with naught more than "title, name, and year of copyright" completed?

The answers to those questions are--about 20%; about 30%; nearly 70% never return the metadata sheet at all, and of the 30% that are returned to us for inclusion, most are simply the title, name of the author, and year of copyright. Period.

If EPUB mandates the use of DC vocabulary in plain text form, it is problem of EPUB. There are simply no alternatives to it.
Because they cannot rely on EPUB metadata. With clearly defined scheme, that support will be trivial.
"John Smith", "Arthur Conan Doyle", "Johannes van der Waals" and "Murakami Haruki" all parse in different way, and there is no way for library software to do that automatically without errors.

And how, exactly, would you control this? How would you use a "dropdown" or SAAS, or whatever you have in mind, that would prevent someone from listing an author incorrectly? Or would not allow them to skip the meta? And if you had such a rigid system, how many self-pubs do you think would use it? Want me to tell you?

You need to hang out at some other fora, where self-pubs hang out. That "95%" that you think you're talking about, in your post about our workout book. (P.S.: you're wrong about that, too, but...<shrug>). Go to the KDP forums for self-publishers, and hang out wherever those Smashwords publishers hang out. See what those folks say about formatting their books. The same people that find uploading a Word file to Amazon, or Smashwords, "too hard."

That's the bulk of self-publishers. You err, sir, in assuming that all self-pubs can manage what you're describing, or can afford to have a company like mine format their books. Many using systems like "Draft2Digital" to clean their HTML, instead of doing it themselves. If these folks can't manage to upload a clean Word file to Smashwords, how on EARTH do you expect them to use the system you're proposing?

And, again: how will you stop people from doing things like typing "xxxxxxx" for any of the text-based metadata? Author name? First name? Description? Year of Copyright, for that matter? I'm an old database girl, and I know firsthand that there's only so much error-trapping you can do when it comes to text. That has nothing to do with whether we're discussing XML, XHTML, or Bob's Big Book Format.

Nothing you're discussing will make things like, "all covers designed by Fred Smith" viable for you to search upon, because, as Toxaris correctly pointed out, publishers don't care. And frankly, by and large (at my guess, 95+%), neither do most readers. The metadata that they are interested in is what's already available via the Amazon, Nook, iBooks, webpages. They have precisely zero interest in whether or not the metadata about the minutiae is correct, inside the book.

Oh, well. I suppose that suffering through this thread is better than attending to my annual agony of getting our Holiday Greeting card done. Zounds, procrastination writ large.

Hitch

[Hitch]

Originally Posted by HarryT

You'd be a fool to dismiss what Hitch says. She is the most experienced ebook creator you're ever likely to come across. You really should treat what she tells you with a little respect.

Harry:

You really are too kind. Trust me when I say that there are at least 10 people here on this forum, regularly, who are far more experienced than I; and, at least, four people in my company who could have me for breakfast. Mostly, I'm the carnival barker these days, not the attraction inside the tent.

Nonetheless, I genuinely thank you for your kind words.

Hitch

[Sarmat89]

Originally Posted by Hitch

How would you use a "dropdown" or SAAS, or whatever you have in mind, that would prevent someone from listing an author incorrectly? Or would not allow them to skip the meta? And if you had such a rigid system, how many self-pubs do you think would use it?

That's demagogy, as any system is expected to give better results than what we have now: a text field.

If these folks can't manage to upload a clean Word file to Smashwords

Now you are getting inconsistent: first you tell about people who need special formatting and '400+ images', now you are telling about 'people uploading Word files'.
It does not matter whether they are using converters or ask third parties to format books for them: in any case results achieved by a specialized format will give better results than a badly made EPUB and still better results with a proper EPUB.

how will you stop people from doing things like typing "xxxxxxx" for any of the text-based metadata?

Why would they do that, again, when there is a convenient and understandable form to do that properly?

[Sarmat89]

Originally Posted by Hitch

bookmakers outputting pure XML files, which would allow the users to style them however they want, using their own XSLT sheets (defaults, presumably, that would be installed on the devices)

Exactly. The reader software is not required to use XSLT or HTML as intermediary formats. At the end it will be styled page elements.

[Sarmat89]

Originally Posted by HarryT

She is the most experienced ebook creator you're ever likely to come across.

Another EPUB problem is that you need to have "experienced book creators" to work around the weirdness of EPUB format and device quirks. A good book format is cross-platform, and put the burden of displaying it on the reader, not the book author.

[eschwartz]

Originally Posted by Sarmat89

That's demagogy, as any system is expected to give better results than what we have now: a text field.

You'll have to think up better insults than "text field". Your proposition also utilizes text fields.
(Named text fields. But our text fields are already named, you just want to give them different names and mark them as mandatory.)

Now you are getting inconsistent: first you tell about people who need special formatting and '400+ images', now you are telling about 'people uploading Word files'.
It does not matter whether they are using converters or ask third parties to format books for them: in any case results achieved by a specialized format will give better results than a badly made EPUB and still better results with a proper EPUB.

Well, her point is that both groups exist.
And you are targeting the Word-users, I think -- the ones who don't put care into the books.
But your format doesn't stop them doing stupid things (and in short order someone will probably write a program that automatically turns a badly-formatted DOCX into a badly-formatted file in your "specialized format").

You have repeatedly made the assertion that your proposed format is better, but you have not backed up your assertion with evidence.

...

Also, on behalf of EPUB, I take offense at your claim that EPUB isn't a specialized format.

Why would they do that, again, when there is a convenient and understandable form to do that properly?

Because they don't give a damn?
If they don't give a damn about doing a good job now, why would they give a damn if it was more convenient?
As Hitch said:

Originally Posted by Hitch

My company has, in the last seven years, formatted and converted more than 3,000 books. I believe that gives me some standing to address that statement.

And I can tell you in absolutely no uncertain terms that publishers do not give two craps about metadata. For every book we've done, since about mid-2009, I've handed out our metadata sheet, to be completed by the Publisher and returned to us for inclusion in their books.

Now, ask me exactly how many have come back finished? How many have come back, at all? How many have come back with naught more than "title, name, and year of copyright" completed?

The answers to those questions are--about 20%; about 30%; nearly 70% never return the metadata sheet at all, and of the 30% that are returned to us for inclusion, most are simply the title, name of the author, and year of copyright. Period.

I think that is pretty damn convenient of Hitch, but somehow despite being convenient, she rarely gets it filled out...

Are you truly arguing with extensive, professional experience in the exact scenario you are raising?

[Toxaris]

Originally Posted by Sarmat89

Another EPUB problem is that you need to have "experienced book creators" to work around the weirdness of EPUB format and device quirks. A good book format is cross-platform, and put the burden of displaying it on the reader, not the book author.

As a reader, I don't want that burden. I want to read. I expect that the publisher where I buy the book from will ensure that my reading experience is good. It is not different when I buy an physical book.

The metadata (or lack of) will not disturb my reading experience.

You only read what you want to read and pick lines from responses and are behaving pretty daft. Try to read and understand the answers instead of glancing at them. Then again, based on your responses in this thread, I will not get my hopes up.
Let's just say you have your opinion, just don't confuse an opinion with the hard cold truth of the real world. Let me just state it again.
* Publishers don't care about metadata
* Almost all readers don't care about metadata
* Publishers are fine with the current format for most types of books, exception being perhaps poetry.
* Almost all readers are fine with the current format for most types of books.
* XHTML can be use semantically if you want. Nothing prevents that.
* You can add any metadata in a book you want
* You can separate first name and last name. If it is more difficuly, the file-as property can help.

You will probably not read this anyway, but fine.

[pdurrant]

Originally Posted by Hitch

Instead of having an XML file with a schema that is not carried with it, but is, instead, divined by the individual device, based on the user's desires, the schema (the CSS, effectively) travels therewith.

Just a minor, technical point. CSS is not a schema for XML. CSS in fact has nothing to do with XML, and CSS files are not written in an XML format.

XML itself just defines the syntax of tags (the angle brackets, the way attributes of tags are given, etc.). A schema defines the structure of an XML file - what tags are available, and when they can be used. A parser for XML files can verify that an XML files matches the specifications given in a schema.

CSS is a way to apply styling to XHTML tagged elements. Not a schema.

By calling for XML, the OP is saying nothing. XML documents require a schema. Which is what DocBook provides.

Everything the OP wants is provided by DocBook - meaningful tags for document elements, for many, many kinds of documents.

And yet the OP has rejected DocBook. And XHTML (another XML implementation).

I think the OP really needs to go away and do a few months research on what's currently available, and on the meanings of the terms XML, Schema, DTD, XTHML, CSS, and especially, DocBook.

[Hitch]

Originally Posted by Sarmat89

That's demagogy, as any system is expected to give better results than what we have now: a text field.

NO. Your imaginary and not-thought-through system will give you UNDEFINED "better results," because you are obsessed with metadata, and forcing everyone else to march to the beat of your drum, even though, as I've mentioned, and many others here have also mentioned a) 99.99% of all publishers AND readers don't give two hoots about metadata, and b) as we've also explained about 100x, your issue isn't with ePUB; it's with the DC definitions. As near as I (or anyone else) can tell, all you actually care about is having the metadata that YOU want available so you can use it to organize and quantify the books that you're reading.

Now you are getting inconsistent: first you tell about people who need special formatting and '400+ images', now you are telling about 'people uploading Word files'.

MWAHAHAHAHA. That's because, padawan learner, that's exactly what the bookmaking world is comprised of; books that can be uploaded as Word files, books that require adequate formatters and books that require very, very experienced formatters to create them. I, for one, don't believe that you actually make eBooks. Not commercially, not for others, and certainly not in any real volume. Your blithe dismissal of many of the things that have been said here is a clear beacon indicating a serious lack of experience.

It does not matter whether they are using converters or ask third parties to format books for them: in any case results achieved by a specialized format will give better results than a badly made EPUB and still better results with a proper EPUB.

So you declare, without any proof whatsoever. What we have here is the "Pie in the Sky" (acronym: PITS) format. An imaginary format, constructed entirely in your own mind, for which you have still failed to give any real meaning, structure, or anything else. You adamantly oppose DocBook, when you clearly have never used it, don't understand it, and don't realize that it's pretty much exactly what you've been alleging you want.

Why would they do that, again, when there is a convenient and understandable form to do that properly?

You mean, in your imaginary, unbuilt, unmapped format? Possibly because, like me, they don't have the information from their author or publisher? If you think that commercial bookmakers will go along with something that's this nuts--"oh, if you don't put in the name of the cover designer, you can't proceed," you are absolutely, 100%, positively WRONG. None of us is going to be willing to be stuck in some inflexible pipeline, because our clients--more than 99% of them--aren't going to give us all the metadata we ask for. Your imaginary plan is to have the bookmakers STUCK and on the hook, awaiting their clients, for your data. Let me think about that...uh, NO.

Originally Posted by Sarmat89

Exactly. The reader software is not required to use XSLT or HTML as intermediary formats. At the end it will be styled page elements.

MWAHAHAHAH. Oh, brother. You think that that will fly, with millions of readers who don't even know how to sideload a file to their device, much less style the book they've bought?

Originally Posted by Sarmat89

Another EPUB problem is that you need to have "experienced book creators" to work around the weirdness of EPUB format and device quirks. A good book format is cross-platform, and put the burden of displaying it on the reader, not the book author.

No, in fact, you are completely and irrevocably wrong. Millions--literally--of Amazon publishers have managed to create MOBI files by uploading a Word file. Mobi and ePUB share 98% identical "format DNA." Hundreds of thousands of ePUBs have been made by authors using Word files at B&N. I don't know how many, but certainly, hundreds of thousands--possibly a million--authors have made ePUBS out of their Word files at Smashwords.

Experienced book creators aren't needed to work around "the weirdness" of ePUB format and alleged "device quirks" (iBooks, anyone?). It has nothing to do with weirdness or quirks. Some books are simply vastly more complex than others. That's not "ePUB weirdness." Nor have you provided ANY freaking indication AT ALL why your proposed PITS would be better. Or how it would, even in your imagination, somehow be free of needing advanced bookmaking to deal with these same layout elements.

For sample, a sidebar, or pull-quote. The PITS that you've proposed is, for all intents and purposes (wait for it) DocBook. The XML uses an XSLT to transform it to--wait for it--ePUB. That's how that XML file is subsequently displayed on a device. How is it that you think that PITS would magically make the creation of a sidebar "easier?"

As near as I can tell, your idea is, when bookmaking, I'd simply call the main content <body> and the sidebar <sidebar>, and then if millions of readers didn't see it right, the onus is on THEM? I'm supposed to say, "well, tough s**t, kiddies, you have to tell your device how to display the sidebar, and you didn't, so nyah nyah nyah?" When they're telling their device how to display "all" sidebars, @Sarmat89, how do they know how small, or how large, sidebars have to be for different books? How do they know how large or how small the heading next to it is intended to be? By PFM? Magic? I've attached just a handful of images (guys: warning--these images are large-ish, set to 600w). as an exemplar. And these are a mere handful; not particularly complex, not spectacularly simple. A typical moderate-to-heavily-formatted ePUB. How would you possibly envision doing these with XML, leaving the rendering up to the individual reader? If you can pass an XSLT with the file--fine. But leaving it up to the individual reader is simply cracked. It reeks of inexperience.

The problem as I see it here is that you've gone as far as thinking that XML is some magic cure-all for what you perceive as "hard stuff" in ePUB, and your desire for more metadata. You have given precisely zero thought to how it actually implements. You've provided all of us with exactly zero real, concrete examples as to how your idea is "better;" you simply keep saying that it IS. You obviously didn't spend a lot of time on your school's debate team, either, or you'd know that simply reasserting your opinion, over and over, without any data or facts or ANYTHING else to back it up, is worthless.

Hitch

[Toxaris]

He has already given the answer how your example pics should be done, with PDF. Because they are an exception in his mind. So, I would think one PDF per reader and for each possible resolution to get the best results. That would indeed be much, much better than ePUB...

[eschwartz]

Originally Posted by Toxaris

He has already given the answer how your example pics should be done, with PDF. Because they are an exception in his mind. So, I would think one PDF per reader and for each possible resolution to get the best results. That would indeed be much, much better than ePUB...

Just like old times.

Get off my lawn...

[Hitch]

Originally Posted by Toxaris

He has already given the answer how your example pics should be done, with PDF. Because they are an exception in his mind. So, I would think one PDF per reader and for each possible resolution to get the best results. That would indeed be much, much better than ePUB...

Yes, of course, you are quite correct. .

Books like these are about 50% of my annual business. If he thinks that these are exceptions, it's only more proof that his experience with real books, in a real book business, is extraordinarily limited. Fiction books are, by and large, simplicity itself.

Originally Posted by eschwartz

Just like old times.

Get off my lawn...

Indeedy. I realize that I'm just wasting my time here. But I suppose it's better to pseudo-vent here than at my clientele. It's a good thing that you guys are here to keep me from going right off the deep end.

Hitch

[Hitch]

Originally Posted by pdurrant

Just a minor, technical point. CSS is not a schema for XML. CSS in fact has nothing to do with XML, and CSS files are not written in an XML format.

Yes, of course, you are entirely correct. I was not correct; I was endeavoring to explain that for all intents and purposes, ePUB, (which has XHTML as the base, and CSS to tell it how to display) as a delivered file to an e-reader, is the equivalent of sending an XML file + an XSLT file, for rendering, to a device. Conceptually. Not...not didactically. Does that make sense? That the two would be effectively equivalent, regarding what the end user would see.

XML itself just defines the syntax of tags (the angle brackets, the way attributes of tags are given, etc.). A schema defines the structure of an XML file - what tags are available, and when they can be used. A parser for XML files can verify that an XML files matches the specifications given in a schema.

Yup, yup, yup, no argument.

CSS is a way to apply styling to XHTML tagged elements. Not a schema.

Troo dat. I was really just saying that he, the OP, is calling for the "half dozen of another" when calling for XML, if the plan is to have XML interpreted at the device level with XSLT that would, presumably, be provided by the device-makers, as defaults. (Six of one...)

By calling for XML, the OP is saying nothing. XML documents require a schema. Which is what DocBook provides.

Everything the OP wants is provided by DocBook - meaningful tags for document elements, for many, many kinds of documents.

Yup.

And yet the OP has rejected DocBook. And XHTML (another XML implementation).

Repeatedly. I agree with you, 100%. DocBook is exactly what he wants. He just refuses to admit it. I think it's because DocBook, for the most part, was a failed format/mechanism. (Excluding Dag, who is clearly a steely-eyed ePUB Pilot). Ergo, he wants to claim that his new XML format/schema would someone work "better" than DocBook. When we all know, that won't happen. If XML was going to scorch the earth and fly the friendly skies of eBook-making, that would have happened with DocBook, which was supported in rather large numbers by the wonk community. And still is, for heaven's sake.

I think the OP really needs to go away and do a few months research on what's currently available, and on the meanings of the terms XML, Schema, DTD, XTHML, CSS, and especially, DocBook.

Indeed. +1,000. Welcome to the DocBook chorus. :-)

Hitch

[Sarmat89]

Originally Posted by Hitch

books that require very, very experienced formatters to create them.

And those are minority if we talk about non-/fiction books.

DocBook

For a book, you need what DocBook don't have, and many features are not needed for books.
Also, you cannot choose an arbitrarily complex format like HTML/CSS or DocBook, as the clients will not be able to implement it fully. You need a clear, compact and comprehensive format that everyone can implement in its entirety.

Possibly because, like me, they don't have the information from their author or publisher?

That is demagogy, again. Every book has the cover artist mentioned in its colophon, along with other copyright holders. The problem is that you need a dedicated person in the library to extract it and put into custom metadata. It would be an obvious decision to include this in format itself to enable machine processing.

if you don't put in the name of the cover designer, you can't proceed

You cannot use other people's work without attribution, you know.

Millions--literally--of Amazon publishers have managed to create MOBI files by uploading a Word file. Mobi and ePUB share 98% identical "format DNA."

You need to have some level of structure in your Word file too, to get a hyperlinked TOC required for Kindles and the TOC itself.
Therefore, same methods can be employed to create a semantically structured document.

The XML uses an XSLT to transform it to--wait for it--ePUB. That's how that XML file is subsequently displayed on a device.

That's just one method for simple readers. There are readers that can interpret XML formats directly.

When they're telling their device how to display "all" sidebars, how do they know how small, or how large, sidebars have to be for different books? How do they know how large or how small the heading next to it is intended to be?

Basing on screen dimensions and user font preferences. If you need a sidebar to occupy exactly 43% of screen width and have a 1.113 em margin, you are doing that wrong and that is not the kind of markup which is cross-platform and cross-device.

And these are a mere handful; not particularly complex, not spectacularly simple.

##1 and 3 are not the typical books: there are color photos incompatible with E-Ink screens.
The other have a simple layout: headings, illustrations with captions, unordered lists, inline images, footnotes. Fonts and colors are for user to choose. Also, a reader rendering the semantically-defined footnote could take measures not to break the line before its marker.

You've provided all of us with exactly zero real, concrete examples as to how your idea is "better;"

To make a sidebar, or an illustration caption, with XHTML, you need to implement the HTML layout. Not all readers and devices can do it.
To render a semantic sidebar, you can use any model your reader is working with: you already know what you need to do, you don't need to copy the physical formatting to imitate it.

[Toxaris]

Originally Posted by Sarmat89

For a book, you need what DocBook don't have, and many features are not needed for books.

Please explain what you need that DocBook doesn't have/support.

Originally Posted by Sarmat89

There are readers that can interpret XML formats directly.

Please name them. We know none.

Originally Posted by Sarmat89

To make a sidebar, or an illustration caption, with XHTML, you need to implement the HTML layout. Not all readers and devices can do it.

Yes, they can. Name the ones that cannot. Please do not name Coolreader, we already established that Coolreader does not honor the specifications.

Put your money where your mouth is and come up with straight answers.