Starting with EPuB, 4 questions (to start out with).

[JSWolf]

Originally Posted by ProDigit

My impression was that almost all of the base iso-8859-1 characters display exactly the same as the utf-8 ones.
The alphabet, numbers, spaces, basic punctuation....
I've seen books on sale in the store, where publishers use the iso-8859-1 format, so I presume that utf-8 or utf-16 isn't a requirement, but iso-8859-1 is also a valid option...

I mean, I would want to make a distinction between encoding 'preferences' and 'requirements'. Requirements mean that unless it's not there, the book will error or not display on an ebook reading device.

If you use an ISO-8859-1 encoding and use that ePub as your source to sell at Amazon, your eBook will not properly display.

The reason publishers have gotten away with incorrect encoding is because ADE forced the ePub to be rendered with UTF-8. Other programs that read ePub might not do so and they would have an incorrectly displayed ePub.

I've also seen some cases where the wrong encoding causes the ePub to display in bold where it should not have. This was on a Sony Reader 505 using an old version RMDSK (ADE). So no, do not thing that the wrong encoding is OK just because whatever software you are seeing the eBook in looks like it should. Not every program will handle the wrong encoding like it was the correct encoding,

[ProDigit]

When I did a search online, I did see how utf-8 is really taking over since the Millennium on web pages.
Seems like IBM introduced a lot of the coding work with ISO-8859-1, and the decline of it, and rise of UTF-8 seems to be linked with the decline of IBM, and rise of microsoft and apple, finally turning the tables with android being introduced to the market.

The numbers/stats correlation might be coincidental, meaning it might be not directly related to IBM, but is still noteworthy that the newer, non-industrial companies, providing software for the home, were introduced at a time ISO-8859-1 was in decline.

[Toxaris]

It has nothing to do with IBM, but with internationalization. More and more websites and applications (don't even mention databases) need to be able to handle multiple languages and characters. The old ISO pages cannot handle that due to restrictions. They invented all kind of tricks in the beginning, creating nightmares (I have seen enough). The adoptation of Unicode solved a lot, but it was a slow process. Funny that you mention Microsoft, since they were actually quite late to the party with regards to applications and Windows.

Quite frankly there is no reason at all at the moment not to use UTF.

[dgatwood]

Originally Posted by Toxaris

It has nothing to do with IBM, but with internationalization. More and more websites and applications (don't even mention databases) need to be able to handle multiple languages and characters. The old ISO pages cannot handle that due to restrictions. They invented all kind of tricks in the beginning, creating nightmares (I have seen enough).

This. ISO-8859-1 is one of the old ISO standards for character encoding. In the ISO standards, each character was exactly one byte long, which meant that it could represent only 256 possible character values. Of those, the English letters, numbers, punctuation, and control characters took up the first half, leaving only about 127 usable slots to represent language-specific characters.

Because of that limitation, different language families required different encodings. You couldn't, for example, have French text and Greek text in the same document. Also, you couldn't support languages like Chinese at all, because there are many thousands of Chinese characters in common use.

Worse, you couldn't determine just by looking at a file whether it was ISO-8859-1, ISO-8859-2, ISO-8859-16, etc., so you had to externally specify the character set in some way, or else French could turn into a gibberish Greek-English hybrid. In the case of HTML, they specified the character set in a meta tag, which meant that the browser would read the file up to that point, say "Oh, crap, I'm using the wrong encoding", then reread the entire file using the right encoding. It was an absolute mess, and I'm being generous here.

The Unicode standard fixed all of this by supporting over a million possible characters. As a result, you can use a single character set to represent text in every language on Earth. With Unicode, you can have Chinese text, English text, French text, and Russian text on a single web page.

UTF-8 is the most common way to encode Unicode text, because English characters, numbers, and punctuation are encoded using a single byte, making English UTF-8 content fully backwards-compatible with software that supports traditional 7-bit ASCII (and ISO-8859-*).

[Freeshadow]

Not to forget codepage workarounds like mazovia encoding because the codepage for Polish had the block graphic symbols screwed up.

So yes. The pre-UTF times were definitely absolutely not funny.
First, as soon as you linguistically left western Europe things tended to became ugly or needlessly complicated.
Second, a working multilingual setup was something to dream about.

[ProDigit]

The question is, if you really need it?
Without looking at any statistics, I would say English books make up of over 50% of the world's books, most of them coming from USA, but also some from Australia, Great Britain, and other parts of the world.

Again, without seeing any real statistics, I'd guess that 99% of these books don't need to use anything above the ISO-8859 standard.

I'm trying to think about how many books have I read before, that used more than one language (except perhaps an occasional short sentence paraphrased)?
Almost none, but I do see the potential for Latin writers writing in English, and having some "Spanglish" moments in their writing.

I'm not trying to advocate for ISO-8859-1, but just mentioning that some people can make a big deal out of using UTF-8, when in reality, the book could have had the ISO-5889 coding and still looked exactly the same.

But it is what it is... Big corporations determine what we need, even if we don't, and the flock just has to follow along...

Kind of reminds me of my cellphone. Android decided that those phones really needed 64 bit, and a processor in it, so the phone barely lasts one day on a charge,
while I still could think most apps and browsing on mobile, could still be done on 16 bit operating systems (32bit already was an unnecessary overhead).

[DaleDe]

Originally Posted by ProDigit

The question is, if you really need it?
Without looking at any statistics, I would say English books make up of over 50% of the world's books, most of them coming from USA, but also some from Australia, Great Britain, and other parts of the world.

Again, without seeing any real statistics, I'd guess that 99% of these books don't need to use anything above the ISO-8859 standard.

I'm trying to think about how many books have I read before, that used more than one language (except perhaps an occasional short sentence paraphrased)?
Almost none, but I do see the potential for Latin writers writing in English, and having some "Spanglish" moments in their writing.

I'm not trying to advocate for ISO-8859-1, but just mentioning that some people can make a big deal out of using UTF-8, when in reality, the book could have had the ISO-5889 coding and still looked exactly the same.

But it is what it is... Big corporations determine what we need, even if we don't, and the flock just has to follow along...

Kind of reminds me of my cellphone. Android decided that those phones really needed 64 bit, and a processor in it, so the phone barely lasts one day on a charge,
while I still could think most apps and browsing on mobile, could still be done on 16 bit operating systems (32bit already was an unnecessary overhead).

It is not just language differences. You need curly quotes, emdash, ndash, and a lot of special characters to make an eBook look like a book. See our wiki. UTF is the only way to go.

Dale

[eschwartz]

Originally Posted by ProDigit

The question is, if you really need it?
Without looking at any statistics, I would say English books make up of over 50% of the world's books, most of them coming from USA, but also some from Australia, Great Britain, and other parts of the world.

Again, without seeing any real statistics, I'd guess that 99% of these books don't need to use anything above the ISO-8859 standard.

I'm trying to think about how many books have I read before, that used more than one language (except perhaps an occasional short sentence paraphrased)?
Almost none, but I do see the potential for Latin writers writing in English, and having some "Spanglish" moments in their writing.

I'm not trying to advocate for ISO-8859-1, but just mentioning that some people can make a big deal out of using UTF-8, when in reality, the book could have had the ISO-5889 coding and still looked exactly the same.

Just because it is often sometimes safe to confuse the two, doesn't mean that given the choice you should.

ISO-8859-1 and UTF-8 are pretty interchangeable, if all you are using is ASCII...


So given that there is no downside whatsoever to using UTF-8, it is very difficult to know why anyone would want to avoid it.
And at the end of the day, that is why people "can make a big deal out of using UTF-8".

[ProDigit]

I wasn't looking forward to learn CSS, but now I think I'll make my epub in a program, and just modify (trim) the files inside the epub, to make it smaller, and repack it.
Anyone has a suggestion on a very basic epub creator program?
I have downloaded Sigil, but wanted to know if there may be simpler versions around?

[AnemicOak]

Originally Posted by ProDigit

Anyone has a suggestion on a very basic epub creator program?

Either Sigil or the Calibre Editor.

[PeterT]

And remember if you use a tool like Sigil (or calibre editor) the CSS will just contain what you put in it; no more, no less.

[Doitsu]

Originally Posted by ProDigit

I wasn't looking forward to learn CSS, but now I think I'll make my epub in a program, and just modify (trim) the files inside the epub, to make it smaller, and repack it.

No matter what editor you decide to use, check out Toxari's ePub Optimizer which is available as a standalone program and a Sigil plugin. Among other things it offers lossless image optimization and font subsetting.
Both Sigil and Calibre Editor also support running Python scripts. I.e., you could write your own Python script to automatically run additional cleanup tasks.

[eschwartz]

calibre has a new builtin feature to losslessly optimize images using jpegtran or optipng, available from either the Editor or Polish book. And has long offered a font subsetting feature.

So it isn't just "Sigil or extra standalone programs".

[Toxaris]

Originally Posted by eschwartz

calibre has a new builtin feature to losslessly optimize images using jpegtran or optipng, available from either the Editor or Polish book. And has long offered a font subsetting feature.

So it isn't just "Sigil or extra standalone programs".

Please be aware that a lot of people, myself included, do not like the UI (and other things) of the Calibre Editor. It is all personal preference.

[crankypants]

Originally Posted by Toxaris

Not true. GIF is supported, as well as SVG for a large part. In the specs it is specified what part of SVG is not supported. In general the animation part.

GIF is supported in EPUB2? Where does it say that? Do you have a link?