How to allow our books to be converted to all ereader formats with calibre?

[webfolk]

On our website http://www.thefolkofyore.com/ there are several books available for free reading and downloading as PDF files. Naively, as soon as we discovered calibre, we placed an invitation to our website visitors to use it to convert these files to the ebook reader of choice, including a link to the calibre website. For example: http://www.thefolkofyore.com/2-Sacre...adsacmiss.html

Of course, it's not that simple. The present PDF files don't convert well, exhibiting many of the symptoms described in the calibre manual, such as headers appearing within the text. In addition to a PDF file suitable for reading and printing from a computer, it looks like we need to offer an additional file dedicated for ebooks. To keep things simple, we'd like to offer one file in an optimal DRM-free format*that people can download and convert to their particular ebooks as quickly and easily as possible.

It would be greatly appreciated if someone would recommend a general course of action, starting with what type of file to make available for conversion, and also address a few issues that have come up so far as I've been trying to do this on my own.

The original source files are in Word 2008 for Mac. I'm using a MacBook Pro with OS 10.5.8, Adobe Acrobat Professional 8.2.5, calibre 0.7.23 and have Sigil 0.2.4, though I know little about Sigil and haven't used it yet. Since I don't own an ereader, to check results I'm using Kindle for Mac 1.0.0 Beta 1 (27214) and Barnes and Noble eReader Version 1.1. Eventually I'd like to check results in other formats, but right now it's complicated enough with these 2 alone.

So far, I've gone back into the*Word*source files and replaced direct formatting with styles, removed headers, changed section breaks from "next page" to "continuous" and removed the table of contents (since it didn't seem to be doing anything once the files were converted to ereader formats). I've tried quite a few test files, some in HTML, some in PDF which I placed in calibre to convert.*

I've tried various iterations. Converting ZIP and PDF to EPUB, MOBI, & PDB was completed in under 3 minutes. Many times*EPUB to MOBI & PDB was still working after 10 and up to 20 minutes so I stopped the job in these cases. Why is this happening? I had hoped to use EPUB*as the source format since that's*recommended in the manual and also the format Sigil uses.

The best result I've achieved so far has come from converting Word to HTML, placing that in Calibre, and converting the ZIP file to MOBI. The appearance on the Kindle reader is close to my original, though the text is justified, whereas the original is left adjusted. All the chapters show up in the Kindle Table of contents, however the foreword and preface don't.

All conversions to PDB viewed on the B&N reader are unsatisfactory because all italicization is lost. Also the text is justified in block form, double spaced between paragraphs, whereas in the original the first line of each paragraph is indented with no additional space between paragraphs.

This is a just a sample of what I've encountered. Does anyone have any suggestions?*Thanks a lot for any help you can offer.

[itimpi]

HTML will almost always be the best source format if you can get a good version. Resulting conversions should look just like they would in a browser as long as you have not used a feature not supported by the particular format. Calibre always converts any other format to HTML as an intermediate step in conversion anyway.

If using Word for the source files, then saving as Web Page (filtered) produces the best results. As to specifics of particular formats hopefully someone else will have a view.

[ldolse]

If you were going to choose a single format it should be either epub or mobi. I would suggest epub because it's an industry/community standard, and you have good control over the markup to make it look quite close to the original book.

If you went that route you could convert it to from html to epub using Calibre and then finalize your book using Sigil. If you're not comfortable with css the justification is easiest to change with Sigil, and you can tweak the Table of Contents further there.

As to why the conversion process seemed to be hanging while converting from epub to mobi/pdb, we'd need more information. You could either open a bug directly at bugs.calibre-ebook.com, with an attached file, or you could just post the output of conversion log here. You can get that by clicking on jobs in the lower left hand corner of the GUI, selecting the job that's hanging, and copying all the conversion details.


The main reasons I'd recommend mobi is because the Kindle probably makes Mobi the most widely used format, and mobi to epub conversions are generally flawless, while epub to mobi conversions depend on whether or not newer css features were used.


For either format you could get the table of contents to include the foreword and preface by changing the chapter detection xpath in the Calibre Structure detection options under conversion:

This is the default:

//*[((name()='h1' or name()='h2') and re:test(., 'chapter|book|section|part\s+', 'i')) or @class = 'chapter']

Try changing it to this:

//*[((name()='h1' or name()='h2') and re:test(., 'chapter|foreword|preface|book|section|part\s+', 'i')) or @class = 'chapter']

[Manichean]

Originally Posted by ldolse

The main reasons I'd recommend mobi is because the Kindle probably makes Mobi the most widely used format, and mobi to epub conversions are generally flawless, while epub to mobi conversions depend on whether or not newer css features were used.

I think how widespread a format is depends heavily on where you live. For example, in Europe, to my knowledge, Kindles are almost nonexistent and ePub is the most widespread format.
That being said, I'd suggest offering PDF, ePub and Mobi for download- that way, you'd have the most relevant formats all available.

[Jabby]

[QUOTE=ldolse;11play a mobi file61634]
As to why the conversion process seemed to be hanging while converting from epub to mobi/pdb, we'd need more information.

I haven't had a epub to mobi conversion hang up and quit but any conversion to mobi runs at a snails pace. Sometimes taking 6-7 minutes to complete. However, I have had the viewer hang up trying to display a mobi file.

I converted a 2.4MB epub to a 3.8MB mobi file that took at least 10 minutes (I didn;t time it). When I displayed them with the viewer the epub file was ready to read in a second or two. The mobi file took 6.5 minutes before it could be read (FBReader was ready to read the same file almost immediately). The viewers reflow is painfully slow on the file when I resize the window or when I display or hide the TOC (not so on the epub version).

I am running Calibre 7.23 and get the same result on my Windows7 or Vista machines.

Regards - John

[desertgrandma]

Welcome to MobileRead, webfolk.

You're sure to get some great suggestions here.

[DoctorOhh]

Originally Posted by webfolk

Many times*EPUB to MOBI & PDB was still working after 10 and up to 20 minutes so I stopped the job in these cases. Why is this happening?

I have had the rare epub conversion take 6 hours to complete. Every time this happens it is because the epub was created from a source originally created with MS Word. When you save Word documents as html the amount of crap that is created in the css file is astronomical. Saving the Word document as web page filtered is better but still has a lot of baggage.

[varelov]

[QUOTE=Jabby;1162645]

Originally Posted by ldolse;11play a mobi file61634

As to why the conversion process seemed to be hanging while converting from epub to mobi/pdb, we'd need more information.

I haven't had a epub to mobi conversion hang up and quit but any conversion to mobi runs at a snails pace. Sometimes taking 6-7 minutes to complete. However, I have had the viewer hang up trying to display a mobi file.

I converted a 2.4MB epub to a 3.8MB mobi file that took at least 10 minutes (I didn;t time it). When I displayed them with the viewer the epub file was ready to read in a second or two. The mobi file took 6.5 minutes before it could be read (FBReader was ready to read the same file almost immediately). The viewers reflow is painfully slow on the file when I resize the window or when I display or hide the TOC (not so on the epub version).

I am running Calibre 7.23 and get the same result on my Windows7 or Vista machines.

Regards - John

Having converted that many books, I have marveled at how good is epub in keeping the file size low...

[ldolse]

Originally Posted by Jabby

I haven't had a epub to mobi conversion hang up and quit but any conversion to mobi runs at a snails pace. Sometimes taking 6-7 minutes to complete. However, I have had the viewer hang up trying to display a mobi file.
Regards - John

The most likely reason the viewer seemed to hang is probably because it converts to whatever the file you're viewing to oeb/epub before displaying it. If you chose a file that takes that long to convert normally that's expected.

Dwanthny hit the nail on the head with the comments about formats that stemmed from word, word creates some hideous css/embedded/nested styles that takes forever to convert. Books with really clean formatting convert quite quickly. Filtered html should help, a number of users have said it's better to use OpenOffice to do the Word doc to HTML conversion instead. YMMV

[phenomshel]

Originally Posted by itimpi

HTML will almost always be the best source format if you can get a good version. Resulting conversions should look just like they would in a browser as long as you have not used a feature not supported by the particular format. Calibre always converts any other format to HTML as an intermediate step in conversion anyway.

If using Word for the source files, then saving as Web Page (filtered) produces the best results. As to specifics of particular formats hopefully someone else will have a view.

Don't forget encoding with HTML files. The encoding of the source HTML files must be the same as what calibre is expecting, or you will end up with odd errors concerning apostrophes and quotation marks. Either you will get none of both, or odd characters in place of, neither of which is ideal. See the manual for more details, here: http://calibre-ebook.com/user_manual/faq.html#id15

[dwig]

Originally Posted by ldolse

...I would suggest epub because it's an industry/community standard, and you have good control over the markup to make it look quite close to the original book. ...

... though you should avoid most attempts at format control beyond the very basic things. Trying to force many of the ePub-supported controls will lead to issues with ePub readers not uniformly supporting the features. It will also reduce the success of format conversions; speed of conversion, look of resulting ebook, and occasionally total failure of the conversion. You definitely don't want to do any formatting in an ePub that isn't supported in other formats and/or doesn't convert well in Calibre if you want users to rely on converting ePubs to their desired format.

Personally, I would recommend that you look to posting both MOBI and ePub. These will cover the vast majority of users. If your market is dominantly Europe, then it might be adequate to post only ePub.

One thing to consider is that Amazon promotes their reader toward a non-geek market, likely resulting in a larger percentage of users that aren't skilled in conversion compared to the ePub community.

[JSWolf]

Forget PDF. That's step 1.

Step 2 is to use ePub as a source format. Then when Calibre converts to something else, it does a good job as long as the ePub is well made.

[ldolse]

Originally Posted by JSWolf

Step 2 is to use ePub as a source format. Then when Calibre converts to something else, it does a good job as long as the ePub is well made.

I've found that's not neccessarily true, as epub is a superset of features found in legacy formats.

If the epub doesn't do anything complex then it's true the conversions are very clean.

However if you use any advanced layout features you can have problems. Dropcaps get screwed up. Floating elements get stuck inline, which may cause real problems in the text layout. Variable margins in Mobi are a mess, which is why there are endless threads with users asking how to fix the margins of their epub sourced files. Even if the majority of the book uses no margins if there is some content in the book with variable margins this could be a problem. Epub has left and right margins support, mobi only has left nested blockquotes/indenting... There are probably a good list of other gotchas, those are just the ones off the top of my head.

Because mobi layout features are a subset of epub if you're going to choose a single format to support mobi is probably better if you don't want to support epub conversions that may have the types of problems listed above. I also agree that two formats would be nicer (I'm a sony/epub user), but the OP did ask what is the best 'single' format to support.

[webfolk]

Thank you all for taking the time and sharing your knowledge. Each one of you has contributed to my growing understanding and knowledge and bringing this closer to a successful resolution. I've been studying your comments and, given my newness to all involved here, it's taking me some time to assimilate and act on your suggestions. As I implement them I'll post the results here.

First:

Originally Posted by ldolse

For either format you could get the table of contents to include the foreword and preface by changing the chapter detection xpath in the Calibre Structure detection options under conversion:

This is the default:

//*[((name()='h1' or name()='h2') and re:test(., 'chapter|book|section|part\s+', 'i')) or @class = 'chapter']

Try changing it to this:

//*[((name()='h1' or name()='h2') and re:test(., 'chapter|foreword|preface|book|section|part\s+', 'i')) or @class = 'chapter']

This worked superbly. It turns out epilog (which was also missing), was included in the default code I have, but spelled "epilogue", as it is in the mother tongue. Once the proper adjustments were made, all were included in the table of contents.

More to come as it does to me. Thanks again, all!

[webfolk]

After working with your suggestions, I've come to the conclusion we need to offer more than one format, which preliminarily would be EPUB because it's supported in all formats except Kindle and MOBI for the Kindle. Also because of what dwig said:

Originally Posted by dwig

Personally, I would recommend that you look to posting both MOBI and ePub. These will cover the vast majority of users. If your market is dominantly Europe, then it might be adequate to post only ePub.

One thing to consider is that Amazon promotes their reader toward a non-geek market, likely resulting in a larger percentage of users that aren't skilled in conversion compared to the ePub community.

I've gotten some good results. After making a few more changes in the source file in Word 2008 for Mac, converting that to HTML and then converting HTML to MOBI, I've gotten an e-book that looks good and works well in Kindle for Mac. The EPUB file converted from HTML looks good in Sigil and the Calibre viewer.

Some questions:

Opening the EPUB file in the Barnes and Noble viewer, the lines of text are shifted and indented strangely: many lines look like they're centered. How can this be corrected?

A PDB file converted from either HTML or EPUB viewed in the Calibre viewer or B&N reader loses all font formatting (no bold, italics, font size variations). There's no book cover or publisher logo GIF. Why is this happening?

How accurately does the Calibre viewer depict the actual book in each format? The MOBI book looks different in Calibre than it does in Kindle for Mac: for example, Calibre changes the font from serif to non-serif.

Just how poorly or how well can I expect an EPUB file to be displayed on the many different devices?

Originally Posted by itimpi

HTML will almost always be the best source format if you can get a good version. Resulting conversions should look just like they would in a browser as long as you have not used a feature not supported by the particular format. Calibre always converts any other format to HTML as an intermediate step in conversion anyway.

If using Word for the source files, then saving as Web Page (filtered) produces the best results. As to specifics of particular formats hopefully someone else will have a view.

Originally Posted by dwanthny

When you save Word documents as html the amount of crap that is created in the css file is astronomical. Saving the Word document as web page filtered is better but still has a lot of baggage.

Word 2008 for Mac doesn't offer a web page filtered option, but you can "Save as Web Page" and then select "Save only display information into HTML" Is this the same thing? It produces smaller files than the other option, "Save entire file into HTML". Even so, as you say, dwanthy, I noticed a lot of things in CSS like

@font-face {
font-family: "Courier New";
panose-1: 2 7 3 9 2 2 5 2 4 4
}

for a gazillion different fonts. Does it make any sense to delete these in Sigil, keeping references only to fonts actually used?

Originally Posted by phenomshel

Don't forget encoding with HTML files. The encoding of the source HTML files must be the same as what calibre is expecting, or you will end up with odd errors concerning apostrophes and quotation marks. Either you will get none of both, or odd characters in place of, neither of which is ideal. See the manual for more details, here: http://calibre-ebook.com/user_manual/faq.html#id15

I don't understand what to do with this. Could someone add a bit more detail as to what this means and what specific action to take?

Thanks again for all your help!