Crossing the Rubicon into Kindle Land

[Agama]

I've been considering a new eReader for some time now and my first choice has been to stick with ePub and Sony. However, I had a hands-on with a Kindle Paperwhite today and was really impressed.

I've built up a collection of ePubs, many of which have been hand-tweaked for styling and structure and I would like to keep all this on my next reader. So my conversion question is quite general: if I convert these to AZW3 using calibre, is the formatting likely to be preserved?

For example:

1) Horizontal and vertical margins on headers and paragraphs, whether in em or pt.
2) Tables including borders.
3) Borders on DIV elements.
4) Multi-level TOC (like ePub's toc.ncx).

Are there any ePub features that cannot be converted?

[kovidgoyal]

An epub to azw3 conversion is essentially the same as an epub to epub conversion. I suggest your download Kindle Previewer, convert a couple of books and see what happens.

[Agama]

Well, that was a huge download, (>150MB), but worth the effort - thanks!

Conversion and rendering are nigh on perfect, provided I disable chapter detection and page-break insertion on the Structure Detection page.

Inspecting an exploded AZW3 shows a book very like an ePub but lacking the META-INF folder with its container.xml.

Is the separation into folders, (fonts/images/styles/text), an AZW3 requirement or a calibre convention?

Why does the stylesheet.css get split into multiple parts?

It looks to me as though the ePub could actually be restructred into an AZW3 without actually going through any conversion process - or am I missing something important here?

[Agama]

I've done a test with an ePub and restructured it into an AZW3. I simply exploded the AZW3 and deleted all the files/folders, then pasted in everything from an exploded ePub, (except the META-INF folder). Finally I renamed content.opf to metadata.opf and rebuilt the AZW3.

Kindle Previewer was perfectly happy with the result, including the TOC. This is much faster than doing a conversion. The only odd thing is that the ePub is 203KB and the AZW3 is 319KB. Looks like I can therefore write a simple plugin to do this job.

One issue I have found with ePub to AZW3 is that Kindle Previewer complains that there is a "Missing Cover Image" and that this is "an integral part of the Kindle reading experience". This happens whether calibre does the conversion or I use my restructure method.

[kovidgoyal]

azw3 is essentially epub, just in an ugly binary wrapper, that only amazon could have designed. However, the various amazon html renderes tend to have a different set of quirks from the various epub renderers, some of which only a full conversion will work around.

[DoctorOhh]

It's great that you have gone and sorted out the epub to azw3 info. I'm sure many folks might be interested.

Originally Posted by Agama

I've been considering a new eReader for some time now and my first choice has been to stick with ePub and Sony. However, I had a hands-on with a Kindle Paperwhite today and was really impressed.

I always said I wouldn't move away from Sony until I had a device that was as well integrated with calibre for collections, tags and automatic metadata updates as the Sony devices are.

When I saw that chaley & son were writing an android app (Calibre Companion) that would essentially duplicate the level of integration I experienced with Sony devices I decided to buy a Nexus 7. I didn't think I could use any backlit display as an ereader because my older eyes are very sensitive to reading for any length of time on a backlit display. I found though that with the right theme in my reading app and the brightness on my device reduced to 1% I have no problem using the Nexus 7 as an ereader. This surprised the heck out of me.

I only mentioned the above because handling collections and where books are placed on Kindles is different then you have been used too. Keep us informed on your progress and I look forward to following your possible ePub to azw3 repackaging plugin.

[Agama]

My plugin will have to use some code from the Tweak Book plugin as this must deal with the AZW3 "binary wrapper". Maybe this wrapper is responsible for bloating the file size, though I guess it must serve some purpose.

Any thoughts on the Missing Cover Image?

@DoctorOhh : I can't get on with backlit displays for reading either. I had assumed that all readers have some sort of collections management which integrates with calibre, but it sounds like I need to do some more research here. Do you have any experience with collections on Kindles?

[DoctorOhh]

Originally Posted by Agama

@DoctorOhh : I can't get on with backlit displays for reading either.

I was as surprised as anyone that the Nexus 7 on the lowest brightness level allowed me to read effortlessly. Since I had Lasik done my eyes have been even more sensitive to backlit displays. I attached my current Moon+reader theme.

Originally Posted by Agama

I had assumed that all readers have some sort of collections management which integrates with calibre, but it sounds like I need to do some more research here. Do you have any experience with collections on Kindles?

My only experience is reading folks initial frustration with setting up collections on the Kindle. There is a kindle collections plugin, but last I heard it wasn't working on the Touch or the PW. Search the forums and ask around to find out if kindle collections plugin is working with PW. I did see some threads on jailbreaking the device and setting up collections. Investigate, they may have everything worked out by now.

After reading all of the frustration I decided I would stick to Sony, which is so well integrated with calibre it is easy to assume every device has the same ease of use. Unfortunately hardly any device has the complete device integration with calibre that Sony had. The Nexus 7 using Calibre Companion is one of those devices. That may have to do with the fact that the same developer (chaley) integrated both devices with calibre.

[Agama]

I've started looking at how to do this via a plugin and noticed that something odd is going on. After deleting all the azw3 contents and copying in the ePub contents, the rebuild works fine. However, re-exploding shows that the rebuilder has done rather more than simply re-assemble the file:

1) The individual html files have been renamed and moved to a text folder.
2) The stylesheet.css has been split into 3 css files and moved to a styles folder.
3) The images have been renamed and moved to an images folder.
4) The metadata.opf has been re-written to reflect the above changes.

This must all be happening inside the Tweak Book rebuild and probably explains why this rebuild takes so much longer for azw3 than for ePub, (which only has to rezip the exploded contents.) Whilst this doesn't invalidate the idea of repacking an ePub into azw3, I need to have a look at the code in Tweek Book to see exactly what's going.

[kovidgoyal]

There are no files inside an azw3, therefore your file structure will never be preserved.

[AlexBell]

Originally Posted by kovidgoyal

An epub to azw3 conversion is essentially the same as an epub to epub conversion. I suggest your download Kindle Previewer, convert a couple of books and see what happens.

This may be off topic, but I wonder why you recommend the Kindle Previewer. I've been using calibre to convert ePub to old mobi+azw3 (ie 'both' setting) and it seems to work. Is there anything I should be careful of?

[kovidgoyal]

The recommendation was to use the Previewer to view the files not convert them.

[Agama]

The previewer has been useful in showing me that calibre will do a good job of converting my ePubs to azw3, which was my real question. I then got diverted into making an azw3 by repacking an ePub - which seems to work.

If there are no files within an azw3 then presumably calibre's Tweek Book exploder is producing the structure of folders and files which are then displayed, and the rebuild turns any structure found after tweeking back into an azw3?

[kovidgoyal]

yes .