Future-proofing my ebook collection given the churn in ebook formats

[DiapDealer]

It all boils down to html. As long as it's DRM-(and obfuscation)free, you'll always have all the options you need to make something else of it later.

[eschwartz]

Originally Posted by conan50

No.
I'm getting all my purchased ebooks from Amazon. I'm converting those to Epub at my leisure as a sort of master library of all my ebooks. I'm using Amazon's format on my Paperwhite as it is downloaded from them.
I'm not telling myself anything. As a Linux guy I appreciate open formats, and have a natural dislike of closed/proprietary formats. Just because you do not see things that way does not mean these things do not matter to others. Some of us see Epub as a safer bet for the long run of preserving our ebooks in a readable format while Amazon keeps moving toward new, less open formats like kfx.

Well, that is a reason to boycott Amazon.

• Not a reason to commit yourself to today's implementation of an open-source reverse-engineering of AZW3. (Software always has bugs, and even calibre still gets bugfixes to the AZW3 input plugin. Which are fixes you will miss out on by pretending EPUB is your master format.)
• And not a reason to use EPUB as the source for conversions. Conversions are lossy, and converting from an intermediate format is ill-advised. UNLESS, you have personally ensured the intermediate format is the NEW master format, by manually editing it to fix the problems and making it (more) perfect.
• And not a reason to start confusing the issue by mentioning KFX.

Which is what I am trying to point out.
Always keep the source material.

Don't discard or disregard the source material out of some misguided attempt to emotionally elevate "open formats".

And if you aren't falling to any of the above, but are doing the right thing...

The final and most important point of all: don't go around advising other people to put EPUB conversions on a pedestal, without explaining the whole story.

[eschwartz]

And in case I wasn't clear enough thus far: I don't think AZW3 is better, and I don't have a preference for it (in fact, I actually agree that all things being equal, EPUB is better).


I just really, really don't like doing repeated lossy conversions, regardless of how "evil" and proprietary a format might be, as long as some solution exists and will continue to exist for as long as the python programming language continues to exist, which would allow me to create all the open formats I like without throwing away valuable data (said valuable data being still unfortunately locked away in a proprietary format).

[Ripplinger]

I first remove DRM from all purchases, no matter where I buy a book from. If it's an Amazon book, I keep that original DRM-free azw3 (or azw or mobi) in my Calibre Library along with a conversion to epub since that's what is sent to my ereaders. That way I still have the original just in case Amazon ever makes an ereader I actually like and would use daily, or need to reconvert to something else down the line. And Calibre Library is backed up on several hard drives and in the cloud.

[eschwartz]

Originally Posted by Ripplinger

I first remove DRM from all purchases, no matter where I buy a book from. If it's an Amazon book, I keep that original DRM-free azw3 (or azw or mobi) in my Calibre Library along with a conversion to epub since that's what is sent to my ereaders. That way I still have the original just in case Amazon ever makes an ereader I actually like and would use daily, or need to reconvert to something else down the line. And Calibre Library is backed up on several hard drives and in the cloud.

[sherman]

I'm a create an epub, but keep the orginal file person myself.

I tried the KindleUnpack plugin for the first time (I didn't even know about it until now!), and I have to admit, for my purposes, it seems rather pointless.

Single large monolithic xhtml file? - Most likely
Inline styles? - Highly probable

Both of those issues are what prompt me to do a conversion in the first place. Calibre (usually) has the smarts to split the xhtml in logical places (such as chapters), and I prefer calibres style numbering over inline styles any day.

(note on the single xhtml file: RMSDK, at least the one on my Kobo, STILL has performance issues with long xhtml chunks, so I try and keep them under the old 260 KB limit where possible)

[Turtle91]

Originally Posted by sherman

I tried the KindleUnpack plugin for the first time (I didn't even know about it until now!), and I have to admit, for my purposes, it seems rather pointless.

Single large monolithic xhtml file? - Most likely
Inline styles? - Highly probable

Both of those issues are what prompt me to do a conversion in the first place. Calibre (usually) has the smarts to split the xhtml in logical places (such as chapters), and I prefer calibres style numbering over inline styles any day.

How many files did you try?? The Unpacker only provides the basic formatting that was there in the first place - it doesn't create it that way. If only one out of 10 of your source files has those issues then the unpacker has saved you 90% of the work...you'd only have to "convert" those files that have that intolerably bad formatting to begin with...

edit:
and the "Single large monolithic xhtml file" is usually a mobi issue...not a azw3 issue...but I already mentioned that it had some improvements I would like to see when unpacking that ancient format.

[DiapDealer]

Originally Posted by sherman

Single large monolithic xhtml file? - Most likely
Inline styles? - Highly probable

Neither of which have a single thing to do with KindkeUnpack. If the Kindlebooks were created from single, monolithic (x)html files with inline styles (or were old-style mobis), that's the output KindleUnpack produces. If the kindlebook was created with an epub that contained logically separate (x)html files with pretty and efficient css, then that's the output KindleUnpack generates.

Being primarily a tool used to reproduce (as closely as possible) the original source used to create the kindlebook (for study or for tweaking), it is doing exactly as it was designed to do.

I have no problem if that doesn't align with someone's needs. Different strokes for different folks. But please don't assume (or unintentionally spread the misinformation) that KindleUnpack is somehow responsible for the ugliness inherent to many kindlebooks that it reveals.

Although many people use it as a conversion tool (and are free to do so), it's a mistake to think that KindleUnpack was created as a hands-off conversion tool. It's an unpacker, just like the name implies.

[JSWolf]

Originally Posted by eschwartz

And in case I wasn't clear enough thus far: I don't think AZW3 is better, and I don't have a preference for it (in fact, I actually agree that all things being equal, EPUB is better).


I just really, really don't like doing repeated lossy conversions, regardless of how "evil" and proprietary a format might be, as long as some solution exists and will continue to exist for as long as the python programming language continues to exist, which would allow me to create all the open formats I like without throwing away valuable data (said valuable data being still unfortunately locked away in a proprietary format).

If you do it correctly, going KF8 > ePub is a lossless process. You use KindleUnpack to shift the Kf8 to ePub. Then you load the ePub into Calibre's eBook editor and fix any technical errors. Once that's done, you have an ePub that's the same as the KF8. The XML code is the same and the CSS is the same. So really it's as lossless a conversion as possible.

DON'T CONVERT KF8 TO EPUB! USE KINDLEUNPACK TO SHIFT THE CODE TO EPUB!

[eschwartz]

Nope, you still lose stuff. Although at that point, you're scraping the barrel with metadata I guess...

Also that doesn't help for mobi7.
And I really don't think keeping the originals is too much to ask, regardless.

But good point, nevertheless.

[Katsunami]

Originally Posted by eschwartz

Not a reason to commit yourself to today's implementation of an open-source reverse-engineering of AZW3. (Software always has bugs, and even calibre still gets bugfixes to the AZW3 input plugin. Which are fixes you will miss out on by pretending EPUB is your master format.)

Oh, bother. After extracting the AZW3, or even converting it if the extraction is not to my liking, I run the book through Modify Epub. After that, I run it through the editor, letting it fix errors, fixing the ones remaining myself, and filtering all CSS I don't need or want. Then I fill out the metadata, put in a cover image I like, and run Polish Book.

Now the book is in EPUB, has no errors as far as can be determined at that point, and has metadata which is as perfect/to my liking as humanly possible, as I put it in there myself. Starting from there, I convert the book to the reader I'm using... which in this case, coincidentally, is a Kindle, causing me to end up with an AZW3 again.

I have never had one single problem using this process, and have never encountered even a single error in over 130 books I've read on two different Kindles in the last three years.

(edit: and as long as my master format is as perfect as possible, I couldn't care less what the e-reader uses and how Calibre converts to it. I can always redo that conversion, starting from the EPUB.)

To be honest, I think there is only a negligible chance that these EPUBs or the resulting AZW3's are going to create problems down the line. Sure, if something major happens, I'll return to the original format (which I also always keep, outside of Calibre), and redo the conversion.

There is only one case I've ever encountered, and that was Calibre not being able to convert huge EPUBs (Delphi) to AZW3 without erroring out. I converted them using a command-line Kindlegen script, but when Calibre gained the ability to convert those huge books, I redid the EPUB->AZW3 conversion in Calibre itself.

And not a reason to use EPUB as the source for conversions. Conversions are lossy, and converting from an intermediate format is ill-advised. UNLESS, you have personally ensured the intermediate format is the NEW master format, by manually editing it to fix the problems and making it (more) perfect.[*]And not a reason to start confusing the issue by mentioning KFX.

I have converted/extracted AZW3's into EPUBs that were more perfect than some original EPUBs... I've bought books in EPUB that returned hundreds of warnings and errors in the Calibre editor, while I have also converted/extracted AZW3's that where squeaky clean.

Which is what I am trying to point out.
Always keep the source material.

Don't discard or disregard the source material o ut of some misguided attempt to emotionally elevate "open formats".

Discarding the original source is never a good idea, but converting and fixing/editing the books to one single format is certainly a good idea. A library becomes much easier to manage. As Calibre can handle EPUBs the easiest (some of its tools and plugins even only work on EPUBs), creating an EPUB-only master-library as perfectly as possible is, in my opinion, the best thing to do.

[sherman]

Hi guys,

I never meant to imply that the problem with the unpacked epubs was due to KindleUnpack (I know the epubs are identical to the source), but rather that a lot of the epubs are rather pointless for my purposes, so I may as well not bother.

[eschwartz]

@Katsunami, do you have a particular reason for accidentally thinking I was talking to you, or even saying something that was applicable to you?

Because I am pretty sure I already pointed out that the manual work you put in overrides the other considerations I have mentioned.
Please stop confusing the issue.

[Katsunami]

Originally Posted by eschwartz

@Katsunami, do you have a particular reason for accidentally thinking I was talking to you, or even saying something that was applicable to you?

Because I am pretty sure I already pointed out that the manual work you put in overrides the other considerations I have mentioned.
Please stop confusing the issue.

No, I had no particular reason to think you were talking to or about me, but the way you are bolding and underlining parts of your posts hint at the opinion that you are irrevocably correct in how to handle the ebook format issue.

There's always more than one way to skin a cat, and I know that you know.

The one thing I am in complete agreement though, is to keep the source material, be it EPUB or AZW3, just in case, and that you should use the Calibre editor to fix any errors in the EPUB files after conversion.

(The one thing I'd like to add is that, had Alf and Calibre not existed, I would not be reading so many non-public domain books. I would possibly not even be reading ebooks.)

[AnemicOak]

Originally Posted by Katsunami

(The one thing I'd like to add is that, had Alf and Calibre not existed, I would not be reading so many non-public domain books. I would possibly not even be reading ebooks.)

Same. If C-Lit hadn't existed back in the day and later the scripts by I cabbages, followed by Alf and friends, I would never have become as highly invested in digital books as I have.