Future-proofing my ebook collection given the churn in ebook formats

[Fbone]

I add books to Calibre in bulk so it's easier to auto convert when adding. That way I'm sure that the DRM is gone.

The tools didn't work consistently in 2013/2014. I didn't know this until I later went to open a Kindle freebie. Further examination revealed about 30% still had DRM attached. There was a timing issue with the DRM-removal process. I sent a bug report to Alf and they fixed the issue in an update a few months later. By that time Amazon updated K4PC and I had to redownload thousands of books one by one. I just recently finished.

To prevent that from reoccurring I auto convert when adding to Calibre. It's easier than opening them up individually.

[pwalker8]

Originally Posted by Fbone

I do this to ensure that the DRM has been removed.

When I load an Amazon book in Calibre, it's saved in a mobi format. I then convert it to epub so I can read all my books in Marvin. When you get books from multiple formats, like I do, then it makes sense to save books to a standard format. I also agree with eschwartz that you want to retain the books in the original formats. I keep my books in three different directories (Kindle, Baen and Calibre) Kindle has the original azw/tpz downloads from Amazon, Baen has the zip files from Baen, calibre has all the books. I actually have a fourth directory, the Sony directory, but since I haven't bought a book from Sony in years, that's simply an archive directory.

[darryl]

Originally Posted by maximus83

Thanks all for the discussion, very useful insights.

Some responses in no particular order:

* Somebody well known here on Mobileread pointed out in a PM that while Amazon may not be an actual monopoly, they could sure use some competition to be more open in their formats. IMO, competition is good for everybody, including consumers and Azon itself. So for me, it makes sense to continue my plan to buy from sellers that are trying to use a format based on an open spec, like Epub. Hoping to eventually nudge Azon in a more "open" direction.

* That said, it's ALSO true what others noted here: you can still convert most if not all Amazon ebooks today if you can obtain the AZW3 file. So for now, there's no harm in getting Azon ebooks provided that you can still convert them into a usable "open" file format.

* While there's not a total consensus on the formats issue, even here at Mobileread as evidenced in the older "format future proofing" threads that somebody linked to, it does seem that of all the imperfect file specs and formats available today, epub probably has the broadest industry and library support, and the most going for it. Being in epub doesn't solve all problems (since everybody implements it differently and then devices also render it differently). But since epub3 is directly based on HTML5 and CSS3, it's at least based on universal W3 web standards and moving in the right direction, basically "converging on a standard." Storing your ebook files in epub today seems to make it most likely that you'll be able to use open source tools in the future to get your files converted to whatever format you need.

I would prefer that we be very clear about what is necessary for future-proofing and what has other motivations, conscious or otherwise.

1. A boycott of Amazon to nudge it in a more open direction is not a ridiculous idea. However, those choosing to do so incur a cost, likely paying higher prices, no access to many of the increasingly Amazon exclusive Indies, and dealing with usually much inferior "competitors". All for something which will attract little support and has little chance of having any effect. Such a boycott stems from a laudable moral/political position, but does nothing for future-proofing.

2. Not only is there no harm in buying Amazon ebooks. They are as accessible as ever. The difference is that if you have a more modern reader than a Paperwhite 1 then some books delivered directly to that device will be .kfx. However, I have not seen a single example of a .kfx book that cannot be downloaded in an Alf-friendly format, either through K4PC or direct download. Yes, Amazon can change this readily with a K4PC version supporting .kfx, and stopping direct downloads, leaving direct delivery to older devices the only way to obtain an Alf-friendly format. However, they have not done so yet, may not do so, and a K4PC version supporting .kfx may well assist with the "cracking" of the .kfx format. Meanwhile, over in the epub world, we have Adobe with a so far uncracked drm system which it would already have made compulsory had it not discovered that its customary disregard for criticism would not fly in this instance. But that also can change effectively overnight. But it hasn't changed yet. No one outside of Amazon seems to like .kfx, but why single out Amazon. Right now either books from Amazon or Adobe Epubs remain Alf-friendly, and rationally plays little role in choosing so far as future-proofing is concerned.

Finally, there is no reason related to future proofing to prefer epub. Once liberated from DRM, either format can be read by open source tools. It is a fantasy to pretend that you will not be able to read a drm-free ebook in azw or mobi format in the future. As for epub3, personally I would prefer .azw3, not for future-proofing but because I personally don't like the format.

The discussion is worthwhile and timely, and I thank you for bringing it up. However, I think that your positions in some respects have little or nothing to do with future-proofing and stem more from current ideology, some of which I understand and sympathise with. However, I do think we should be clear on which is which.

[eschwartz]

@darryl

Like I said,

But it didn't answer the question of "how to future-proof".

Important to keep clear about what is what.

[Turtle91]

Originally Posted by Fbone

I add books to Calibre in bulk so it's easier to auto convert when adding. That way I'm sure that the DRM is gone.

Just wanted to point out that doing a "conversion" from azw3 to epub is not the cleanest way to go. It is much cleaner, and much faster, to simply unpack the azw3 into the epub format using the KindleUnpack PI.

Although the technical details are interesting, and there's more to it than this, the simplified way to think about it is that the unpacker removes the azw3 "wrapper" around the internal epub without having to send the file through Calibre's converter. I can unwrap 10 azw3s in just a few seconds...with no "calibrization" of the internal css!

The unpacker will also work on mobi files, but only one at a time, and then you have to import the resultant files into Sigil or Calibre Editor, or otherwise wrap them up in an epub. It would be really nice if this functionality was improved little...

Cheers!

[Katsunami]

I've been thinking about this issue for some time now, and have mostly come to the same conclusions as you have. I also think EPUB (version 2) is the most future-proof format, having the best chance of still being able to be be read and written many years down the line.

What I do is the following:
- Import books into Calibre, removing DRM in the process.
- Export the book (Save to Disk), and stash the DeDRM-ed original somewhere else.
- If not EPUB: convert/unpack (in case of azw3) to EPUB.
- Use the Calibre tools to remove errors and/or change the book as I see fit.
- Convert the book to AZW3 (as I have a Kindle).

So yes, my main format is EPUB, even if the original book is an AZW3 that would be usable on my Kindle without any changes.

[mbovenka]

Originally Posted by JSWolf

I've read of some people keeping an older unused Kindle registered to their account just so they can use the USB download function without having to worry about KFX sometime in the future.

Yep. I still have my DXG registered to my account, even if a friend of mine is now actually using it. Used to be mostly for my Analog subscription, but now that Alf supports K4Android, that's not really neccessary any more. Still, with those KFX shenanigans, I'm keeping it...

[eschwartz]

Originally Posted by Katsunami

I've been thinking about this issue for some time now, and have mostly come to the same conclusions as you have. I also think EPUB (version 2) is the most future-proof format, having the best chance of still being able to be be read and written many years down the line.

What I do is the following:
- Import books into Calibre, removing DRM in the process.
- Export the book (Save to Disk), and stash the DeDRM-ed original somewhere else.
- If not EPUB: convert/unpack (in case of azw3) to EPUB.
- Use the Calibre tools to remove errors and/or change the book as I see fit.
- Convert the book to AZW3 (as I have a Kindle).

So yes, my main format is EPUB, even if the original book is an AZW3 that would be usable on my Kindle without any changes.

Well, that's different.

You've modified the EPUB manually, it is now your primary, master, source format.

[Katsunami]

Originally Posted by eschwartz

Well, that's different.

You've modified the EPUB manually, it is now your primary, master, source format.

What does the OP do then; only convert from AZW3 to EPUB, and then back to AZW3 if needed? I can understand that if he'd want an EPUB as main file, but then I'd just unpack the AZW3 with KindleUnpack.

I myself want to have every book as perfect as possible, with EPUB 2 as main format, and after creating that, I also convert to AZW3. That should settle me for at least the next 10-15 years or so.

(Perfect for -me-, that is: as I use the original publication date, not the one of the e-book, but I do use the publisher of the e-book, and not the original one, if they're different. I use a cover I like, not necessarily the one that belongs to the said e-book or ediiton the e-book is based on... etc. Also I use Modify EPUB and the Editor to remove and/or fix stuff.)

[Tarana]

Maximux83, your best bet is to simply keep your book in the format that you bought it in. Just make sure the DRM is removed and then you can convert it to whatever format you need in the future. I think you are overthinking the situation.

If you have books in non-standard formats, convert them to epub. That is the easiest format to fix things, in any case.

Also, don't count on the cloud. There have been at least 2 services that went belly up and closed down with no warning to their users who PAID for storage. Use it as a convenience and possibly a secondary backup, but don't count on it to be there.

At some point in the future, maybe I'll have to convert all the mobi books to epub or AZW. That day isn't here yet and if it came tomorrow, Calibre would let me convert.

[ProfCrash]

Originally Posted by eschwartz

Cool, an actual K1 owner!

I knew you existed...

I have the original box, cover, and everything. I don't use Tavar anymore but I cannot see myself ever giving him up.

Originally Posted by DiapDealer

I didn't back up my physical books. Fire or flood could have wiped me out at any time. Therefore, I'm not going to be overly concerned with backing up or future-proofing my ebooks either. I don't own very many that I haven't already read anyway. *shrug*

Yeah, this is me. I had backed up my books until a year ago but I realized that I have never used anything other then a Kindle or the Kindle App to read. I tried Marvin and liked it but was not so enthralled that I felt I needed to keep using it. I guess I am a low maintenance reader. And I have a bunch of Kindles and the Kindle App so accessing my books is easy enough.

I am not worried about Amazon collapsing so I have not been saving for that reason.

Who knows, maybe this weekend I'll download my library on to the new computer and think about converting but who knows.

[Fbone]

Originally Posted by ProfCrash

Yeah, this is me. I had backed up my books until a year ago but I realized that I have never used anything other then a Kindle or the Kindle App to read. I tried Marvin and liked it but was not so enthralled that I felt I needed to keep using it. I guess I am a low maintenance reader. And I have a bunch of Kindles and the Kindle App so accessing my books is easy enough.

I am not worried about Amazon collapsing so I have not been saving for that reason.

I would do something similar if I owned multiple Kindles. I wouldn't remove DRM, back-up or even use Calibre. Buy and read.

However, I'm 100% dependent on K4PC and, therefore, my future proofing is more labor intensive.

[conan50]

I've wrestled with this same issue, and made a few posts about it. Where I'm at right now is:
Nonfiction: I buy a hardcover version as I usually want to keep these.
Fiction: I buy it if it is drm-free, otherwise where possible get it via Overdrive. I don't necessarily need to keep or will always reread fiction.
I convert what I can to Epub. I figure it has the longest shelf life of all formats, and is my personal preference to have a master library in an open format.
I seriously considered only buying from Kobo and Google Books, but Amazon is the 800 pound gorilla you just can not get past if you are buying ebooks. They have a bigger selection, sometimes a better price, and getting books directly to an ereading device via wifi is just too painless to resist.
This also decided my reading device of choice: Paperwhite 3. I know files can be converted for Kobo, but there is a point at which convenience trumps all for me, and Amazon has done its job too well in that area.

[eschwartz]

Originally Posted by conan50

I've wrestled with this same issue, and made a few posts about it. Where I'm at right now is:
Nonfiction: I buy a hardcover version as I usually want to keep these.
Fiction: I buy it if it is drm-free, otherwise where possible get it via Overdrive. I don't necessarily need to keep or will always reread fiction.
I convert what I can to Epub. I figure it has the longest shelf life of all formats, and is my personal preference to have a master library in an open format.
I seriously considered only buying from Kobo and Google Books, but Amazon is the 800 pound gorilla you just can not get past if you are buying ebooks. They have a bigger selection, sometimes a better price, and getting books directly to an ereading device via wifi is just too painless to resist.
This also decided my reading device of choice: Paperwhite 3. I know files can be converted for Kobo, but there is a point at which convenience trumps all for me, and Amazon has done its job too well in that area.

So you use off-the-shelf, lossy conversions to EPUB as a base format, then reconvert the EPUB to AZW3 for sideloading to your Kindle, all with the same software tool, and then tell yourself "my library now has a longer shelf life"?

Sure. Whatever floats your boat.

[conan50]

Originally Posted by eschwartz

So you use off-the-shelf, lossy conversions to EPUB as a base format, then reconvert the EPUB to AZW3 for sideloading to your Kindle, all with the same software tool, and then tell yourself "my library now has a longer shelf life"?

Sure. Whatever floats your boat.

No.
I'm getting all my purchased ebooks from Amazon. I'm converting those to Epub at my leisure as a sort of master library of all my ebooks. I'm using Amazon's format on my Paperwhite as it is downloaded from them.
I'm not telling myself anything. As a Linux guy I appreciate open formats, and have a natural dislike of closed/proprietary formats. Just because you do not see things that way does not mean these things do not matter to others. Some of us see Epub as a safer bet for the long run of preserving our ebooks in a readable format while Amazon keeps moving toward new, less open formats like kfx.