KFX to KFX to remove DRM

[Tomifonication]

Hey all,

Historically, I've had no issues with converting KFX to AZW3 to strip DRM. My understanding now, however, is that KFX is superior to AZW3, and I should stop converting to an inferior format.

As such, I figured if I convert from KFX to KFX (assuming you must convert a KFX file for it to strip the DRM), that this would resolve the issue of retaining the integrity of the original KFX file, while stripping DRM.

This worked, but when viewing the files, the original source file is 0.90mb and the new KFX file is 0.20mb, which suggests that I'm losing 'something' during the conversion, and my intention is to retain the full integrity/quality of the original file.

So, my questions:

1. Is the only way to strip DRM from KFX to convert it?
2. If so, what can I do to prevent a loss in 'quality' when converting KFX to KFX via the Calibre KFX Input > KFX Output plugins?

Thanks!

[Tomifonication]

Update: after further reading, I've learned that KFX DRM is stripped on import, not on conversion.

Is there a way to check that DRM is properly stripped from the KFX file when I've imported it to Calibre?

[jhowell]

Originally Posted by Tomifonication

This worked, but when viewing the files, the original source file is 0.90mb and the new KFX file is 0.20mb, which suggests that I'm losing 'something' during the conversion, and my intention is to retain the full integrity/quality of the original file.

Converting from KFX to KFX actually converts from KFX to EPUB, runs that through the calibre conversion pipeline, and then converts back to KFX. Because KFX is not based on HTML, irreversible changes occur during conversion. In addition calibre can make various alterations, such as downsizing images and regenerating the table of contents.

Originally Posted by Tomifonication

Is there a way to check that DRM is properly stripped from the KFX file when I've imported it to Calibre?

If DRM removal fails you will be left with a book in KFX-ZIP format instead of KFX format. That is the easiest way to tell if something went wrong.

You can also view the book after import to verify that it is readable. See "Viewing books in KFX format" in the KFX Input documentation for ways to do that.

[Tomifonication]

Originally Posted by jhowell

Converting from KFX to KFX actually converts from KFX to EPUB, runs that through the calibre conversion pipeline, and then converts back to KFX. Because KFX is not based on HTML, irreversible changes occur during conversion. In addition calibre can make various alterations, such as downsizing images and regenerating the table of contents.



If DRM removal fails you will be left with a book in KFX-ZIP format instead of KFX format. That is the easiest way to tell if something went wrong.

You can also view the book after import to verify that it is readable. See "Viewing books in KFX format" in the KFX Input documentation for ways to do that.

Thanks very much for your reply, and amazing work on your plugins - they've been very helpful.

To give some context, this venture started because I bought a new Kindle Oasis. As per normal, I put some books into Calibre and the KFX plugin appeared to strip them of their DRM. I then altered the titles and covers of said books, and put them onto my Kindle.

When I looked at my Kindle, however, the cover art and the titles are what they were before, as if the DRM remained, or the metadata I changed in Calibre didn't carry over.

Any thoughts?

[jhowell]

Originally Posted by Tomifonication

Any thoughts?

It is hard to speculate without more detail. I need to know what you did step by step. How did you modify the book? Did you make sure that the original was not still on the device before copying the new one? Did you check both the cover thumbnail and the cover at the beginning of the book?

You may need to remove the ASIN of the book in order to prevent the kindle from using the cover thumbnail (and possibly other information) from Amazon. This will prevent it from syncing reading position, notes and highlights.

[Tomifonication]

Originally Posted by jhowell

It is hard to speculate without more detail. I need to know what you did step by step. How did you modify the book? Did you make sure that the original was not still on the device before copying the new one? Did you check both the cover thumbnail and the cover at the beginning of the book?

You may need to remove the ASIN of the book in order to prevent the kindle from using the cover thumbnail (and possibly other information) from Amazon. This will prevent it from syncing reading position, notes and highlights.

Thanks for your reply. To give you a rundown:

1. Using the latest version of Kindle for Mac, I download a Kindle book. I find its AZW in Finder and drag and drop it into Calibre.
2. After a few moments, the DRM appears to be stripped, and I'm left with a KFX file that I can edit the metadata of.
3. I edit the cover and the book description.
4. I make sure the original Kindle copy is deleted from my Oasis
5. I drag and drop the copy from Calibre onto the Oasis

After, though, the book I've now manually added to the Oasis retains the original cover and information from the eBook purchase, despite having removed the copy from the Kindle.

[Tomifonication]

Originally Posted by Tomifonication

Thanks for your reply. To give you a rundown:

1. Using the latest version of Kindle for Mac, I download a Kindle book. I find its AZW in Finder and drag and drop it into Calibre.
2. After a few moments, the DRM appears to be stripped, and I'm left with a KFX file that I can edit the metadata of.
3. I edit the cover and the book description.
4. I make sure the original Kindle copy is deleted from my Oasis
5. I drag and drop the copy from Calibre onto the Oasis

After, though, the book I've now manually added to the Oasis retains the original cover and information from the eBook purchase, despite having removed the copy from the Kindle.

Alternatively, am I wasting time trying to get the best out of KFX? Effectively I want to archive my Kindle purchases; I'm using KFX as it seemed the most sensible way to do that, given its the current standard.

[jhowell]

Originally Posted by Tomifonication

Thanks for your reply. To give you a rundown:

1. Using the latest version of Kindle for Mac, I download a Kindle book. I find its AZW in Finder and drag and drop it into Calibre.
2. After a few moments, the DRM appears to be stripped, and I'm left with a KFX file that I can edit the metadata of.
3. I edit the cover and the book description.
4. I make sure the original Kindle copy is deleted from my Oasis
5. I drag and drop the copy from Calibre onto the Oasis

After, though, the book I've now manually added to the Oasis retains the original cover and information from the eBook purchase, despite having removed the copy from the Kindle.

In those steps I didn't see anything that would actually apply the metadata you changed in the calibre database to the KFX format copy of the book. That only occurs when you use certain specific functions in calibre: "Embed metadata", "Save to disk" (with the "Update metadata in saved copies" option enabled), or "Send to device".

Unless you did one of those things the book in KFX format will retain its original metadata.

Originally Posted by Tomifonication

Alternatively, am I wasting time trying to get the best out of KFX? Effectively I want to archive my Kindle purchases; I'm using KFX as it seemed the most sensible way to do that, given its the current standard.

KFX format enables extra features and so I recommend it for the best reading experience on Kindle devices. But it is highly processed and very proprietary to Amazon.

For archival of books I recommend KF8 (azw3) format. That is the closest to what was originally provided to Amazon by the publisher and is more future-proof than KFX.

[Tomifonication]

Originally Posted by jhowell

In those steps I didn't see anything that would actually apply the metadata you changed in the calibre database to the KFX format copy of the book. That only occurs when you use certain specific functions in calibre: "Embed metadata", "Save to disk" (with the "Update metadata in saved copies" option enabled), or "Send to device".

Unless you did one of those things the book in KFX format will retain its original metadata.



KFX format enables extra features and so I recommend it for the best reading experience on Kindle devices. But it is highly processed and very proprietary to Amazon.

For archival of books I recommend KF8 (azw3) format. That is the closest to what was originally provided to Amazon by the publisher and is more future-proof than KFX.

Thanks again, I'll keep playing around. Just to pick your brain one last time, why is AZW more future-proof than KFX, considering KFX is the newer format?

[Tomifonication]

Originally Posted by jhowell

In those steps I didn't see anything that would actually apply the metadata you changed in the calibre database to the KFX format copy of the book. That only occurs when you use certain specific functions in calibre: "Embed metadata", "Save to disk" (with the "Update metadata in saved copies" option enabled), or "Send to device".

Unless you did one of those things the book in KFX format will retain its original metadata.



KFX format enables extra features and so I recommend it for the best reading experience on Kindle devices. But it is highly processed and very proprietary to Amazon.

For archival of books I recommend KF8 (azw3) format. That is the closest to what was originally provided to Amazon by the publisher and is more future-proof than KFX.

Just to elaborate, I've made sure that the metadata is being embedded in the file, and it is. Now, when I click on the file and Send to Device, the book appears on the Oasis but it has no cover at all, just a placeholder image, despite the file and Calibre both having the image.

[jhowell]

Originally Posted by Tomifonication

Just to pick your brain one last time, why is AZW more future-proof than KFX, considering KFX is the newer format?

KF8 (aka azw3) format in most cases holds the book in the form originally sent to Amazon by the publisher with only minor modification. It can usually be converted to standard EPUB format by just extracting the html contained within it.

That is not the same thing as the original MOBI (aka azw) format which is based on an old html standard.

KFX is a highly processed binary format and conversion of it to more standard formats is based on my trial and error efforts. Even the best result will be missing some of the formatting details that were originally provided to Amazon by the publisher.

Originally Posted by Tomifonication

Just to elaborate, I've made sure that the metadata is being embedded in the file, and it is. Now, when I click on the file and Send to Device, the book appears on the Oasis but it has no cover at all, just a placeholder image, despite the file and Calibre both having the image.

That is a known problem with sideloaded books in KFX format. See the documentation for the KFX Output plugin for possible work arounds.

[Tomifonication]

Originally Posted by jhowell

KF8 (aka azw3) format in most cases holds the book in the form originally sent to Amazon by the publisher with only minor modification. It can usually be converted to standard EPUB format by just extracting the html contained within it.

That is not the same thing as the original MOBI (aka azw) format which is based on an old html standard.

KFX is a highly processed binary format and conversion of it to more standard formats is based on my trial and error efforts. Even the best result will be missing some of the formatting details that were originally provided to Amazon by the publisher.



That is a known problem with sideloaded books in KFX format. See the documentation for the KFX Output plugin for possible work arounds.

Had a look, and that makes sense. Now to try and get the Extract app to work, ha! Thanks again for your assistance, very appreciated.

[jhowell]

Originally Posted by Tomifonication

Had a look, and that makes sense. Now to try and get the Extract app to work, ha! Thanks again for your assistance, very appreciated.

Others have reported problems with ExtractCoverThumbs and I haven't tried it myself in a long time. I believe it may only find books in the "documents" folder, not in subfolders.

The next release of calibre should be able to create thumbnails for books in KFX format that are sent using "Send to device". I don't know when that will be available.