EPUB3: ...obfuscated fonts, but...key missing from OPF

[enuddleyarbl]

I don't know if this is a problem with Calibre 6.5.0 or with the book (it's C.S. Friedman's "When True Night Falls"). But, when I convert a book to EPUB3 (either from the library page or from within the editor), I get this when trying to open it:

calibre, version 6.5.0
ERROR: Failed to open book: Failed to open book, it has obfuscated fonts, but the obfuscation key is missing from the OPF. Do an EPUB to EPUB conversion before trying to edit this book.

If I Add the AZW3 from Amazon, I can open it in the Calibre editor without problem. Under the Fonts section, there are two red-shielded fonts (00068.otf and 00069.otf). There's no encryption.xml type file under the Miscellaneous section. The only thing font related in metadata.opf that I can see is:

<manifest>
    <item id="id1" href="fonts/00068.otf" media-type="font/otf"/>
    <item id="id2" href="fonts/00069.otf" media-type="font/otf"/>

If I right-click the AZW3 and tell Calibre to convert it (individually) to EPUB2, I can open the epub in the editor without problem. The two red-shielded fonts are still there and the encryption.xml is now under Miscellaneous. It contains:

<encryption
                    xmlns="urn:oasis:names:tc:opendocument:xmlns:container"
                    xmlns:enc="http://www.w3.org/2001/04/xmlenc#"
                    xmlns:deenc="http://ns.adobe.com/digitaleditions/enc">
                    
                <enc:EncryptedData>
                    <enc:EncryptionMethod Algorithm="http://ns.adobe.com/pdf/enc#RC"/>
                    <enc:CipherData>
                    <enc:CipherReference URI="fonts/00068.otf"/>
                    </enc:CipherData>
                </enc:EncryptedData>
                

                <enc:EncryptedData>
                    <enc:EncryptionMethod Algorithm="http://ns.adobe.com/pdf/enc#RC"/>
                    <enc:CipherData>
                    <enc:CipherReference URI="fonts/00069.otf"/>
                    </enc:CipherData>
                </enc:EncryptedData>
                
</encryption>

The fonts are still listed in content.opf, but look slightly different:

<item id="id1" href="fonts/00068.otf" media-type="application/vnd.ms-opentype"/>
    <item id="id2" href="fonts/00069.otf" media-type="application/vnd.ms-opentype"/>

The Calibre viewer will happily view either the AZW3 or the EPUB2 (and I've read the book before on both a Kindle and a Kobo Forma).

If I convert from that EPUB2 to an EPUB3 in the editor and look at the changes (before closing the editor down and re-opening it), the only instances of "font" are those same two unchanged font lines in content.opf.

Once I save, shut down the editor and re-open the epub (as an EPUB3), I get that same error message. Worse, I can't see any of the content/files in the epub.

If I run the epub through the Modify ePub PI and have it Remove Embedded Fonts, I can once again edit the book, and all the contents are there. The fonts are gone (obviously), but the encryption.xml is still there (and looks the same) and the fonts are missing from the content.opf (again, obviously).

Doing conversions like this in the previous Calibre version went fine. Is there something else I can check or post here?

EDIT: I was hoping I could check this with the previous book in the series, since I edited under Calibre 6.4.0 without problem. But, it has no fonts (and converts just fine). I'll have to look around and see if I can find any other books with embedded fonts and see how they convert under 6.5.0.

EDIT2: Still haven't found another book with fonts. But, I redownloaded this book fresh from Amazon and got the same behavior. Are those fonts supposed to have a red shield on them? Does that indicate a problem or does it indicate that the fonts are encrypted?

[JSWolf]

Originally Posted by enuddleyarbl

I don't know if this is a problem with Calibre 6.5.0 or with the book (it's C.S. Friedman's "When True Night Falls"). But, when I convert a book to EPUB3 (either from the library page or from within the editor), I get this when trying to open it:

I've taken a look at the ePub and there are no embedded fonts. I borrowed it from Overdrive, had a look, deleted it and returned it.

When you have a look at the KF8 version, what are the fonts that are embedded? You should be able to find out from looking at the font code in the CSS. Do you actually need them?

But if you do want the fonts, you can use KindleUnpack to convert the KF8 to ePub and then load it into Sigil and remove the obfuscation. You do also have to check for errors using epubcheck and fix any errors. There will be some.

[enuddleyarbl]

It looks like the font(s) is "Shift Light":

@font-face {
  font-family: "Shift Light";
  font-style: normal;
  font-weight: normal;
  src: url(../fonts/00068.otf);
}
@font-face {
  font-family: "Shift Light";
  font-style: italic;
  font-weight: bold;
  src: url(../fonts/00069.otf);
}

I don't actually need the fonts. In the past, I'd automatically delete all the font stuff when I messed with the book. I only recently started keeping it (I figured there was no down-side to keeping it (except maybe some extra size)). But, since this might be a new issue with Calibre, I thought I'd report it.

Also, I found another book with fonts (and they're red-shielded as well): Jim Butcher's "Peace Talks." It has the same problem all around. Oddly, it too uses those two Shift Light fonts. So, I'd guess it's an actual Calibre issue.

[JSWolf]

Originally Posted by enuddleyarbl

It looks like the font(s) is "Shift Light":

@font-face {
  font-family: "Shift Light";
  font-style: normal;
  font-weight: normal;
  src: url(../fonts/00068.otf);
}
@font-face {
  font-family: "Shift Light";
  font-style: italic;
  font-weight: bold;
  src: url(../fonts/00069.otf);
}

I don't actually need the fonts. In the past, I'd automatically delete all the font stuff when I messed with the book. I only recently started keeping it (I figured there was no down-side to keeping it (except maybe some extra size)). But, since this might be a new issue with Calibre, I thought I'd report it.

Also, I found another book with fonts (and they're red-shielded as well): Jim Butcher's "Peace Talks." It has the same problem all around. Oddly, it too uses those two Shift Light fonts. So, I'd guess it's an actual Calibre issue.

It's not a Calibre issue. Not at all. You have to do what I posted in my previous post if you do want those fonts.

Those shift fonts are garbage. They are from Penguin Random House for an advertisement. You can delete the fonts and you can delete that HTML file. It's the last HTML file.

[enuddleyarbl]

Well, I've now found several AZW3 books with fonts in them (not even red-shielded fonts). Calibre won't convert any of them to EPUB3 correctly. Every one of them opens with that same problem once converted. If I convert an AZW3 without fonts, it converts just fine.

So, it looks like a Calibre issue to me. I'm going to see if I can load up the previous version and see what happens with that.

[kovidgoyal]

Opena bug report and attach an epub2 file that reproduces the issue.

[enuddleyarbl]

Originally Posted by kovidgoyal

Opena bug report and attach an epub2 file that reproduces the issue.

Hopefully, I've done it correctly. Included an epub2 that you'll have to convert to epub3 to see the issue. I've marked the bug as Private because of the copyrighted attachment.

https://bugs.launchpad.net/calibre/+bug/1990185

[enuddleyarbl]

Just a wee bit more information. It's not an issue with just 6.5.0. I uninstalled it and re-installed 6.4.0. Same issue.

[DNSB]

@enuddleyarbl: Please note that the key for font encryption is very often the dc:identifier found in content.opf and anything that modifies it will disable font decryption. And yes. a calibre conversion can and will break the ability to decrypt fonts. One of the reasons I prefer to use KindleUnpack to generate epubs from a KF8/azw3 file.

See Font obfuscation in EPUBs and why you shouldn't change the dc:identifier after exporting from InDesign and/or Font Embedding for Open Container Format Files for more information.

Samples from a KindleUnpack and a calibre converted epub:


Kindle Unpack (note the epub v3.0 version):

<?xml version="1.0" encoding="utf-8"?>
<package version="3.0" unique-identifier="uid" prefix="calibre: https://calibre-ebook.com" xmlns="http://www.idpf.org/2007/opf">
  <metadata xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:opf="http://www.idpf.org/2007/opf">
    <dc:identifier>uuid:53f2730e-fc4f-43ef-9be0-1f8dcd019e01</dc:identifier>
    <dc:identifier id="uid">1607628343</dc:identifier>

calibre converted (note the epub v2.0 version):

	<?xml version="1.0" encoding="utf-8"?>
<package version="2.0" unique-identifier="uuid_id" xmlns="http://www.idpf.org/2007/opf">
  <metadata xmlns:calibre="http://calibre.kovidgoyal.net/2009/metadata" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:dcterms="http://purl.org/dc/terms/" xmlns:opf="http://www.idpf.org/2007/opf" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <dc:identifier id="uuid_id" opf:scheme="uuid">05ac5b14-e9b8-47ec-9ff6-96431094e93d</dc:identifier>

[enuddleyarbl]

Originally Posted by DNSB

@enuddleyarbl: Please note that the key for font encryption is very often the dc:identifier found in content.opf and anything that modifies it will disable font decryption. And yes. a calibre conversion can and will break the ability to decrypt fonts. One of the reasons I prefer to use KindleUnpack to generate epubs from a KF8/azw3 file....

There's definitely a difference between the dc:identifier lines from the AZW3 brought into Calibre and those from that same file run through KindleUnpack.

The KU version of a book:

<dc:identifier>mobi-asin:B0030P1W9A</dc:identifier>
    <dc:identifier>isbn:9780307575357</dc:identifier>
    <dc:identifier>calibre:461</dc:identifier>
    <dc:identifier>uuid:9280af86-82f9-4a3d-8087-e2091a2bcecb</dc:identifier>
    <dc:identifier id="uid">3359571780</dc:identifier>
    <dc:identifier>urn:uuid:f886b17f-3a7d-4826-8ad7-fcfe0c38f40a</dc:identifier>

The Calibre AZW3 version:

<dc:identifier id="calibre_id" opf:scheme="calibre">9280af86-82f9-4a3d-8087-e2091a2bcecb</dc:identifier>
    ...
    <dc:identifier opf:scheme="ISBN">9780307575357</dc:identifier>
    <dc:identifier opf:scheme="MOBI-ASIN">B0030P1W9A</dc:identifier>

The Calibre EPUB2 version:

<dc:identifier id="uuid_id" opf:scheme="uuid">urn:uuid:9280af86-82f9-4a3d-8087-e2091a2bcecb</dc:identifier>
    ...
    <dc:identifier opf:scheme="calibre">9280af86-82f9-4a3d-8087-e2091a2bcecb</dc:identifier>
    <dc:identifier opf:scheme="MOBI-ASIN">B0030P1W9A</dc:identifier>
    <dc:identifier opf:scheme="ISBN">9780307575357</dc:identifier>

And, the Calibre editor EPUB2 > EPUB3 version:

<dc:identifier id="uuid_id">uuid:9280af86-82f9-4a3d-8087-e2091a2bcecb</dc:identifier>
...
    <dc:identifier>calibre:9280af86-82f9-4a3d-8087-e2091a2bcecb</dc:identifier>
    <dc:identifier>MOBI-ASIN:B0030P1W9A</dc:identifier>
    <dc:identifier>ISBN:9780307575357</dc:identifier>

The AZW3, EPUB2 and EPUB3 versions are mostly the same (cosmetic changes). The KU version adds that calibre one, the id="uid" one and the urn one. Interesting. I'm assuming the missing urn one is the key.

EDIT1: it's odd, then, that the book seems fine as an AZW3 and EPUB2. But, as an EPUB3, with the same dc:identifiers, it fails. /EDIT1.
EDIT2: I guess, from your first link, the dc:identifiers have to be exactly the same, and, possibly, in the same order. /EDIT2.

Of course, at my Joe-Bag-O-Donuts level, I just assumed Calibre could convert anything it opened and I wouldn't have to pay attention to whether or not the book contained fonts.

I guess I'll go back to stripping all the font stuff out of the books when I play with them. Since I use my Kobo's fonts instead of the Publisher Defaults, I should be fine.

[JSWolf]

Originally Posted by enuddleyarbl

I guess I'll go back to stripping all the font stuff out of the books when I play with them. Since I use my Kobo's fonts instead of the Publisher Defaults, I should be fine.

I would not willy nilly strip fonts without checking them out first. I do strip the main font as it's not the one I want to use to read with. But there are sometimes other fonts that I do keep as they do enhance the eBook such as a font for handwritten notes (for example).

[jhowell]

Relevant to what is going on here in case it is not clear, Kindle formats do not contain an OPF. The one you see is constructed by either KindleUnpack or calibre using metadata from the AZW3 file. Each piece of software has its own method for doing it. There is no single correct way.

There is no direct equivalent to the EPUB identifier in AZW3. AZW3 format supports obfuscated fonts but uses a separate key stored with each font.

There really does seem to be a calibre bug in conversion of books with multiple identifiers and obfuscated fonts to EPUB 3 but is it only tangentially related to AZW3 as the source format.

[enuddleyarbl]

Just downloaded Calibre 6.6.1 and the fonts are coming over just fine when converting to EPUB3. Thanks!

I've got another font question, but since it's merely an informational one, I'll start another thread.