UNIX CLI workflow tools

AliceWonder · 09-07-2019, 07:32 AM

I generally prefer a CLI workflow.

And yes, I do have IBM punch cards on my desk...

Anyway - as I'm learning ePub from a CLI workflow perspective I'm taking notes so that when I forget something I can look at my notes.

I'm actually taking those notes in .xhtml files and eventually will pop them out as an ePub.

But since ePub is not ideal for the distribution of scripts, scripts I'm creating for my CLI workflow I am putting on github.

https://github.com/VeryGoodErotica/geekPub

There may be bugs, always back up before running random stuff off the Internet, but I thought they may be of interest to some here - especially the resource obfuscation script.

AliceWonder · 09-08-2019, 05:38 AM

Tested the obfuscation script and fixed a bad bug - when I read the ePub obfuscation specification, I read that the first 1024 bytes get obfuscated but its actually the first 1040 - my eyes, when they get tired, they fill in the blanks and sometimes not correctly.

But that is fixed and I tested it with several different readers.

Interestingly when I obfuscate an SVG file - it works but ePubCheck plugin in Calibre does not run. It exits w/o giving a report. The ePub files work just fine. I tested by obfuscating SVG first just to make sure broken fonts weren't being replaced by same fonts from my system.

When I obfuscate fonts (what obfuscation is intended for) ePubCheck plugin with Calibre works as expected.

JSWolf · 09-08-2019, 06:10 AM

Could it be that SVG is not meant to be obfuscated?

AliceWonder · 09-08-2019, 07:09 AM

Quote:

Originally Posted by JSWolf

Could it be that SVG is not meant to be obfuscated?

While I would agree that obfuscating SVG is rather silly, I think it is a bug in the plugin. Not ePubCheck itself as far as I can tell, uploading the ePub to websites that provide it ePubCheck, it passes validation.

If it was a violation of the spec then the ePubCheck should have complained rather than exiting w/ no information good or bad.

I do not think it is a priority bug. I may see if I can identify where it happens myself before reporting it to the maintainer.

Doitsu · 09-08-2019, 12:14 PM

Quote:

Originally Posted by AliceWonder

While I would agree that obfuscating SVG is rather silly, I think it is a bug in the plugin.

If you provide a test case, I'll update the EPUBCheck plugin.

AliceWonder · 09-08-2019, 02:41 PM

Quote:

Originally Posted by Doitsu

If you provide a test case, I'll update the EPUBCheck plugin.

Test case (6.2 kB)

https://librelamp.com/testCase.epub

Doitsu · 09-08-2019, 04:45 PM

Quote:

Originally Posted by AliceWonder

Test case (6.2 kB)

https://librelamp.com/testCase.epub

According to EPUBCheck, your test case either is an invalid epub book or you've discovered an EPUBCheck bug.

I got the following error messages for your test case:

Code:

Validating using EPUB version 3.2 rules.
java.lang.NullPointerException
        at com.adobe.epubcheck.xml.XMLParser.process(XMLParser.java:200)
        at com.adobe.epubcheck.ops.OPSChecker.validate(OPSChecker.java:150)
        at com.adobe.epubcheck.ops.OPSChecker.validate(OPSChecker.java:119)
        at com.adobe.epubcheck.ops.OPSChecker.runChecks(OPSChecker.java:107)
        at com.adobe.epubcheck.opf.OPFChecker.checkItemContent(OPFChecker.java:455)
        at com.adobe.epubcheck.opf.OPFChecker.runChecks(OPFChecker.java:159)
        at com.adobe.epubcheck.opf.OPFChecker30.runChecks(OPFChecker30.java:88)
        at com.adobe.epubcheck.ocf.OCFChecker.runChecks(OCFChecker.java:315)
        at com.adobe.epubcheck.api.EpubCheck.doValidate(EpubCheck.java:224)
        at com.adobe.epubcheck.tool.EpubChecker.validateFile(EpubChecker.java:221)
        at com.adobe.epubcheck.tool.EpubChecker.processFile(EpubChecker.java:296)
        at com.adobe.epubcheck.tool.EpubChecker.run(EpubChecker.java:158)
        at com.adobe.epubcheck.tool.Checker.main(Checker.java:31)
Messages: 0 fatals / 0 errors / 0 warnings / 0 infos

AliceWonder · 09-08-2019, 04:50 PM

It's almost certainly an EPUBCheck bug then. I'll post it there.

pdurrant · 09-08-2019, 05:00 PM

Quote:

Originally Posted by AliceWonder

Tested the obfuscation script and fixed a bad bug - when I read the ePub obfuscation specification, I read that the first 1024 bytes get obfuscated but its actually the first 1040 - my eyes, when they get tired, they fill in the blanks and sometimes not correctly.

But that is fixed and I tested it with several different readers.

Interestingly when I obfuscate an SVG file - it works but ePubCheck plugin in Calibre does not run. It exits w/o giving a report. The ePub files work just fine. I tested by obfuscating SVG first just to make sure broken fonts weren't being replaced by same fonts from my system.

When I obfuscate fonts (what obfuscation is intended for) ePubCheck plugin with Calibre works as expected.

Note that there are two versions of font obfuscation in ePubs. The one Adobe used, and the one the IDPF adopted. I can't remember which way round they are, but that's the reaosn for the 1024/1040 difference.

AliceWonder · 09-08-2019, 05:44 PM

IDPF is 1040.

I am aware Adobe has one but I know nothing about it and don't really care to, the 1024 bug I initially had was from reading the spec at W3C page wrong, technical docs are bland and my eyes get tired, that's the only obfuscation algorithm for ePub I've read so I can't blame it on the Adobe spec.

Fortunately I didn't spend too much time figuring it out, when it didn't work I read the spec again and the 1040 jumped out at me.

KevinH · 09-08-2019, 06:45 PM

The epub spec is for font obfuscation. Is your svg an image or an svg font? If the former, obfuscation in epub3 is not meant for normal svg image files.

See http://www.idpf.org/epub/30/spec/epu...nt-obfuscation

KevinH · 09-08-2019, 06:48 PM

Other questions: Was the encryptional.xml properly formed? What mime type did you assign to the svg font file (assuming that is what you used obfuscation on)?

AliceWonder · 09-08-2019, 08:58 PM

ePub spec at w3c says

``In the case of fonts, the primary use case for obfuscation''

Fonts are the primary use but the algorithm is not limited to fonts, any resource can be obfuscated (or encrypted) as long as it is listed in the encryption.xml file along with the algorithm so that readers that support de-obfuscation (or decryption) can do so. Encrypted files I believe are suppose to be compressed before encryption, but obfuscated files are not compressed before obfuscation.

mime type is image/svg+xml which correct for SVG, the algorithm does not involve using a different mime type.

encryption file is correct.

A developer of EpubCheck has acknowledged it as a bug on their github.

https://github.com/w3c/epubcheck/issues/1077

KevinH · 09-09-2019, 08:42 AM

The bug is epubcheck throwing an exception, not that obfuscation is correctly used on other resources in 3.0 spec. Perhaps this is part of 3.2 spec? The silly thing, if you do not compress that resource and instead pad it with a 1040 byte xml comment at the top, you will have achieved nothing in the form of "protection".

DRM to protect files I understand, but obfuscation here (non-font) is not something all mainstream readers are going to grok so you lose readers and gain no real protection.

KevinH · 09-09-2019, 08:55 AM

Hmm ... where exactly in the epub 3.2 spec does it say that?

I was looking in the latest version of 3.2 in the section on obfuscation here:

https://www.w3.org/publishing/epub/e...ce-obfuscation

That entire section only ever uses font examples when it talks about resource obfuscation.
If you are correct, then that is a backwards incompatible change between the epub 3.0, epub 3.01, and epub 3.2.

That plus the fact that epub 3.2 spec says it package version must report "3.0" (something they have not thought through well or some part of the spec that they forgot to change) means there is no easy way to detect between the two cases. And character data that is not compressed first (unlike binary font tables) then obfuscated (to meet the new 3.2 spec) is pretty worthless.

09-07-2019, 07:32 AM	#1
AliceWonder Connoisseur Posts: 97 Karma: 200000 Join Date: Aug 2019 Device: none	UNIX CLI workflow tools I generally prefer a CLI workflow. And yes, I do have IBM punch cards on my desk... Anyway - as I'm learning ePub from a CLI workflow perspective I'm taking notes so that when I forget something I can look at my notes. I'm actually taking those notes in .xhtml files and eventually will pop them out as an ePub. But since ePub is not ideal for the distribution of scripts, scripts I'm creating for my CLI workflow I am putting on github. https://github.com/VeryGoodErotica/geekPub There may be bugs, always back up before running random stuff off the Internet, but I thought they may be of interest to some here - especially the resource obfuscation script.

09-08-2019, 06:10 AM	#3
JSWolf Resident Curmudgeon Posts: 79,512 Karma: 145863177 Join Date: Nov 2006 Location: Roslindale, Massachusetts Device: Kobo Libra 2, Kobo Aura H2O, PRS-650, PRS-T1, nook STR, PW3	Could it be that SVG is not meant to be obfuscated? Last edited by JSWolf; 09-08-2019 at 06:15 AM.

09-08-2019, 08:58 PM	#13
AliceWonder Connoisseur Posts: 97 Karma: 200000 Join Date: Aug 2019 Device: none	ePub spec at w3c says ``In the case of fonts, the primary use case for obfuscation'' Fonts are the primary use but the algorithm is not limited to fonts, any resource can be obfuscated (or encrypted) as long as it is listed in the encryption.xml file along with the algorithm so that readers that support de-obfuscation (or decryption) can do so. Encrypted files I believe are suppose to be compressed before encryption, but obfuscated files are not compressed before obfuscation. mime type is image/svg+xml which correct for SVG, the algorithm does not involve using a different mime type. encryption file is correct. A developer of EpubCheck has acknowledged it as a bug on their github. https://github.com/w3c/epubcheck/issues/1077 Last edited by AliceWonder; 09-08-2019 at 08:59 PM. Reason: typo

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Easiest way to generate a new cover using cli tools only?	signum	Conversion	4	02-27-2015 01:25 PM
Can't locate calibres cli-tools on OSX	elementz	Calibre	3	06-13-2014 05:56 AM
Can I use Calibre's CLI tools for an app?	markvb	Conversion	5	04-15-2012 10:00 PM
Issue with german translation of CLI tools	Manichean	Calibre	4	10-24-2010 11:52 AM

09-08-2019, 05:38 AM	#2
AliceWonder Connoisseur Posts: 97 Karma: 200000 Join Date: Aug 2019 Device: none	Tested the obfuscation script and fixed a bad bug - when I read the ePub obfuscation specification, I read that the first 1024 bytes get obfuscated but its actually the first 1040 - my eyes, when they get tired, they fill in the blanks and sometimes not correctly. But that is fixed and I tested it with several different readers. Interestingly when I obfuscate an SVG file - it works but ePubCheck plugin in Calibre does not run. It exits w/o giving a report. The ePub files work just fine. I tested by obfuscating SVG first just to make sure broken fonts weren't being replaced by same fonts from my system. When I obfuscate fonts (what obfuscation is intended for) ePubCheck plugin with Calibre works as expected.

09-08-2019, 04:50 PM	#8
AliceWonder Connoisseur Posts: 97 Karma: 200000 Join Date: Aug 2019 Device: none	It's almost certainly an EPUBCheck bug then. I'll post it there.

09-08-2019, 05:44 PM	#10
AliceWonder Connoisseur Posts: 97 Karma: 200000 Join Date: Aug 2019 Device: none	IDPF is 1040. I am aware Adobe has one but I know nothing about it and don't really care to, the 1024 bug I initially had was from reading the spec at W3C page wrong, technical docs are bland and my eyes get tired, that's the only obfuscation algorithm for ePub I've read so I can't blame it on the Adobe spec. Fortunately I didn't spend too much time figuring it out, when it didn't work I read the spec again and the 1040 jumped out at me.

09-08-2019, 06:45 PM	#11
KevinH Sigil Developer Posts: 8,647 Karma: 5703586 Join Date: Nov 2009 Device: many	The epub spec is for font obfuscation. Is your svg an image or an svg font? If the former, obfuscation in epub3 is not meant for normal svg image files. See http://www.idpf.org/epub/30/spec/epu...nt-obfuscation

09-08-2019, 06:48 PM	#12
KevinH Sigil Developer Posts: 8,647 Karma: 5703586 Join Date: Nov 2009 Device: many	Other questions: Was the encryptional.xml properly formed? What mime type did you assign to the svg font file (assuming that is what you used obfuscation on)?

09-09-2019, 08:42 AM	#14
KevinH Sigil Developer Posts: 8,647 Karma: 5703586 Join Date: Nov 2009 Device: many	The bug is epubcheck throwing an exception, not that obfuscation is correctly used on other resources in 3.0 spec. Perhaps this is part of 3.2 spec? The silly thing, if you do not compress that resource and instead pad it with a 1040 byte xml comment at the top, you will have achieved nothing in the form of "protection". DRM to protect files I understand, but obfuscation here (non-font) is not something all mainstream readers are going to grok so you lose readers and gain no real protection.

09-09-2019, 08:55 AM	#15
KevinH Sigil Developer Posts: 8,647 Karma: 5703586 Join Date: Nov 2009 Device: many	Hmm ... where exactly in the epub 3.2 spec does it say that? I was looking in the latest version of 3.2 in the section on obfuscation here: https://www.w3.org/publishing/epub/e...ce-obfuscation That entire section only ever uses font examples when it talks about resource obfuscation. If you are correct, then that is a backwards incompatible change between the epub 3.0, epub 3.01, and epub 3.2. That plus the fact that epub 3.2 spec says it package version must report "3.0" (something they have not thought through well or some part of the spec that they forgot to change) means there is no easy way to detect between the two cases. And character data that is not compressed first (unlike binary font tables) then obfuscated (to meet the new 3.2 spec) is pretty worthless.

Advert

Advert