Scrambling copyright ebooks to help troubleshoot problems ???

[JSWolf]

One suggestion, don't scramble characters with a high-bit because you never know if they can be part of the problem.

[eschwartz]

Originally Posted by WT Sharpe

It is my belief that as long as the story itself is scrambled, other elements such as Title, Author, ISBN, Table of Contents, & etc. should not be a problem. As this is not something that has come up before, I can't guarantee you that the current policy will never be amended in light of future consequences, but I can say you will have no trouble from the mod team for posting it.

[jackie_w]

Originally Posted by WT Sharpe

It is my belief that as long as the story itself is scrambled, other elements such as Title, Author, ISBN, Table of Contents, & etc. should not be a problem.

OK, thank you. That's quite a lot less strict than originally thought
- OK to reveal the TOC
- stop worrying about stray ISBNs in unusual places
- no need to remove the links to external websites, if any

Originally Posted by WT Sharpe

As this is not something that has come up before, I can't guarantee you that the current policy will never be amended in light of future consequences, but I can say you will have no trouble from the mod team for posting it.

Understood.

[jackie_w]

Originally Posted by JSWolf

One suggestion, don't scramble characters with a high-bit because you never know if they can be part of the problem.

In English, please. Do you mean things like mdash and smart quotes, or something different?

Punctuation isn't currently scrambled at all.

[PeterT]

I think he means characters with an ordinal # greater than 127.

[jackie_w]

Originally Posted by PeterT

I think he means characters with an ordinal # greater than 127.

Chars like ÅÉçčè, you mean? That might be quite a lot of unscrambled text, e.g. European books. I don't think that would be allowed for upload.

[Turtle91]

Originally Posted by jackie_w

Chars like ÅÉçčè, you mean? That might be quite a lot of unscrambled text, e.g. European books. I don't think that would be allowed for upload.

Who's to know if they were part of the original...

Especially if "high-bit" characters are part of the replacement algorithm.

[Jellby]

Originally Posted by WT Sharpe

It is my belief that as long as the story itself is scrambled, other elements such as Title, Author, ISBN, Table of Contents

The decision was that the Table of Contents (chapter headers, etc.) must be scrambled too, at least if it contains any "creative" titles (other than "Chapter 1", "Chapter 2"...). For author name, title, ISBN, external links, etc., I agree, they shouldn't be a problem copyright-wise. The part of the metadata that must be scrambled is any description, synopsis or similar; but publisher name, dates, genres... those are OK.

[Jellby]

Originally Posted by JSWolf

One suggestion, don't scramble characters with a high-bit because you never know if they can be part of the problem.

Originally Posted by Turtle91

Who's to know if they were part of the original...

That may be OK for English (or even other languages with Latin-based alphabet), but applying that rule for Arabic, Japanese or Greek books will mean almost nothing is scrambled.

I say scramble all letters and digits (use Unicode properties to determine what's a letter or digit), at least by default. If those characters were part of the problem, then the scrambled book will not show the problem, and that can be used for debugging.

[JSWolf]

Originally Posted by PeterT

I think he means characters with an ordinal # greater than 127.

Yes, exactly.

[JSWolf]

Originally Posted by Jellby

That may be OK for English (or even other languages with Latin-based alphabet), but applying that rule for Arabic, Japanese or Greek books will mean almost nothing is scrambled.

I say scramble all letters and digits (use Unicode properties to determine what's a letter or digit), at least by default. If those characters were part of the problem, then the scrambled book will not show the problem, and that can be used for debugging.

If you do scramble high-bit characters, I suggest using a similar high-bit character as a replacement. The reason being is that if any of these high-bit characters is causing a problem, than you want them where they are to be able to see the problem so someone can try to debug the prolem.

[jackie_w]

Originally Posted by Jellby

The decision was that the Table of Contents (chapter headers, etc.) must be scrambled too, at least if it contains any "creative" titles (other than "Chapter 1", "Chapter 2"...). For author name, title, ISBN, external links, etc., I agree, they shouldn't be a problem copyright-wise. The part of the metadata that must be scrambled is any description, synopsis or similar; but publisher name, dates, genres... those are OK.

Re: the TOC ... I propose the following, which would be very easy to do:

• TOC ncx: scramble alpha text but leave digits as-is
• Main content (incl. inline TOC): scramble both alpha and digits

Might this be a reasonable compromise as a starting point? Content headings still won't match their TOC ncx entries but at least the TOC won't look quite as weird as it does in the v0.1 alpha version I posted.

[HarryT]

That sounds like a sensible compromise to me.

[jackie_w]

Originally Posted by Jellby

That may be OK for English (or even other languages with Latin-based alphabet), but applying that rule for Arabic, Japanese or Greek books will mean almost nothing is scrambled.

I say scramble all letters and digits (use Unicode properties to determine what's a letter or digit), at least by default. If those characters were part of the problem, then the scrambled book will not show the problem, and that can be used for debugging.

The current logic for scrambling is:

• if a char has different upper- and lower-case versions - scramble to a value from LOWERS, adjusting to retain case.
• if a char is a digit - scramble to a value from DIGITS
• otherwise leave as-is

where
LOWERS = list('abcdefghijklmnopqrstuvwxyz')
UPPERS = uppercase equivalent of LOWERS
DIGITS = list('0123456789')

I think that should work OK for European, Greek, Cyrillic alphabet languages but probably not for CJK and other Eastern alphabets. I don't have any detailed knowledge about non-Latin alphabets.

If/when calibre-plugin-ised, I could envision giving the user limited control of what's in the LOWERS list (e.g. single character, word, phrase) if that might be useful. In which case, adding some special type-able chars of choice should also be possible.

You mentioned 'unicode properties'. I'm open to suggestions for a better simple algorithm to include a wider variety of languages.

[jackie_w]

Originally Posted by HarryT

That sounds like a sensible compromise to me.

Consider it done