Adding a limited Automate Feature To Sigil

[KevinH]

@Hitch,
Tex2002ans, has already posted a few test cases for me so no worries. Thanks,

KevinH

[Tex2002ans]

Originally Posted by Hitch

Okay. That wouldn't be one of ours (our production I mean) but it's entirely possible that I have files like that, from other designers that we used for export to ePUB or HTML and subsequent conversion. I will take a look. I mean, to be clear--I know we have had those, but I don't know if I still have one in-house that would be available to Borkify for you. I'll check.

I just PMed KevinH 3 of my examples.

2 InDesign EPUBs -> Merged -> Calibre EPUB->EPUB conversion.

This is where you can see 1 CSS file per chapter + overlapping class names:

Spoiler:

CSS #1:

p.ParaOverride-1 {
	margin-bottom:0px;
}
p.ParaOverride-2 {
	margin-top:1px;
	text-indent:18px;
}
p.ParaOverride-3 {
	text-indent:14px;
}
p.ParaOverride-4 {
	text-indent:18px;
}
span.CharOverride-1 {
	font-size:1.454em;
}
span.CharOverride-2 {
	font-size:1.091em;
}
span.CharOverride-3 {
	font-size:58%;
	vertical-align:super;
}

CSS #2:

p.ParaOverride-1 {
	text-align:center;
}
p.ParaOverride-2 {
	margin-top:0px;
	text-align:center;
	text-indent:0px;
}
p.ParaOverride-3 {
	text-align:center;
	text-indent:0px;
}
p.ParaOverride-4 {
	margin-top:2px;
	text-align:center;
	text-indent:0px;
}
span.CharOverride-1 {
	font-family:"Myriad Pro Semibold", sans-serif;
	font-size:1.801em;
	font-style:normal;
	font-weight:normal;
}
span.CharOverride-2 {
	font-family:"Minion Pro Medium";
	font-size:0.909em;
	font-style:normal;
	font-weight:normal;
}
span.CharOverride-3 {
	font-family:"Myriad Pro Semibold", sans-serif;
	font-style:normal;
	font-weight:normal;
}

1 Word -> HTML -> Calibre EPUB->EPUB conversion.

This is where you can see a typical CSS mess:

Spoiler:

.calibre7 {
    font-family: "Times New Roman", serif
    }

[...]

.calibre12 {
    font-size: 1em
    }
.calibre13 {
    font-family: "Times New Roman", serif;
    font-size: 1em
    }
[...]

.calibre14 {
    font-size: 1.125em;
    line-height: 1.2
    }
.calibre15 {
    color: black;
    font-family: "Garamond", serif;
    font-size: 1em;
    line-height: 1.2
    }
[...]

.calibre17 {
    line-height: 1.2
    }
.calibre18 {
    color: black;
    display: none;
    text-decoration: none
    }
[...]
.calibre20 {
    color: black;
    display: block;
    font-family: "Garamond", serif;
    font-size: 1.48148em;
    font-weight: normal;
    line-height: 1.2;
    page-break-after: avoid;
    text-align: center;
    text-autospace: none;
    margin: 30pt 0
    }
.calibre21 {
    color: black;
    display: block;
    font-family: "Garamond", serif;
    font-size: 1.25926em;
    font-weight: normal;
    line-height: 1.2;
    page-break-after: avoid;
    text-align: justify;
    text-autospace: none;
    margin: 20pt 0
    }

Originally Posted by KevinH

I am looking for an epub created from InDesign that uses individual stylesheets (one per chapter) with many chapters (many stylesheets) that I can use to test some ideas for techniques to merge the large number of stylesheets down into a small hand full of stylesheets and in the process remap styles if possible.

InDesign's EPUB export actually only outputs a single CSS file.

When designing a print book, one type of workflow is:

- individual "chapter file"s
- then link them together into a single "book file".

(This allows you to easily swap/remove chapters, auto-renumber pages/endnotes, etc.)

In my case though, as a converter, I don't have that single "book file"... I only get the 20 separate "chapter file"s.

So, when I'm exporting, I export each individual chapter -> EPUB... hence the 20 different similar-but-not-quite CSS files.

Mix Direct Formatting and lots of other cruft in there, and you get a giant, conflicting mess on your hands.

IF I had the monolithic "book file", I'd be able to export a single EPUB... but you'd still have a spaghetti mess, but no conflicting names. :P

(Same as cleaning up Word->HTML, etc. etc.)

Originally Posted by KevinH

All hopefully *without* having to convert all selectors to class selectors with non-mnemonic numbered names that end up littering the html.

Yeah, I don't believe InDesign or Word/LibreOffice generates complicated selectors.

I think they all just break it down to individual classes.

So the bulk of consolidate/cleanup would probably be this simple conversion cruft:

.class1 {
	text-align: center;
}
.class2 {
	text-align: center;
	font-size: 1em;
}
.class3 {
	text-align: center;
	font-size: .9em;
}

not necessarily trying to tackle all the advanced CSS3 selectors, etc.

Originally Posted by KevinH

If In-Design can handle style mapping from .docx styles, why isn't this handled by In-Design when inputting the .docx files?

Hmm... the Export (Styles Mapper) is definitely there.

I'm not familiar with Import. (I don't actually use InDesign, I only know enough to get text OUT OF IT as soon as possible.)

I believe it's built-in. See this video as one example:

Nukefactory: "How to import text into InDesign without losing basic formatting"

But as usual, the thing is:

• 99+% of people don't use Styles
• they don't use them consistently
  
  • lots of Direct Formatting
• and InDesign Styles =/= Word Styles
  
  • InDesign is much more powerful.
• Print-focused designers probably don't have one clue about HTML or ebooks
  
  • That's just technical gobbledeegook. Everything looks fine with my eyes!
    
  • And hey, great, InDesign "exports" EPUBs. Looks "perfect" on my iPad!!! What's the problem?

So each stage in the conversion workflow has the potential to introduce nonsense or lose key information.

And again, as a converter... I don't have control over what these people are doing in intermediate steps. I just have to clean up the cruft and create the ultimate ebook.

Minor Rant:

Spoiler:

Grumble, grumble.

My latest is trying to get them to understand the text:

For more information, click here and here.

might be 'usable' in a web article... but this type of text CANNOT be used in a physical book (and is very very bad in an ebook).

Originally Posted by KevinH

I am thinking of using ngram scoring to try to identify the most similar set of selector properties (after a filtering step) and presenting those for the userto approve of, then doing the merge.

Yeah, I was thinking of something similar. A similarity score.

You click on a class, it ranks everything that's close.

Then you can Shift+Click or Ctrl+Click and merge the classes together.

* * *

Usually, I try to do this stripping/consolidating in passes. Clean up:

• Fonts
• Colors
• font-size
• italics
• superscripts
• [...]

and at each stage, I try to merge what I can to my "normalized" (human-readable) classes:

• All classes with "vertical-align: super"
  
  • I'll try to convert to class="super" or <sup>.
• Many classes with "font-variant: italic"
  
  • I'll try to convert to class="italics" or <i>/<em>.
• Colors (black text + blue links), I'll instantly strip.
  
  • Then take a closer look at oddities (red, orange, green text, etc.).
    
    • Sometimes these things slip in (especially when authors are doing "Track Changes").
  • Commonly see very dark gray text instead of black.
    
    • CMYK -> RGB or copy/paste-from-other-source issue.
      
    • Once I spot the shade of gray and see it's irrelevant, I strip it.
• All classes with "font-size: 1em;"
  
  • I remove that line.
• Most fonts
  
  • I'll remove the CSS for main text font, then take a look at classes that DON'T use that font.
    
  • For example, the book is "Times New Roman", but there's a few classes with "Arial" or "Symbol" or something different. I'll take a closer look to see exactly where/how that was used.
    
    • Very common when there's Greek letters or Maths symbols.
• [...]

This is where I got excited when I stumbled upon that Calibre "Transform Styles" tab.

It will allow me to at least come up with a set of some property-stripping rules that would save some time.

But the frustrating thing about Calibre EPUB->EPUB is it changes the class names.

And it's hard to know ahead-of-time what junk is going to be in this specific book! Each one will introduce their own unique niggles:

Like one book might use font-size: .88889em, another might have .888em and .8em.

One book might be typeset in "Times New Roman" with "Arial" crept in, another book "Arial" as the main with "Times New Roman" crept in.

This is why I mostly do CSS consolidation as THE VERY FIRST STEP after merging, then do successive rounds of EPUB->EPUB to make sure I get down to more bare bones.

But, of course, at later stages, when looking at CSS details, that's when you spot more consolidation that could've been done.

(Hence, a nice GUI, CSS Comparison/Merger, Style Mapper, etc.)

Originally Posted by KevinH

I am thinking that by paretos rule we should be able to take a large number of stylesheets and merge them into a much small number but keep most of the individuality present.

[Hitch]

Originally Posted by Tex2002ans

I just PMed KevinH 3 of my examples.

2 InDesign EPUBs -> Merged -> Calibre EPUB->EPUB conversion.

What, the ONE TIME I could have actually been useful, since they stopped accepting donations and you send him the files! You are Dastardly!

InDesign's EPUB export actually only outputs a single CSS file.

Yes ^.

When designing a print book, one type of workflow is:

- individual "chapter file"s
- then link them together into a single "book file".

(This allows you to easily swap/remove chapters, auto-renumber pages/endnotes, etc.)

In my case though, as a converter, I don't have that single "book file"... I only get the 20 separate "chapter file"s.

You should be getting a master "file" called an indb. However, to use it, you'd have to have INDD.

So, when I'm exporting, I export each individual chapter -> EPUB... hence the 20 different similar-but-not-quite CSS files.

Mix Direct Formatting and lots of other cruft in there, and you get a giant, conflicting mess on your hands.

IF I had the monolithic "book file", I'd be able to export a single EPUB... but you'd still have a spaghetti mess, but no conflicting names. :P

(Same as cleaning up Word->HTML, etc. etc.)

Oh, god, yes.

My latest rant, unrelated to Kevin and Diap, is that we're getting INDD package files from "designers" that

Do
Not
Use
Any
Styles
At
All.

Apparently, they licensed INDD on Monday, got a job on Wednesday and now, even though they couldn't spell DEZINER on Monday, now they IZ ONE!

GRRRRRMMMMMMMBLLLL

Yes, me too^^^

Hitch

[phillipgessert]

Originally Posted by Hitch

My latest rant, unrelated to Kevin and Diap, is that we're getting INDD package files from "designers" that

Do
Not
Use
Any
Styles
At
All.

Isn’t that most of em? Just kidding, most do use styles. Gives em something to constantly override.

[Tex2002ans]

Originally Posted by Hitch

What, the ONE TIME I could have actually been useful, since they stopped accepting donations and you send him the files! You are Dastardly!

You're still useful, Hitch, you're still suuuuper useful... (not ).

Still look for some baaaad InDesign files.

I sent relatively tame ones.

(I know I have some really disgusting ones over the years, but I just grabbed one off the top of my head + my latest from two weeks ago.)

You've definitely seen a lot worse stuff than me too. Oh boy... some of your horror stories...

Originally Posted by Hitch

You should be getting a master "file" called an indb. However, to use it, you'd have to have INDD.

Because of Adobe's cloud/subscription bullshit, and insisting on INDD only opening in the latest-and-"greatest" version...

... I fallback to IDML files.

This allows me to open it up in whatever older InDesign version I have.

There's no way I'm going to pay some preposterous fee, only to open the files for a split second and export EPUBs out.

Originally Posted by Hitch

Apparently, they licensed INDD on Monday, got a job on Wednesday and now, even though they couldn't spell DEZINER on Monday, now they IZ ONE!

GRRRRRMMMMMMMBLLLL

I only met one, one typesetter, who knew what he was doing with Styles.

He was a pleasure to meet/talk with in-person, and he had a very nice Styles workflow. (Also in charge of typesetting monthly/quarterly journals + lots of other random documents.)

I haven't had the pleasure to work directly with him though.

But that's the kind of person I could trust coming up with a consistent Styles workflow from:

Word Styles -> InDesign Styles -> HTML/EPUB Classes -> Sigil cleanup.

But that requires someone who knows what they're doing at that Input/Output stage.

(Similar to those videos I link to showing how to use Word to cleanup Styles + catch Direct Formatting. [LibreOffice 7.1 recently added a Styles Inspector too!])

Word Styles... a few of the editors I work with know/use them... but the problem there is lots of them are working in CMSes (Content Management Systems) or Google Docs... so they're mostly working directly on the web.

Yes, they may get a Word file initially from an author, but then it quickly goes into web collaboration... and then, with many fingers in the pie, more and more Direct Formatting and crap gets introduced.

(Not to mention all the absolute hidden TRASH that gets introduced when doing Comments, Track Changes, copying/pasting from the web, using the Rich Text Editors, using the mobile version of these apps, etc.)

Complete Side Note: LibreOffice is going to be having their 2021 conference tomorrow:

https://blog.documentfoundation.org/...-participants/

There's dozens of talks with lots of interesting information in there.

I'm very interested in the:

Built-in "Xray" like UNO object inspector

which is similar to the Inspector built into web browsers. You'd be able to open up a document and see the DOM + exactly what properties are set:

https://tomazvajngerl.blogspot.com/2...inspector.html

That might help in debugging some particularly tricky documents.

[un_pogaz]

Originally Posted by KevinH

And of course, if you wanted more of a voice then volunteering to test things and providing early input when it was asked for would generally have been good ideas.

You have a point, I should have followed and made this suggestion earlier.

But in my defense, I didn't care about the development details because I thought it would be logical that you develop this feature as it is already the case for plugins: an extensive list of which the GUI buttons are only shortcuts to the favorites.
Hence my bad surprise to see a static system.

(Oh, and stop developing Sigil according to your own specification and preference. Some things I understand for technical reasons, but "oh no, I just need 3, deal with it" is stupid, you're not the only one using it.)

[DiapDealer]

Originally Posted by un_pogaz

(Oh, and stop developing Sigil according to your own specification and preference. Some things I understand for technical reasons, but "oh no, I just need 3, deal with it" is stupid, you're not the only one using it.)

We will continue to develop Sigil as we see fit (while continuing to welcome the recommendations of users who don't present themselves as rude and entitled). If you'll note: there are several USERS in this thread who have acknowledged that 3 lists will likely be sufficient.

Please consider not posting here any more if you can't do so without lashing out with personal insults when development doesn't go the way you'd like. YOU'RE not the only one using Sigil either. If you can't refrain on you're own, I will help you in that regard.

[KevinH]

Originally Posted by un_pogaz

You have a point, I should have followed and made this suggestion earlier.

But in my defense, I didn't care about the development details because I thought it would be logical that you develop this feature as it is already the case for plugins: an extensive list of which the GUI buttons are only shortcuts to the favorites.
Hence my bad surprise to see a static system.

(Oh, and stop developing Sigil according to your own specification and preference. Some things I understand for technical reasons, but "oh no, I just need 3, deal with it" is stupid, you're not the only one using it.)

Wow so much anger over a new feature! We've removed nothing from Sigil. Interested users took an an idea and helped us refine it to something they thought would be useful. That seems like a very good development model to me.

You did not bother to become involved, provided no feedback at all during the process, and then whined and complained when it did not turn out the way you wanted. You are like a child on a playground when things do not go their way. And your ability to win friends and influence people astounds me.

And you obviously do not understand how Open Source software development works. Open source has been and will always be about the developers "scratching their own itch first" followed by merit based input from interested and involved users who "earn" their voice. Look at all the users who have already earned their voice by volunteering their time to help others here on the forums, help test, help report bugs, help when asked for input, etc. That is what an open source community is all about. If you do not like that, then use something else instead of leaching off the goodwill and hard work of others.

[Themycles]

Is there a way to stop the log window from opening unless there is an error?

[KevinH]

No, since an errors may cascade or be ignored depending on earlier decisions. So the Log needs to be examined to determine full success as well as failure for each step.