Possible To-Do List for Future Sigil Releases Post Sigil 1.8

[KevinH]

For those interested ...

Here is a list of possible To-Do items for future Sigil releases.


Under consideration for future Sigil releases

• - Add Indicator in Status Bar if epub in old Sigil "Standard Format"
• - Add some form of templating supported for HTML TOC and Index generation (but maybe better by plugin)
• - Multiple paragraph justification setting via the CV alignment icon (although PageEdit does this now)
• - Add a regex to be used to build TOC when no headings tags exist (considering but this is against accessibility)
• - Add GetInfo dialog to BookBrowser to add/change Semantics and Manifest Properties on a File
• - Add sorting of Validation results by bookpath and error message
• - Add export of Validation results to csv
• - Update to version 2 of the C regular expression library
• - Add checkbox to limit search replace to be outside tags (and add to Saved Search controls)
• - Add CSSToolbox and GUI to help users create, manipulate, and merge multiple css stylesheets/selectors
• - Update gumbo parser with recent WhatWG changes
• - Add case-sensitivity checkbox for Spellcheck filter

If you have other ideas that you feel would benefit MANY users that would NOT be better as plugins, please let us know.

FYI:

Based on ideas from Tex2002ans, over the last month or so I have been working offline of a CSSToolbox codebase to act as the backbone for a GUI tool to:

• - merge selectors across separate css stylesheets
• - convert script tags into external stylesheets
• - convert inline-styles into an external stylesheet
• - delete selectors from a stylesheet
• - rename class selectors in stylesheet and then updating all html code that uses it
• - strip out a set of properties across all selectors and stylesheets
• - compare two selector property lists to find matching selectors across stylesheets

This is still a long way from fruition but will hopefully help people who work with epubs that have separate but near duplicate css stylesheets (one per chapter) as might be produced by In-Design.

[Tex2002ans]

Originally Posted by KevinH

Here is a list of possible To-Do items for future Sigil releases.

Excellent!

Originally Posted by KevinH

- Add GetInfo dialog to BookBrowser to add/change Semantics and Manifest Properties on a File

- Update to version 2 of the C regular expression library

More details on these?

Originally Posted by KevinH

- Add a regex to be used to build TOC when no headings tags exist (considering but this is against accessibility)

Heh. I like the way Sigil currently does it:

You do bad Accessibility? You don't get access to the amazing TOC Generator!

(Same with Word or LibreOffice's Table of Contents. Don't use Headings properly? Now you can't easily jump around your documents or generate page numbers.)

Maybe it nudges people to create better code.

I could see this MAYBE being helpful if you wanted to point to a Copyright Page, Title Page, etc., without necessarily having a heading inside. But that's what I use Tools > Table of Contents > Edit Table of Contents for.

What were the example use-cases being given?

Originally Posted by KevinH

Based on ideas from Text2002ans, over the last month or so I have been working offline of a CSSToolbox codebase to act as the backbone for a GUI tool to:

Who's this Text guy? He sure sounds like a smart guy!

And to go into more detail on the CSS tools:

These will help you catch and merge exact/similar CSS:

Exact match:

.normal {
	font-size: 1em;
}
[...]
.class98 {
	font-size: 1em;
}

Very close:

.normal {
	font-size: 1em;
}
[...]
.class99 {
	font-size: 1.1em;
}

Or similar:

.normal {
	font-size: 1em;
}
[...]
.class100 {
	font-size: 1em;
	color: black;
}

All 3 extra classes could then merge into 1, and Sigil would take care of the renaming/cleanup for you:

Before:

Spoiler:

<p>An <span class="normal">example</span>
where <span class="class98">there</span>
are <span class="class99">similar</span>
and <span class="class100">classes</span>.</p>

CSS:

.normal {
	font-size: 1em;
}

[...]

.class98 {
	font-size: 1em;
}

.class99 {
	font-size: 1.1em;
}

.class100 {
	font-size: 1em;
	color: black;
}

After:

Spoiler:

<p>An <span class="normal">example</span>
where <span class="normal">there</span>
are <span class="normal">similar</span>
and <span class="normal">classes</span>.</p>

CSS:

.normal {
	font-size: 1em;
}

Very very helpful when working from not-so-clean documents.

Originally Posted by KevinH

If you have other ideas that you feel would benefit MANY users that would NOT be better as plugins, please let us know.

I re-thought of this one while messaging with someone else:

Saved Searches: Count All

In Saved Searches, when you press "Count All" on a Group, Sigil just gives you the total # of hits.

Would be nice to get a list of # hits per Saved Search:

Name           |   # Hits
_______________|______________
Group #1       |
  - Search 1   |       5
  - Search 2   |       4
  - Search 3   |       0
_______________|______________
Total          |       9

Right now, you press "Count All", and Sigil just says 9.

Example: I have large groups I run together, like my 12-step "Finereader Cleanup".

Getting a breakdown, I could know things like:

• "How much smallcaps is in this book?"
• "Are there tables?"
• "Any double footnotes?"

Right now, I run all 12 steps, thousands of hits total, but some important replaces may only get 0/1 hits.

Side Note: Maybe show this window after a Group "Replace All" too?

Or this might get too annoying?

* * *

And this one, we privately discussed a month ago, but I recently formatted my PMs a little nicer + detailed some use-cases.

Here's a little more:

Advanced Find/Replace (List-Based)

You'd have a Spellcheck List-type menu with:

• Find
• Replace
• Filter

and 3 sortable columns:

• Found
• Replace
• # Hits

You'd be able to visually see, at a glance, all hits on your searches.

Like this regex could "find all Dialogue Tags within a fiction book" + replace with "said":

Find: ,” \b(Alex|Bob|Joanne|Suzie|s*he|they)\b (\w+)
Replace: ,” \1 said
Filter:

Found              | Replace        |  Hits
___________________|________________|______
,” Alex opined     | ,” Alex said   |    10
,” Suzie accused   | ,” Suzie said  |     9
,” Joanne agreed   | ,” Joanne said |     4
,” Alex explained  | ,” Alex said   |     2
,” Alex ejaculated | ,” Alex said   |     1
,” she beseeched   | ,” she said    |     1

Then you could filter the list. Like only searching for "Alex":

Filter: Alex

Found              | Replace        |  Hits
___________________|________________|______
,” Alex opined     | ,” Alex said   |    10
,” Alex explained  | ,” Alex said   |     2
,” Alex ejaculated | ,” Alex said   |     1

And you could selectively apply Find/Replace ONLY on specific rows:

Found              | Replace        |  Hits
___________________|________________|______
,” Alex opined     |                |    10
,” Alex explained  |                |     2
,” Alex ejaculated | ,” Alex said   |     1

or sort the columns:

v Found            | Replace        |  Hits
___________________|________________|______
,” Alex ejaculated | ,” Alex said   |     1
,” Alex explained  |                |     2
,” Alex opined     |                |    10

and/or you could double-click on a row, and jump to its exact location in the EPUB (like Spellcheck Lists!).

With the current Find/Replace methods, you have to go through the entire book and:

• Replace All
• Find/Replace one-by-one

Although in many real-life cases, you're also dealing with:

• Replacing nearly all, with a few exceptions
  
  • Like Hyphens -> EN DASH between number ranges.
    
  • You want most of these converted, but NOT URLs, ISBNs, Phone Numbers, etc.
• Finding all, but replacing only a few exceptions
  
  • The "creative dialogue tags" above.

I also believe list form would also help when dealing with complicated regex, because you could visually see before/after without applying.

Example regex:

UPPERCASE -> <span class="smallcaps">

Search: (*UCP)([[:upper:]])([[:upper:]]{2,})
Replace: <span class="smallcaps">\1\L\2\E</span>
Filter:

Found   |  Replace                              |  Hits
________|_______________________________________|______
BC      | <span class="smallcaps">Bc</span>     |   100
AD      | <span class="smallcaps">Ad</span>     |    47
FIGURE  | <span class="smallcaps">Figure</span> |    10
ALEX    | <span class="smallcaps">Alex</span>   |     2
CIA     | <span class="smallcaps">Cia</span>    |     1
FBI     | <span class="smallcaps">Fbi</span>    |     1
USA     | <span class="smallcaps">Usa</span>    |     1

You'd probably want to skip replacing acronyms + "AD" and "BC", but apply that regex on all others:

Found   |  Replace                              |  Hits
________|_______________________________________|______
BC      |                                       |   100
AD      |                                       |    47
FIGURE  | <span class="smallcaps">Figure</span> |    10
ALEX    | <span class="smallcaps">Alex</span>   |     2
CIA     |                                       |     1
FBI     |                                       |     1
USA     |                                       |     1

With the current method, you'd have to:

• Click through 150 false positives to dig out that 12 you actually want to fix.
• Replace All, then correct "Ad" and "Bc", "Cia", ...
• Create a more complicated, complicated regex. (Or set of Saved Searches.)

With the list form, you can see patterns, double-check, then selectively apply Replaces (or Replace All just like now... just with more information at your fingertips).

[Turtle91]

Is it possible to incorporate something similar to Calibre's "regex-function" in the find/replace. It seems to add a huge range of options without requiring individual plugins.

The option I have my eye on right now is being able to use a find/replace to convert strings to different case types (eg Sentence case, or Title Case - yes I know you can already do lower or upper case).

Thanks!

[KevinH]

Originally Posted by Tex2002ans

More details on these?

The new re library is just to update to a rewritten version 2 of the pcre library from version 1 that Sigil uses. That is internal but that change should be invisible to end users.

Add a GetInfo box to BookBrowser would add a menu item to BookBrowser to pop up a Dialog showing the semantics set on that file, manifest properties set on that file, file media type, file size , etc. It would allow the user to add multiple semantics to the same file (by id) , and easily change the files manifest properties. Based on MacOS GetInfo dialog in many ways, except for epub files.

[KevinH]

I thought about that previously, but based on earlier feedback by others, it does not appear to be an often used feature outside of the built-in case provided by calibre. To make it useful you need to pass along storage to act as its memory and a whole bunch of internal environment info, which would be a pain to add and keep, given how easy a basic plugin can be written.

And uppercasing/lowercasing single letters in words is doable with regex right now so a regex function should not be needed to handle that and of course doing this in a plugin is easy as well.

But I am open to adding it, if others feel it is an important and often used feature. I just do not want add features just to duplicate what can be done now in calibre via writing python regex functions and in Sigil by writing and running a plugin.

Originally Posted by Turtle91

Is it possible to incorporate something similar to Calibre's "regex-function" in the find/replace. It seems to add a huge range of options without requiring individual plugins.

The option I have my eye on right now is being able to use a find/replace to convert strings to different case types (eg Sentence case, or Title Case - yes I know you can already do lower or upper case).

Thanks!

[KevinH]

@Tex2002ans, (got it right that time!)

I will think about how best to get some kind of count info for each saved search item collected and presented to the user. Perhaps a CountAll SavedSearch function for Automate Lists and log file like in Automate Lists could be used.

Alternatively building up a table of results from doing the equivalent of a Find All and just reporting back each and every result of where the find was matched might be useful. Sorting that would give count info easily enough.

I will think more on it when I have finished the "CSS Munger" tool project and can see if it is doable and useful or not.

[Coleccionista]

FWIW this is the stuff I wonder about.

The newly introduced feature of "Automation Lists", how do you see it evolving?

I'd like for it to have more Sigil commands exposed such as being able to rename and add files (A suggestion for the User Guide would be a section on valid parameters for the commands). Use of variables as parameters if not already possible should be welcome.

A minor petition, many times I end up with 20 xhtml tabs and 1 style sheet -lost somewhere- open. I can close "All other tabs" but I would like to have another option of closing "All other XHTML" tabs so I can keep the tab I am working on and don't lose the style sheet.

Saved Searches, if you run a search from the top of a group it will stop in the first subentry that has a match and it won't move forward unless you do the replacement. I'd like it to continue with the rest of matches in all the entries.

Code Folding. I'd like to be able to close/hide entire sections of the code, so I can get a clearer view of the structure of the page. Ideally I'd like to be able to also basically hide all text and have a "tag-only" view. I believe that would help me understand the pattern of usage of CSS classes and help refine any S&R.

[KevinH]

For automate lists, I honestly do not see it evolving much at all. You can do most of what you want in plugins and plugins can be run from the Automate Lists right now. So there is no real need to extend the parameters of the list just to effectively create our own scripting language when python in plugins works just fine and is an established scripting language.

So plan to write python plugins that do the more specialized things you want and then add them to your Automate Lists.

As for renaming files, that can be done with the RERenamer now or via BookBrowser.

There is only one valid parameter and it is used on one tool command only that runs saved searches. It is the full name to the saved search. It is covered in the User Guide chapter.

So no variables, no macros, and no new scripting language. Use custom plugins and python for that and add them to your automate lists.

As for saved searches, if you do no replacement then you are changing nothing. Effectively you end up with a Find All without replacement. Just use that and Find Next (unless you are asking for something similar to what Tex2002ans is suggesting above). Or am I misunderstanding you?

Sorry, Code Folding was proposed earlier and it is not going to be added as it makes the more specialized things we do in CodeView editor (with positional tag listing) much harder.

As alternatives:

- use "Open With" and select an external editor that supports Code Folding.

- use Mend and Prettify to indent and clean up the code and enable tag pair highlighting to help you see structure.

- use Preview's Inspector tool to see a code folded version of the xhtml being edited in CodeView and even see the exact computed styles for any tag and where they come from.

As for "close other xhtml tabs", you do know you can close all other tabs and then right click on any class attribute and select GoToLinkOrStyle and Sigil will happily open the correct Stylesheet in a new tab.

Thanks for your improvement suggestions.

KevinH

Originally Posted by Coleccionista

FWIW this is the stuff I wonder about.

The newly introduced feature of "Automation Lists", how do you see it evolving?

I'd like for it to have more Sigil commands exposed such as being able to rename and add files (A suggestion for the User Guide would be a section on valid parameters for the commands). Use of variables as parameters if not already possible should be welcome.

A minor petition, many times I end up with 20 xhtml tabs and 1 style sheet -lost somewhere- open. I can close "All other tabs" but I would like to have another option of closing "All other XHTML" tabs so I can keep the tab I am working on and don't lose the style sheet.

Saved Searches, if you run a search from the top of a group it will stop in the first subentry that has a match and it won't move forward unless you do the replacement. I'd like it to continue with the rest of matches in all the entries.

Code Folding. I'd like to be able to close/hide entire sections of the code, so I can get a clearer view of the structure of the page. Ideally I'd like to be able to also basically hide all text and have a "tag-only" view. I believe that would help me understand the pattern of usage of CSS classes and help refine any S&R.

[Coleccionista]

Originally Posted by KevinH

As for saved searches, if you do no replacement then you are changing nothing. Effectively you end up with a Find All without replacement. Just use that and Find Next (unless you are asking for something similar to what Tex2002ans is suggesting above). Or am I misunderstanding you?



As for "close other xhtml tabs", you do know you can close all other tabs and then right click on any class attribute and select GoToLinkOrStyle and Sigil will happily open the correct Stylesheet in a new tab.

Thanks for your improvement suggestions.

KevinH

Hi Kevin.

"GoToLinkOrStyle" that option I always ignored well, that helps. By the way I LOVE the "delete tag pair" but sometimes I miss to have a "delete tag attributes".

Regarding saved searches it is something like this.

Group 1

1. Child1
2. Child2
3. Child3
4. Child4
5. Child5

I choose "Group1" "Search", the first match is in "Child2" (say 1 match) but I don't want to replace this match so I click "Search" again and it will again match Child2 expression instead of continuing to whatever matches can be on Child3 to 5. Basically searching a whole group has a hard stop at the first subentry that has a match and won't move forward.
In the case of searching a group I want the "Search" button to not be limited to cycle between one subentry findings but cycle into all the matches for all the subentries.

[KevinH]

It is a sequential search. It will search for a match for child1 in each xhtml file until none can be found, then move on to child2, and then rinse and repeat.

I think what you want is the ability to ignore a match made for child1, and then go on to find other child1 matches? Or do you want to abort child1, and move on to child2 immediately (effectively skipping all remaining searches for child1)?

[Coleccionista]

A sequential search would be OK. I talk about having a flat match space between all childs. As an example I have right now in a book.

Group Code Cleaning --> Count Matches (returns 531)
SubExp 1: id & comments (1 match)
.....
....
SubExp 11: <em> (525 matches)

I click on "Search" in the group and it matches SubExp1 only result. I click on Search again and it stays in the same match while the status bar in Saved Searches shows (Loaded: SubExp1). It only works sequentially between childs if yoiu replace all the matches from the previous childs

In my view once I click Search on the Code Cleaning group the full 531 matches should cycle as if they were the results of a single search in the book window. The status bar in the Saved Searches window should still display which child search are you currently running.

If a small screen recording could illustrate better what I mean let me know.

[KevinH]

So it is not an "ignore" button then? You may be asking for something similar to what Tex2002ans is asking. I am still not sure so a short video would definitely help.

Just so that you and others better understand searching and replacement inside Sigil:

Searching a group is not a single result set (ie. it is not a database-like search result set). It always sequentially finds each term in each xhtml file (also sequentially) and either counts them or does a replacement.

In replacement mode, it will sequentially replace that item and then start looking from there for the next match. If you decide not to replace, the cursor is not updated and so I think it is re-finding the same match. Moving the cursor forward one character before find next should skip to the next match in that same child (which is what my proposed ignore button would do under the hood).

Viewing it like a flat set is not easy as any replacement earlier in a file (or lack thereof) will invalidate all match positional information later in that same file as characters are added or removed from that string / file. This is especially true for the next child in the search group as earlier and later children positional information will be invalidated. That is why children are searched for sequentially in a string/file and a single match at a time.

For a single child search in a single file you can build up a file change set and sort them by starting position and apply the changes in reverse order to the file to prevent invalidating other results for that search term in that one file. But this is not possible for other search term replacements from a search group as they may actually overlap or build upon each other by design.

In fact with a sequence of regular expressions you can use the first to identify things and mark them and then use the second to examine the resulting marks to decide which to update and change and then in the third clean up any leftover markers generated by the first but not handled by the second. This must be done sequentially but it allows for very very complex conditions in regular expressions to be broken down unto a sequence of much simpler to follow regular expressions.


So yes, please show a short video of what is wrong and what you want it to do given the above.

Thanks,

KevinH

[KevinH]

@Coleccionista

Okay, I have been playing with the Saved Search Editor using the Load Search button to load Find and Replace and to try to use the Search Editor Find, Replace, and Replace/Find and Replace All buttons directly.

I think I can recreate what you are seeing. I think without using ReplaceAll the first group search child term is searched for and when the search cursor reaches the end it is never reset for the second group search child. The reset only happens if a replacement is made.

So I think this is a bug actually. It just never hits for ReplaceAll, it only hits when no replacement is done.

I will look into fixing this for the next release.

[Binchen]

Originally Posted by KevinH

You can do most of what you want in plugins and plugins can be run from the Automate Lists right now.

But its not a full automation. Plugins still require manual interaction due to two reasons:

a) The are not build for that (being fed from another source than the user)
b) there is no communication from sigil to the plugin

I'm dreaming of a list of key/value pairs an automated list can send to a plugin. And of course the plugin needs to able to recognize this and than is interpreting the key value pairs instead of displaying a gui.

[KevinH]

But its not a full automation. Plugins still require manual interaction due to two reasons:

a) The are not build for that (being fed from another source than the user)
b) there is no communication from sigil to the plugin

I'm dreaming of a list of key/value pairs an automated list can send to a plugin. And of course the plugin needs to able to recognize this and than is interpreting the key value pairs instead of displaying a gui.

Sigil was not designed to be fully "automated" and never will be. Specific tasks were designed to be done by users or via plugins. That said ... there is communication both ways from Sigil to plugins in the form of environmental info, paths, settings, language, etc and of course what files are selected in BookBrowser and from plugins back to Sigil in the form of a result XML file that tells Sigil exactly what to files to update and how.

There is no requirement for a plugin to use a gui and a plugin does support its own preferences to "remember" things. The plugin console and the plugin iself can autostart and auto close as determined by the plugin xml file.

And since plugins are open source, you can easily adapt or change any existing plugin to remove its GUI and hard code its functionality or change it to be based on a settings file you write or control.

Exactly what would your list of key value pairs entail and how specifically (with examples) would this be helpful given the above?