I was wondering about that.

The only advantage I could see to supporting dialog based tools is to have an automate list end with one of those but not wait for completion as using any of them requires significant user interaction.

I did some looking and EditMetadata, Generate Table of Contents, and Edit Table of Contents appear to be fully modal so progress will block until they are closed anyway.

And Create HTML Table of Contents, Spellcheck, Saved Searches (F&R) and Reports are fully non-modal (progress will not block unless we wait for them to close) and you can leave them to edit inside the ebook at will using other Sigil features.

Furthermore, closing them is not needed to signify you are done with them as they may remain open until you use them next.

So it is a big question as to what value automating these dialog based tools would be anyway outside of ending in one of them to save a single keystroke shortcut, icon press, or menu click.

Given how flexible plugins are a lot of more complex conditionalized changes can be made including very advanced replacements in a plugin if needed.

So how in general do we deal with these dialog tools that require lots of input from the user itself to happen?

Do we try to automate them and if so to what benefit?

Are they even needed given that plugins and saved searches on groups can be done and run easily with just a few clicks/keystrokes/icons.


Open to any thoughts as to if/why they should be supported and how they will benefit people specifically.

I am starting to agree with Doitsu does here but would like to hear from others as well.

Enhance the plugin category from edit/validate/output? Or only plugins which are for validating can be automated?

Just a quick idea...

Plugin validation can be epub version specific as well. All plugins are already supported. The question was about the Sigil built-in C++ tools that are Dialog based tools.

I earlier wondered if it would be feasible (or even possible) to optionally set some sort of “auto-confirm” on some of these that wait for input. I stopped myself suggesting it because I realize that’s kind of a risky option to unleash. Personally I’d find it handy.

And I agree that ten automations does sound like a lot, though I could see some benefit wrt folks that might like their automations a little more bite-sized.

The following automation changes have been made:

- Epub3 only C++ tools will now succeed but with a log message that they were skipped when run on epub2. Hopefully this will make just 3 automation lists more feasible.

- I have extended the list of known C++ MainWindow Command Line tools with "GenerateTOC", and "CreateHTMLTOC" and have made the "GenerateTOC" automate routine skip the Heading Selector dialog.

The full list of C++ MainWindow Tools supported in master is now:

• UpdateManifestProperties
• Save
• WellFormedCheckEpub
• MendPrettifyHTML
• MendHTML
• ValidateStylesheetsWithW3C
• RepoCommit (Checkpoint)
• GenerateNCXGuideFromNav
• RemoveNCXGuideFromEpub3
• DeleteUnusedMedia
• DeleteUnusedStyles
• StandardizeEpub (Restructure Epub to Sigil Norm)
• SplitOnSGFSectionMarkers
• AddCover
• GenerateTOC (will create from existing headers without the Heading Editor functionality)
• CreateHTMLTOC

I have also modified the PluginRunner but this impacts macOS users of input plugins only.

On macOS input plugins always opened a new Window which would break automation.

Now macOS will replace the current book with the one from the input plugin in the same window just as Linux and Windows has always done. This is now similar to how we have implemented Restore from Checkpoint on macOS.

All of this was just pushed to master.

New Automate Feature Testers Wanted
 

Hi All,

If anyone is interested in testing this new automate feature seeing if it is useful and providing feedback, bug reports, etc; And if you use Windows (64 bit) or macOS:

Please pm "KevinH" here on MobileRead and I will pm you back with a link to a pre-release build of Sigil master for you to download and install.

Bandwidth on these builds is limited so only reply if you are serious about helping us out with this new feature.

Thanks!

KevinH

Hi,

I have only had one volunteer tester for the Automate feature so far so if you can help, please pm me.

Thanks,

KevinH

More Automate Related Features added:

Based on comments and requests from the testers of this proposed automate feature, the following improvements and changes have been made:


- Added Support for reformatting all CSS to Multiple Line format

- Added Support for reformatting all CSS to Single Line format

- Added Support for running a named Saved Search as ReplaceAll

- Changed the Automate Toolbar to use 3 QToolButtons that when clicked and held will pop-up a menu to allow editing Automate Lists

- Added a new AutomateEditor and support dialogs to allow creation and editing of the 3 automate lists that is based on the approach used in the MetaData editor with buttons
to choose a supported Tool command or an existing/installed plugin, and arrow keys to easily move lines.


All of this has been pushed to master.

To help document things:

The 3 Automate Lists are stored inside the Sigil preferences folder:
automate01.txt
automate02.txt
automate03.txt

Each file is utf-8 encoded with one tool command or plugin name per line.
A leading # indicates a comment.

To run an automate list, simply click on the corresponding robotic arm icon in the Automate Toolbar.

To edit/create an automate list, simply click and hold on the corresponding robotic arm icon in the Automate Toolbar and select Edit from the resulting pop-up menu.

All currently installed Plugins are supported by name.

The Tools now supported:

• AddCover
• CreateHTMLTOC
• DeleteUnusedMedia
• DeleteUnusedStyles
• GenerateNCXGuideFromNav
• GenerateTOC (will create from existing headers without the Heading Editor)
• MendPrettifyHTML
• MendHTML
• ReformatCSSMultipleLines
• ReformatCSSSingleLines
• RemoveNCXGuideFromEpub3
• RepoCommit (Checkpoint)
• RunSavedSearchReplaceAll [full name of a saved search]
• Save
• SplitOnSGFSectionMarkers
• StandardizeEpub (Restructure Epub to Sigil Norm)
• UpdateManifestProperties
• ValidateStylesheetsWithW3C
• WellFormedCheckEpub

I will be sending new test build links out to those who volunteered to test for further feedback and bug reports.

If interested to help out and willing to test and provide feedback, please PM "KevinH" and I will send you a link.

Thanks to all who have provided input on this process. Hopefully it will result in something useful for our more advanced users.

People who build from source your input is, as always, welcomed. Most importantly I am looking for feedback on the following:

Is this automate feature truly useful as implemented?

What additional Tool truly *needs* to be added (if any)?

Is the built-in Automate List Editor adequate given users can
use any text editor to accomplish the same thing easily enough?

Ad RunSavedSearchReplaceAll: How to run Saved Search from inside the group?

EDIT: I already know. RunSavedSearchReplaceAll GroupName/SavedSearchName

Yes.

Probably not, but I will go over all the functions carefully and maybe I will find something.

The new editor for automateXX.txt files was a surprise for me and I like it very much, because I don't have to remember the exact names, but choose them calmly from the list and arrange them in the right order.

The currently supported RunSavedSearchReplaceAll accepts one parameter on the line which represents the full name of the saved search or saved search group. Remember Group names end with a "/" just like you see as the tooltip when hovering over a group or search in the SavedSearch Dialog.

For example I have a search group called AutomateTest with two searches in the group named test1 and test2.

If I add the command:

RunSavedSearchReplaceAll AutomateTest/

it will run the group (both tests)

instead if I do:

RunSavedSearchReplaceAll AutomateTest/test2

it will run just the test2

I am finding it very useful.

It was a very pleasant surprise. Compared to remembering the function/plugin names and mistyping them, it makes creating and modifying actions much easier. Automating the automation tool. :)

Just checked out the latest. Fun stuff!

It's probably only cosmetic, but the automate log seems to contain spurious entries about being "skipped since not an epub3."

With the exception of RemoveNCXGuideFromEpub3 and UpdateManifestProperties, none of my tasks should have been epub3 specific. Plus they all seemed to be successful (even if many had nothing to do). Checkpoints were created, MendPrettify was run, CSS was reformatted, etc...

Yes must be a cosmetic bug. I am on it.


Yes, it was cosmetic only, and now fixed in master with a new deploy in case you want to play with it.

Thanks!

Kevin

Yeah, I also really like this. It actually surfaces features I'd not have thought of using, necessarily--a pretty cool side effect.

Not an error, but in order to see how Sigil can process an erroneous command, I opened automate01.txt and I included by hand the name of the plugin "XX_Everything" (of course, the plugin is imaginary, doesn't exist). Well, I received the following message:

Maybe it could be preferable that Sigil -when there is an error- instead of aborting the list, it will be able to offers the users the option to continue or to abort.

FWIW, I do think terminating the list is the right thing to do for non-existent plugins, plugins that report failures, and Tools that report failure. Continuing would be dangerous and the log file will show you exactly what went wrong so you can fix the Automate list, then revert to the last save Checkpoint or reload the file and run the corrected list.

This sounds wonderful. I have no idea how easy it is to implement, as I am not a programmer.
If possible, it would be like an early Sigil christmas present.
THANK YOU.

Maybe add additional checking the contents of the automateXX.txt file (eg in the "Automate List Editor" window or as a separate option in the submenu)?
Fast parsing and checking if the names of tools, plugins, and Saved Search are correct.

I am thinking these Automate lists are more like simple .bat files. If you use the editor to create them you will aways have valid commands, but that still does 't mean they will run successfully. So a validator is a bit of overkill in my opinion when you just revert the checkpoint and run again if needed.

I noticed the "RemoveNCXGuideFromEpub3" function and was wondering if it is also possible to change an ePub3 to an ePub2.

Currently I manually open the opf from the zip file and change the version=3 to version=2 then save and reopen. Not a difficult thing to do, just time consuming.

I'm thinking this functionality would require closing the epub but leave the sigil shell open to run the script, then reopening the epub...

I think there is a plugin that tries to do that. However not all epub3 features are supported by epub2 unlike going from epub2 to epub3.

Since all new epub readers support epub3, and Kindle can accept epub3 for input why is going backwards important, as it will not help for archival purposes? And any "normal" fiction epub3 can add in an NCX and guide info to make them compatible with older epub2 only readers.

To future proof things, converting from epub2 to epub3 but making sure they have an ncx and guide for backwards compatibility would seem to be a better approach.

What am I missing?

I know there are those around here that pontificate about the joys of ePub2 but it's just me being lazy and not learning ePub3-isms yet. I still do everything in ePub2. One thing I DON'T like is ePub3's meta entries in the opf. :shrug:

You could try the Doitsu's epub2 output plugin. I tried it a couple of times and the output seemed quite clean but I found myself working more with epub3 files so never did a full job of testing it.

I think it's a very good idea, but you see a lot of different ideas and wishes in this post.
Why not put the commands to be executed in a JSON file? This makes it much easier to add and maintain a program step (and its sequence), as well as the associated options and actions on the result of that step.

There are no associated options or actions for each step. It's not that complicated. It's a list of tools to be run in succession, nothing more.

Users only needed to manually edit automate lists in earlier beta versions. The latest beta comes with a dedicated Automate List Editor for creating/editing automation lists.

https://i.imgur.com/ggmndxY.png

Besides, so far only one task requires a parameter, which is clearly indicated.

https://i.imgur.com/f5O2O5L.png

Based on feedback from testers of the betas, the automate feature appears to be quite useful and will be present in the next release.

That leads to a 2 more things:

- How likely will this feature be used by newer or occasional users of Sigil? If not much, should the Automate Toolbar be hidden for all new Sigil users until enabled with the View menu?

- Is there anyone willing to contribute an Automate chapter for our Sigil User Guide? Perhaps with a couple of sample Automate Lists that users might find useful? A picture of the Automate Toolbar, and a picture of the Automate Editor and maybe even a picture of the resulting log?

If not, I will take a shot at it but I am a horrible writer.

Thanks,

ps:

I just pushed to master support for the following 4 new automate Tool commands (and added AutomateEditor support for them). Thank BeckyEbook for the idea!

SetBookBrowserToAllCSS
- "Select all CSS Files in BookBrowser"

SetBookBrowserToAllHTML
- "Select all HTML Files in BookBrowser"

SetBookBrowserToAllImages
- "Select all Image Files in BookBrowser"

SetBookBrowserToInitialSelection
- "Reset BookBrowser to its initial selection"

A very good idea! Especially useful for plugins that only operate on files selected in BookBrowser.

If there are no other volunteers, I would be willing to put something together with two caveats:

1) I am swamped right now, so it may be some weeks before I can do this. Not sure what the time-frame is.

2) While I am intrigued by the new Automate feature, it's unlikely that I will actually use it. But I like learning new things and will be playing with the new Automate feature just to figure out how it works. If there's someone who will be actively using Automate and is willing to write the chapter, that would be better. If not, I'm game.

No problems! We can wait. I am in no rush to release Sigil 1.8 since Sigil 1.7 seems to be quite stable and doing its job well.

To help speed things up, I have attached my very very rough draft of a new automate.xhtml chapter as a starting point. I have also added a few images and things.

When you have time, just let me know and I can get you a link to a pre-release build of Sigil for macOS so that you can see and play around with this new feature.

Please feel free to change anything you want in this chapter as I admit my writing style is not for everyone (or even for me!).

Thanks!

KevinH

Thanks! The rough draft looks not so rough at all and will certainly speed things up.

If you have a pre-release build ready, I'd be interested in the link so that I have it on hand if I have some time here and there to play with it. Thanks!

I will pm you the latest link.

Thanks!

I’d vote surface it up front. Personally, I get in my habits early and despite using sigil daily, this very feature has helped me find OTHER features I didn’t know existed (a category this feature would have also landed in).

↑ ↑ ↑ ✔
Once we find an app we like, we tend to prefer the way things were done. Thus I do my heavy lifting with Sigil. New features would end up buried and undiscovered in many cases (and have) if someone had not posted here.

OTOH don't do a full reset of a users toolbar setup :( without asking:
"new features require major toolbar x adjustments. OK? Skip"

I didn't really follow the implementation of this feature, but now that the translations have appeared, which means that its arrival is imminent...
Only 3 automate lists?!

But, but, But No! What is this heresy?
10 automate (2 groups of 5, like plugin) would be a minimum for a such powerful feature!!

My god. It's not even released yet and it's not good enough. I'm afraid you'll have to make do with only three new things you've never had before.

And the fact is: with a little file renaming, you can have as many automate lists as your heart desires. The limitation only applies to being able to have 3 (with their associated keyboard shortcuts and menu icons) at your immediate fingertips. Hell, someone could even write a plugin that would make it quick and easy to pick which of a curated multitude of automate lists are the ones assigned to the 3 available slots at any given time.

Oh, sorry.
I judged with too little information.
Yes, of course, it's normal that only a limited number of them are in shortcut.
But how to activate/edit the others? A menu, with customizable dynamic sub-menu option, like some Calibre plugin?
(also all this was rather said in a humorous tone, thanks for creating this feature)

https://i.imgur.com/GEwW5r0.png

There'll be a new toolbar with 3 icons. If you long-click one of them, you can select the Automate List Editor and add several automation tasks or regular plugins.

https://i.imgur.com/ggmndxY.png

You can also run any number of saved searches or search groups.
https://i.imgur.com/f5O2O5L.png

In addition to the creation/editing of lists within Sigil, the lists will be stored as files in the Sigil preferences directory. You could manually create as many lists as you like, but only automate01.txt, automate02.txt, and automate03.txt will be loaded into the 3 available menu slots. With some manual renaming you could maintain more than three lists. My idea of a plugin was that it could handle the renaming to determine which lists currently occupy the 3 available slots.

IMHO, 3 lists should be more than enough for most users. You could use one for epub2-specific tasks, one for epub3-specific tasks, and a third one for "special tasks."