Suggestions for Sigil Improvements or New Features

[KevinH]

Hi All,

It is that time again ...

I am again opening a new thread for your suggestions for improvements or new features for future versions of Sigil WITH THE FOLLOWING CAVEATS:

1. No "advanced editor" features. If you have a file editor you love more, then use it with Sigil's "open with" feature.

2. Nothing that is better done in a plugin (ie, no AI generation of summaries, no AI translations, etc, a separate plugin or even a real wordprocessor is the right place for those kinds of tools). Sigil is an epub editor not a Word Processor.

3. No changes just for the sake of change (.ie no new Mac icons, etc)

4. Your suggestion will benefit the vast majority of Sigil users.


Please post your suggestions in this thread for all users to comment on.


Thanks

[KevinH]

For the record, we had one Enhancement request left over from last time:

- show clip number in Clip Editor window to make assigning shortcuts easier

[Doitsu]

The Calibre Editor includes a lightweight HTML syntax checker and can automatically correct some common errors. It also applies certain workarounds for issues that EPUBCheck might not detect, such as enforcing attribute order expected by specific readers or apps. A similar tool could be especially useful for complete beginners.

[KevinH]

Are there still e-readers that require specific attribute orders? I thought that those e-readers died away as requiring a specific attribute order is completely against the spec?

And I have not noticed any auto correcting syntax checker in calibre but I have not updated calibre in years on my machine. So would you please include specific examples of what is typed, and how it is autocorrected for syntax? Or is this just guessed autocomplete at work?

[mrprobert]

Feature Request: Session ID for Plugin Cache Invalidation

I've written up a feature request for adding a Session ID to Sigil's plugin API. This would enable plugins to reliably cache expensive computations by detecting when Sigil has restarted.

Full proposal with implementation details:
https://gist.github.com/mrprobert/77...d5c20a23af8638

TL;DR:
- Problem: Plugins can't tell if Sigil restarted between invocations
- Solution: Add a simple session identifier (timestamp_pid format)

[Doitsu]

Originally Posted by KevinH

Are there still e-readers that require specific attribute orders? I thought that those e-readers died away as requiring a specific attribute order is completely against the spec?

I'm not sure if that's still the case. Calibre gives this example:

Originally Posted by Calibre Editor

Some e-book readers such as the Nook fail to recognize covers if the content attribute comes before the name attribute. For maximum compatibility move the name attribute before the content attribute.

<meta content="my-cover-image" name="cover"/>

Originally Posted by KevinH

And I have not noticed any auto correcting syntax checker in calibre but I have not updated calibre in years on my machine.

The syntax checker works like EPUBCheck. To see this checker in action open any epub with the Calibre Editor, delete a closing tag, then select View > Check book, click the Run Check link, then click "Try to correct all fixable errors automatically."

For more information see the source code of the tool, which appears to be in this folder.

https://github.com/kovidgoyal/calibr...b/polish/check

[JSWolf]

Originally Posted by Doitsu

The Calibre Editor includes a lightweight HTML syntax checker and can automatically correct some common errors. It also applies certain workarounds for issues that EPUBCheck might not detect, such as enforcing attribute order expected by specific readers or apps. A similar tool could be especially useful for complete beginners.

One other thing calibre's error checker does better then epubcheck is it checks CSS more thoroughly.

[KevinH]

Please provide some justification for this. Plugins must be shutdown when exiting Sigil otherwise their output is lost (by design) and if Sigil is crashing, that should not happen.

What potential use is this? How would this help the majority of Sigil users? As you pointed out, the implementation of a session id is easy to do. The real question is why? Please provide complete examples of how this helps.

Originally Posted by mrprobert

Feature Request: Session ID for Plugin Cache Invalidation

I've written up a feature request for adding a Session ID to Sigil's plugin API. This would enable plugins to reliably cache expensive computations by detecting when Sigil has restarted.

Full proposal with implementation details:
https://gist.github.com/mrprobert/77...d5c20a23af8638

TL;DR:
- Problem: Plugins can't tell if Sigil restarted between invocations
- Solution: Add a simple session identifier (timestamp_pid format)

[KevinH]

I think special code to support ancient epub2 nook ereaders is really not something for Sigil.

I will look into the autocorrect for Sigil but the well-formed check is supposed to pick these types of errors up and explain the error.

That said, I have no desire to recreate or replace epubcheck inside Sigil. AFWIW, the check code you referred to only detects errors. There is no code to guess at an autocorrect. That must be someplace else in calibre's codebase, as it does not appear directly in the check folder (or at least I could not find it).

I did see comments that said the autocorrect is a guess and may not be what the user wants. My theory is for many of the structural errors, Kovid is autofixing using the equivalent of Sigil's gumbo Mend. Kovid's html repair parser borrows code from our gumbo. And Mend is always available as a right click pop-up menu in Sigil's CodeView.



Update
--------

At random I deleted one ending tag in 3 different files in a long epub. Then ran Sigil's Well-formed Check and it found and reported all 3. I could then either navigate to each and hand fix it or simply run Mend on the epub and it automatically fixed each case.

Lots of the checks in that "check" folder were for named entities (which Kovid got rid of even in epub2 because he did not want to support doctypes). These are automatically fixed by Mend as well.

There are checks for invalid id attributes not starting with an "alpha" character, and a few other more minor things.

If someone could provide a list of the most common errors they would like to have detected and auto "fixed", I will look into creating a new python based tools for that.

But nothing that would replace epubcheck. Epubcheck is the right tool in Sigil for detecting and reporting more complicated errors that running Mend will not fix.

For most common structural errors, using Sigil's Mend fixes things.

[mrprobert]

Originally Posted by KevinH

Please provide some justification for this? Plugins must be shutdown when exiting Sigil otherwise their output is lost (by design) and if Sigil is crashing, that should not happen.

What potential use is this? How would this help the majority of Sigil users?

I'm not sure what you mean by, "Plugins must be shutdown when exiting Sigil otherwise their output is lost". In any case, I wrote a plugin that does a significant amount of work, some of which could be cached for use the next time the plugin is run, if it is run from the same Sigil session. I guess it would be useful to plugin authors and follow-on plugin users.

Suppose a plugin creates a parse tree the first time it is run. If it is cached then the same plugin can continue to modify it every time the plugin runs, until Sigil exits. Would need a session id to make this work.

[KevinH]

I mean just that. Launching a plugin creates/forks a process and opens a pipe to it to wait and retrieve an xml file describing the changes the plugin wants to have made along with a folder with new or changed files in it. If you shudown Sigil, or cancel the running process this pipe is broken, so no changes inside Sigil are made. You are not given an option to attach or reattach to an already running plugin process.

So given the above, you really need to explain why creating a cache matters. And fwiw parsing a file or files is quite fast because the plugin has access to a fast C language gumbo variant parser. That gumbo parser instance is fully thread safe and can easily be parallelized for even more speed.

So what takes so much time that a cache is needed to keep state between invocations which would use different process ids for each invocation of a plugin anyway. And how would this help the majority of Sigil users?

You could always run a sha sum in your plugin on the epubs files and derive a launch time from the process id creation time inside your plugin and so create your own session key that would work for a sustainable cache that would detect if any incompatible changes have been made under Sigil (no matter when launched) and that requires no changes to Sigil at all.

What am I missing that would make this a useful change that can not already be done now?

[mrprobert]

Originally Posted by KevinH

I mean just that. Launching a plugin creates/forks a process that opens a pipe to it to retrieve an xml file describing the changes the plugin wants to have nade alongwith a folder with new or changed files in it. If you shudown Sigil, or cancel the running process this pipe is broken, so no changes inside sigil are made. You are not given an option to attach or reattach to an already running plugin process.

And I don't want do that, if I'm understanding what you wrote. The workflow is:
1. Start Sigil.
2. Run MyPlugin until it completes.
3. Do some Sigil stuff.
4. Run MyPlugin again.
5. Rinse and repeat.
6. Exit Sigil.

So you are saying that wouldn't work?

[KevinH]

No I am saying each time you shutdown Sigil the connection to any running plugin is lost. So your plugin needs to have successfully completed before returning to Sigil.

So what needs to be cached and why? Why can't you create your own session key equivalent from the epub files you care about and want to cache if you truly want to cache?

Could work done in Sigil between invocations of your plugin, undo previous plugin work? If so sha256 checksums on a file level would be needed to detect if modifications have been made. Again something you can already do in a plugin.

[mrprobert]

Unless, the plugin has a method to find Sigil's process ID, it won't know if Sigil has quit and restarted.

I withdraw my request.

[KevinH]

Originally Posted by mrprobert

Unless, the plugin has a method to find Sigil's process ID, it won't know if Sigil has quit and restarted.

I withdraw my request.

You still have not explained why that matters? Why is this a problem for a plugin cache?

1. Startup Sigil load epub A
2. Run your plugin which completes
3. Save epub A
4. Quit Sigil and restart it and load epub A
5. Run your plugin again.

How does Sigil restarting invalidate your plugin's cache as that cache must depend on the state of the files inside the epub it is run on. So a plugin derivable cache key is already possible with no changes based on file sha sums.

In other words, why do you need the Sigil process id? How does that help?