Possible To-Do List for Future Sigil Releases Post Sigil 1.8

[KevinH]

Master now supports a new Automate List Tool (and the AutomateEditor Dialog also supports it as well).

SetPluginParameter YOUR_PARAMETER_HERE

where YOUR_PARAMETER_HERE is passed as a string to the plugin.

The plugin interface has also been extended to add two related interface calls:

    # Newly Added to Support Plugins running in Automate Lists

    # returns boolean True if plugin was launched by the automate False otherwise
    def using_automate(self):
        return self._w.using_automate

     # returns a string of the automate plugin parameter set before plugin launch
     # and the null string "" if nothing was passed
    def automate_parameter(self):
        return self._w.automate_parameter

These new calls are supported by launcher_versions of 20220101 or greater.

Comments welcome.

Originally Posted by KevinH

Yes that was what I as proposing. FWIW, PluginRunner gets many of its cfg values from the MainWindow that it is launched from (ie its BookBrowser, etc) so a MainWindow member variable is what the Automate Tool would need to be able to set. It would be easy to create a new AutomateList Tool command "SetPluginParameter" that accepts a single parameter whose value is used to set the MainWindow's member variable value so that PluginRunner can pick it up (empty when not set).

[democrite]

Thank you for the plugin work.

Thanks too for the CRC32 change. Running just one file 5 times — the one with 6700 images — and then averaging, ~30% difference. If you wish for more benchmarks, I can try later. Unsure if you consider that difference significant. I also am not sure how I feel about not being able to rename file while keeping date. For the various reasons listed, renaming too I think should keep mod dates if the file is unmodded, e.g. I may merely rename all files to extension .xhtml, or make further renames. Sometimes they may make files change, such as if there links elsewhere, other times not.

For certain publishers, those that separate each chapters xhtml and images into folders, there could be multiple duplicates of some like shared images (e.g. graphical bullets), and in such cases, merging can take a long time, I haven't tested recently on new computer, though before it'd run overnight and not complete. I before have renamed such files outside Sigil. For that and other reasons mentioned, some dialog to perform rename, merge, and perhaps other operations where one can see first toc entry, possible header title, and maybe other metadata could be helpful someday.

I still really really would like any file that is not modified to not have the date changed. Perhaps you disagree, but for me, one might say it is merely the "right" thing to do.

[KevinH]

Hmm if epub import times still show a huge difference then something is fishy with your timings. On import with current master no file hashing is done at all. Perhaps some OS level file caching or something else on your system is impacting things.

You seem to misunderstand and think that these zip dates are real file modification dates but they only exist in the zip file. Sigil caches xhtml files and only flushes to real files when needed and but files are often opened for write which resets the actual file modification time. This is done to detect exactly external changes from OpenWith.

So Sigil has to remember them and that means using a key to store that info.

But you do not want that key to be based on file contents, nor the file relative path in the zip, nor even on the file name. The zip archive has no concept of files being renamed or moved. It stores its modification time based on zip entry (relative path in the zip).

You want it to set unpacked file mod dates and somehow have them survive opening for write (which is needed for how Sigil does file caching).

This is just not possible without major changes to how Sigil works. Changes I am not willing to make when Checkpointing and Compare already exists and is based in git and works just fine and can provide both files changed and internal change details.

To make your date changes work, open any epub, standardize and rename at will and save it as an epub to create a base set of modification dates in the epub zip.

So what I have in Sigil master now is all that I am willing to do. If you and others do not find it useful, I will remove it.

Just let me know if as it stands it is not useful. Comments on this versions usefulness from others welcome as well.

Originally Posted by democrite

Thank you for the plugin work.

Thanks too for the CRC32 change. Running just one file 5 times — the one with 6700 images — and then averaging, ~30% difference. If you wish for more benchmarks, I can try later. Unsure if you consider that difference significant. I also am not sure how I feel about not being able to rename file while keeping date. For the various reasons listed, renaming too I think should keep mod dates if the file is unmodded, e.g. I may merely rename all files to extension .xhtml, or make further renames. Sometimes they may make files change, such as if there links elsewhere, other times not.

For certain publishers, those that separate each chapters xhtml and images into folders, there could be multiple duplicates of some like shared images (e.g. graphical bullets), and in such cases, merging can take a long time, I haven't tested recently on new computer, though before it'd run overnight and not complete. I before have renamed such files outside Sigil. For that and other reasons mentioned, some dialog to perform rename, merge, and perhaps other operations where one can see first toc entry, possible header title, and maybe other metadata could be helpful someday.

I still really really would like any file that is not modified to not have the date changed. Perhaps you disagree, but for me, one might say it is merely the "right" thing to do.

[democrite]

Such is still so very useful. Perhaps I’ve become as they say a bit demanding. Just concerned about and unsure of how best to approach. Perhaps someday, if possible, edit state could be more easily tracked as it seems to be with documents in OS APIs. Maybe Qy doesn’t (yet?) have a concept of document(s).

Many thanks. Still such a thing is so very important to me. What is there is truly helpful.

[KevinH]

Last night after some further thought, I think I may be able to store the additional zip file info in each Resource Object we create to wrap each file. By doing that, it should be able to withstand both move and rename. I will give that a try and push it to master if it improves upon what is in master now.

[democrite]

Thank you.

You very well know what I am hoping for. More words from me may just perhaps be tiresome. You may very well feel Sigil needs other things and other is more significant. I trust and have as they say faith that any possible future improvements if any you’ll self-realize with time and implement whenever you feel like it.

[KevinH]

Okay, I just pushed these changes to master. Please try "standardizing" an epub that starts out non-standard and in particular try renaming an image file but not editing it. Please verify that the epub zip modification times remain unchanged.

Note: The epub mimetype file, and META_INF/container.xml are not unpacked from the zip or editable in Sigil, they are created on the fly upon write. But for all files listed in the manifest and the opf file itself, the zip modification dates should hopefully now be saved and restored with no filename clashing issues.

Timings for file load should be reasonable as well. For an epub with 6000 files it will still have to build a hashmap table with 6000 entries containing the zip file info on loading an epub and then copying that info to the actual Resource object associated with that file. That will not take 0 time but it should be much faster than sha256ing the contents of 6000 files.

Please let me know how it goes when you get a chance.

[KevinH]

Okay based on this thread, related recent threads, and earlier To-Do list threads:

Here is the latest list (in no particular order, and NO PROMISES!!!!!)

Sigil To-Do List
--------------------

• Allow users to create their own replacement functions in python3 for Search and Replace (@The_book, and others)
• Some form of supported templating for HTML, TOC and Index generation (under consideration but doable by plugin) (@BeckyEbook)
• Multiple paragraph justification setting via the CV alignment icon (although PageEdit does this now) (@Vanguard3000)
• Add a regex to be used to build TOC when no headings tags exist (considering but this is against accessibility) (@patrik)
• Try to update to lastest BS4 (but be careful of license change) (dev team)
• Make search and replace of text be handled in Text only (outside tags) (dev team) But note this can be done now via regex now:
  - using this regex prefix: <[^<>]*>(*SKIP)(*F)| before the text to find
  or
  - this regex postfix after the text to find: (?=[^>]*<)
• Create way to make dual support for PyQt5 and PySide6 usable for plugin developers (dev team)
• Update Sigil official plugins to support PySide6 (dev team)
• Add GetInfo dialog to BookBrowser to add/change Semantics and Manifest Properties on a File (dev team)
• Add CSSToolbox and GUI to help users create, manipulate, and merge multiple css stylesheets/selectors (@Tex2002ans)
  • merge selectors across separate css stylesheets
  • delete selectors from a stylesheet
  • strip out a set of properties across all selectors and stylesheets
  • compare two selector property lists to find matching selectors across stylesheets
• Explore developing Spell-Check like advance editing features (@Tex2002ans)
• Find some way to report sub-counts when counting using saved search groups (@Tex2002ans)
• Update to Mathjax 3.x when possible

[KevinH]

For those keeping track since we started using To-Do threads (pre-Sigil 1.5.0)

Completed To-Do List Requests
(as of Sigil master today)
-----------------------------------

• Add Indicator in Status Bar if epub in old Sigil "Standard Format" (many)
• Add tag begin end pair highlighting to CV (@Doitsu)
• Enable automatic selections based on cursor position for CV editing (@elibrarian)
• Enable Saved Searches to read in from csv and text files to automate search and replace (@elibrarian)
• Enable Nav to include of role="doc-toc" on nav toc and similarly for page-list (@elibrarian)
• Update User's Guide (split into workable sections and ask for volunteers) (dev team, @Thasaidon)
• Simplify Metadata Editor to be understood without knowing syntax (@Doitsu, @BeckyEbook, @Tex2002ans, etc.)
• Add a regex validator to validate regex expressions used in Find and Replace (@Doitsu, @BeckyEbook)
• Add a flag to limit search to Find in only Open Tabs (@Vanguard3000)
• Make Saved Searches keep track of all the search flags they were used with (by saving options flags) (@elibrarian, @PapaKilo, and others)
• Environment Var to control writing of Sigil version to metadata (@BeckyEbook)
• Add a tool and icon to automatically remove matched begin/end tags based on cursor position (@Thasaidon)
• Add limited automate list ability to Sigil that works with plugins and a set of commands (was Big Green button) (@phillipgessert)
• Move to version 2 of the C regular expression library (PCRE2) as our search engine (dev team)
• Completely redesign Find and replace to better remember search position to allow skipping replacements in saved searches much easier. Add Restart Search button in place of overloading wrap (@Coleccionista)
• Extend Plugin interface to handle running under automation lists (dev team)
• Make template epubs easier to use (done via plugins)
• Update to embedded Python 3.9.9 for mac/win builds (dev team)
• Make building Sigil with Qt6 and Python 3.9 to support macOS M1 (arm64 native builds) (@democrite, dev team)
• Move to PySide6 (official Qt versions) to replace PyQt5 for plugins (dev team)
• Add sorting of Validation results by bookpath and error message (@Doitsu)
• Add export of Validation results to csv (@Doitsu)
• Update gumbo parser with recent WhatWG changes (dev team)
• Preserve modification dates to the extent possible from Epub zips (@democrite)
• Try to speed up worst case Add Blank HTML in huge epubs (> 1000 files) (@Tex2002ans, @Dire_Storm)
• Add blank javascript and link javascripts (@RbgJrg)
• Add ability to clear Find and Replace history (@Magess)
• Add ability to export Reports as real quoted csv (@Binchen)
• Add menu item to rename a class in a css stylesheet and have it properly propagated throughout the epub (@Tex2002ans)
• Remove the outdated Sigil Wiki
• make default shortcuts safe for intl keyboard users

[democrite]

Originally Posted by KevinH

Okay, I just pushed these changes to master. Please try "standardizing" an epub that starts out non-standard and in particular try renaming an image file but not editing it. Please verify that the epub zip modification times remain unchanged.

Thank you very much. Looks good.

Perhaps I am unfamiliar with parsing and formatting upon first open; I see some date changes of XML files such as the ADE page map and page template. From what I recall perhaps CSS too. If it is a quick answer, I am curious. No need to respond, not a big deal and I can diff files sometime if I really want to know.

Benchmarks: tried to ensure nothing else running that might use disk or CPU. Reran all. Compared against git checkout before all this began. Not sure why for larger files I used, current master is faster. Rebuilt both to make sure, but maybe I still missed something.

- 330 MB, 2700 images, 40 XHTML

w - 3.78, 3.84, 3.79, 3.77, 3.80

w/o - 4.05, 4.14, 4.07, 4.13, 4.08

- 175 MB, 5500 images, 250 XHTML

w - 10.38, 10.38, 10.49. 10.47, 10.54

w/o - 10.83, 10.75, 10.98, 10.92, 10.95

- 470 MiB, 6700 images, 20 XHTML

w - 7.41, 7.42, 7.46, 7.41, 7.49

w/o - 7.83, 7.85, 7.93, 7.97, 7.99

- 40 MB, 380 images, 30 XHTML

w - 1.30, 1.31, 1.33, 1.30, 1.29

w/o - 1.30, 1.30, 1.30, 1.30, 1.33

- 1.1 MB, 12 images, 76 XHTML

w - 0.45, 0.44, 0.44, 0.43, 0.44

w/o - 0.45, 0.45, 0.45, 0.45, 0.45

[KevinH]

When files are moved, especially when css files are moved and images are moved, each css sheet is parsed to look for url(), background images, importing other css files, etc and any that need updated due to the file move. That is true of page maps and templates that link to any files moved. So all of that is expected behaviour given the movement of that file or links into that file or out of that file.

In an unrelated change, I tried to speedup OPF parsing itself to try to speed up adding blank xhtml files. You are probably seeing those slight speedups in your timings as Importing an Epub is only a light user of opf parsing in general. Mainly just the manifest.

[democrite]

I am not sure if this is ideal place for feature requests. So far seems to be yet I haven't checked other threads. If so, maybe the first post could be edited as a running list as others who may not read this thread or regularly check forums may find such less effort to find and remain up-to-date. Is this turning into the thread for requests?

Hope you don't mind random things. As I wasn't trying to edit some EPUB but from recent testing, I was more aware of actions I tend to ignore during regular use:

- forgot to mention, it's been there for a while yet, keyboard shortcuts on macOS show ctrl instead of cmd and meta rather than ctrl. might be confusing for those who don't know what is meta.

- preference to set linear=no on add cover?, and auto-set if cover is detected?

- adding cover, often with commercial specified in some other way undetected, Sigil adds a new file. possibly instead of needing to delete the old and edit toc, maybe in that dialog some place, perhaps expandable not shown by default, to replace some existing cover?

- renaming files, if focus could remain in file list post-rename, I think easier to continue. e.g. select files, press shortcut to rename, arrow keys + selection, repeat.

- as many or most commercial EPUBs bring up the message of missing DOCTYPE, html, etc., if there is no malformed XHTML, perhaps there could be an option to do such automatically? I am unaware if suggested, spoken of before, or of any dangers in such cases.
Mentioned before, I remain unclear at times when there is indeed malformed HTML vs merely a change in header. Tested one where such was fixed, made something malformed (removed closing tag somewhere), and same message. Thus times where I may have, outside of Sigil, introduced some html syntax error of missing tag or something, vs. a harmless (?) DOCTYPE, html, … that is autorepaired, I do not know how to distinguish such.

- I was thinking, over time, I might contribute to making the app more macOS native if interested: UI elements (noticed status bar size), icons, and other. If someday buttons or other icons could use available resources such as recommended UI fonts per platform or environment (Gnome/KDE no idea if there is such) if available, as I kind of like seeing such refinement, I may experiment with finding such icons for macOS Big Sur+ (SF Symbols) or making use of the Qt API for a more native look. Perhaps uninterested or low priority, but maybe nice someday like possibly using such also on newer Windows. if qt 6 offers new such capabilities. maybe very far off yet perhaps Sigil could later slowly introduce changes to in the future make use of platform native offered by Qt.

edit: and possibly an option to set epub3 nav to linear unless also used as html toc and file is in ncx/nav TOC.

[democrite]

Originally Posted by KevinH

When files are moved, especially when css files are moved and images are moved, each css sheet is parsed to look for url(), background images, importing other css files, etc and any that need updated due to the file move. That is true of page maps and templates that link to any files moved. So all of that is expected behaviour given the movement of that file or links into that file or out of that file.

I may be mis-recalling, yet I thought the ADE files (and possibly CSS, forgot) were modded on merely open and save without restructuring, so I thought it strange. Maybe such are parsed, linefeeds changed or something. Minor but I thought it curious. Same seems to be the case from what I recall (but maybe not) with OPF. Perhaps somethings are changed merely on first open, fixing DOCTYPE, etc. and save. Perhaps to be expected.

As you mention, after first open and possible restructuring, everything is afterwards fine.

[Ashjuk]

Just a thought, but is there any chance of bundling a more comprehensive English dictionary with Sigil?

My default user dictionary now contains nearly 2000 words (all checked as being valid and correct English against two on-line dictionaries) that are not included in the dictionary that comes with Sigil.

I realise that you just pick up the Hunspell ones, but I was wondering if there was a better alternative that updated their dictionaries on a more regular basis.

As I said it's just a thought and not a big issue.

[KevinH]

Ashjuk,
On Windows and macOS we include our own copy of the hunspell en_US dictionary. So we could certainly improve our local copy. If you are sure that your 2000 long wordlist are all valid en_US words then please zip up the list, post it someplace and pm KevinH (me) with a link and when I get a chance I will add them to the en_US dictionary and rebuild our local copy.