Possible To-Do List for Future Sigil Releases Post Sigil 1.8

[Ashjuk]

I have no idea if this is possible or not but one item on my wish list would be the ability to run find and replace on a selection of code.

Currently, as far as I can tell, the only options for F&R are for whole files. What I would like to do is to be able to highlight a section and perform a find and replace operation only on that and not the whole file.

You are probably going to tell me it can't be done, but it is worth asking anyway.

Merry Christmas and a Happy New Year everyone!

[Turtle91]

Designate the selected text as “marked text” (highlight then Ctrl-shift-m iirc)…

Hit the space bar to remove the selection.

[Ashjuk]

Originally Posted by Turtle91

Designate the selected text as “marked text” (highlight then Ctrl-shift-m iirc)…

Hit the space bar to remove the selection.

Thanks - that's just what I was looking for.

[KevinH]

Okay ... I can see the general usefulness of keeping modification dates in zips/epubs.

I still think using Sigil's Checkpoint and compare built in tools (that require no knowledge of git to function) are a much better way to compare two versions of the "same" epub but if all you want to see is which files changed, the mod dates would be useful.

So I took a shot at implementing this in master. I tested it and it seems to work but the first time loading any old epub into Sigil, will update doctypes, add xml headers, and that can end up modifying many files just once. After that initial load and update, then the zip modification dates become more useful.

Note: all of this only works when Importing and exporting an epub. Any use of Add Existing will make the file be marked as newly modified in the next epub save since it will be new to that zip.

This required cryptographic hashes be used to detect file modifications (SHA 256), and if that ends up slowing down import or export, I will end up making this a user preference feature that needs to be enabled so it will not impact others who do not care about zip modification dates.

Since you build your own, please try a fresh Sigil build from Sigil master (it has all my personal repo changes already in it) and let me know if this is what you and others asked for.

KevinH

Originally Posted by democrite

As a mention, I cannot overemphasize enough how much I hope someday, and how much it means to me (and maybe others). I'd think despite unzipping to a temp folder, archive dates could be preserved, and if edit state is tracked, perhaps just such files would have the date changed on save.

It may be a little while before I get to trying it myself; maybe I could figure it out with enough effort, yet maybe the change isn't do bad, if merely you thought too it to be important and were willing.

As for using checkpoints, as mentioned, I may just update metadata, restructure to Sigil Norm, compress files (typically outside w/external tool), edit CSS, subset fonts, edit TOC, or other, or some combination of all or a few or one. E.g., for commercial epubs, I may merely make a few changes or substantially change. It would be difficult to checkpoint every EPUB. Imagine doing such for one's entire calibre library? And if someday Sigil is no longer the tool of choice? Years later, if I wish to know if I've compressed images or what is the possible (corrected) revision of some purchased EPUB (dates is one indicator), or have some other question, much can be determined, at least enough for me for various reasons, that such dates are so significant. You yourself seemed to have suggested mod dates are too important.

[democrite]

I cannot thank you enough. That was the one thing I've wanted with certainty for so long, yet never thought of asking.

Just did one test. Seems ok on initial save. Then I tried restructure to Sigil Norm -- I think that's what did it --, all dates were changed. In such a case, I'd think images, mimetype, CSS, and maybe other should be unchanged if unmodified.

Concerning performance, for this purpose, maybe there are faster hashes? A quick look, there are many SO questions about such; and at least a few reports that the fastest seems to be xxHash, or possibly choose one of the faster ones that perhaps has more of a history (MurmurHash or SpookyHash?). Of links I found this may be helpful:

Non-crypto hashes in C++: Farm, City, xxHash, MUM, Spooky 2, Murmur, Metro and t1ha0
https://asecuritysite.com/encryption...e%20lazy%20dog

Of how much of speed difference there is on open and save, mine may not be a good indicator of relative performance compared to some average computer, but I think I can either thru the command-line or AppleScript run some tests and report back on time differences. I have some fairly large EPUBs, a few with thousands of images. That might be helpful.

[KevinH]

If you standardize to Sigil norm you will move image files (and almost all files) plus any links or urls to them so there is very little chance of files not changing. Moving a file or renaming a file is going to be equivalent to deleting the original and creating a new (copy) of the file in a new location or under a new name.
Remember, the same file name can exist in different folders in the same zip/epub so the modification date must tied to the full path to the file in the zip not just a filename.
Hope this explains what you reported.

The only alternative is to tie modification time to sha256 checksum instead of path to file in epub. If we do it that way, just moving or renaming a file would not impact its modification date in the new zip. I will give that a try to see if that is more useful.

As for hashes, we are sticking to QCryptographicHashes available back to Qt 5.10 or earlier to stay cross-platform. Again if it impacts load time significantly, we will make it a user preference setting.

Originally Posted by democrite

I cannot thank you enough. That was the one thing I've wanted with certainty for so long, yet never thought of asking.

Just did one test. Seems ok on initial save. Then I tried restructure to Sigil Norm -- I think that's what did it --, all dates were changed. In such a case, I'd think images, mimetype, CSS, and maybe other should be unchanged if unmodified.

Concerning performance, for this purpose, maybe there are faster hashes? A quick look, there are many SO questions about such; and at least a few reports that the fastest seems to be xxHash, or possibly choose one of the faster ones that perhaps has more of a history (MurmurHash or SpookyHash?). Of links I found this may be helpful:

Non-crypto hashes in C++: Farm, City, xxHash, MUM, Spooky 2, Murmur, Metro and t1ha0
https://asecuritysite.com/encryption...e%20lazy%20dog

Of how much of speed difference there is on open and save, mine may not be a good indicator of relative performance compared to some average computer, but I think I can either thru the command-line or AppleScript run some tests and report back on time differences. I have some fairly large EPUBs, a few with thousands of images. That might be helpful.

[democrite]

Originally Posted by KevinH

The only alternative is to tie modification time to sha256 checksum instead of path to file in epub. If we do it that way, just moving or renaming a file would not impact its modification date in the new zip. I will give that a try to see if that is more useful.

Pretty pretty please, I hope that is possible. I would otherwise need to be conscious that first I would restructure to Sigil norm before any other changes. That is not so bad but if it isn't too much effort, other would be great.

I haven't had a chance to look much at the code yet recently I've been thinking I will start more. I do not know if other file metadata is somehow kept track of. Perhaps such would help for possible future features. I recall, though I do not know if such has changed, that when adding links to IDs (I think it was) that there was some delay for larger EPUBs. In such cases, I have a fair amount of files with thousands or tens of thousands of IDs, common with academic works from certain publishers. Maybe the indexing of Sigil could grow over time such that more becomes possible. I was thinking of navigation features such as goto (id, text content, or title). Perhaps you are used to such in whatever editor or IDE you use; such would too be nice someday for Sigil.

When you say restructuring moves files, I'm not sure if that means such is really what happens in the temp folder. If so, I do not know if that is necessary over moving the file. Would such be less efficient? In my case of APFS, disk usage and battery life wouldn't be affected though unsure about NTFS, ext4, or other Linux file systems.

Of some informal testing of speed of opening an EPUB, with one with ~6000 images, 13 seconds vs 8.

edit: I'd guess restructuring to Sigil norm also recalculates checksums. Of a somewhat medium sized file, 1200 XHTML and ~250 images, restructuring took a while. I didn't time it but it seemed like half a minute or more. You may want to make such a preference. A faster hashing algorithm too might well be worth investigating if possible.

[KevinH]

A few things:

There is *never* a need to Restructure an epub to Sigil norm. Sigil norm is just an epub layout that old Sigil FORCED your epub into just to open it. It is no better than any other adhoc epub layout. And restructuring an epub to an arbitrarily chosen layout need never be done more than once (if at all).

Restructuring does not use cryptograohic hashes. Restructuring does in fact move actual files in Sigil's temp (having underlying actual files is needed for things like listening for external changes to those files via open with) even though caches of file contents for quicker editing are used. Moving then needs to fully gumbo parse *all* xhtml files to fix and update all relative links everyplace as well as update all css urls and file references and update the nav/toc. The OPF must also be fully parsed and updated.

Being an editor means that file hash tables and advance indexing are worthless, as the only thing you can index in a file is file position and that becomes worthless after a single character change.

So worker threads are used where ever possible to concurrently parse all xhtml files in the fly to find ids, etc. only when needed.

I still find the thought of 1200 chapter files being truly needed for even any textbook to be absurd. 1200 pages maybe but 1200 chapters just means someone has not thought through the best organization and structure.

I will look into storing zip mod dates by file contents hash and not file path.

Originally Posted by democrite

Pretty pretty please, I hope that is possible. I would otherwise need to be conscious that first I would restructure to Sigil norm before any other changes. That is not so bad but if it isn't too much effort, other would be great.

I haven't had a chance to look much at the code yet recently I've been thinking I will start more. I do not know if other file metadata is somehow kept track of. Perhaps such would help for possible future features. I recall, though I do not know if such has changed, that when adding links to IDs (I think it was) that there was some delay for larger EPUBs. In such cases, I have a fair amount of files with thousands or tens of thousands of IDs, common with academic works from certain publishers. Maybe the indexing of Sigil could grow over time such that more becomes possible. I was thinking of navigation features such as goto (id, text content, or title). Perhaps you are used to such in whatever editor or IDE you use; such would too be nice someday for Sigil.

When you say restructuring moves files, I'm not sure if that means such is really what happens in the temp folder. If so, I do not know if that is necessary over moving the file. Would such be less efficient? In my case of APFS, disk usage and battery life wouldn't be affected though unsure about NTFS, ext4, or other Linux file systems.

Of some informal testing of speed of opening an EPUB, with one with ~6000 images, 13 seconds vs 8.

edit: I'd guess restructuring to Sigil norm also recalculates checksums. Of a somewhat medium sized file, 1200 XHTML and ~250 images, restructuring took a while. I didn't time it but it seemed like half a minute or more. You may want to make such a preference. A faster hashing algorithm too might well be worth investigating if possible.

[KevinH]

Okay, just pushed a sha256 hash based storage for zip modification date approach top master.

Caveats: if not tied to the file's relative path in the zip and instead tied to the file's contents (via a hash), it is possible to confuse the zip modification dates of two files that are absolutely identical in everything but zip modification date. But that is just something to live with if this is more useful.

Please give that a try. If that is more useful, please let me know and I will add a General Setting box, something along the lines of:

[x] Preserve Zip Entry File Modification Dates when possible


But before adding any setting I would like to see epub loading times comparing loading the same large epub 5 times both before and after these changes so we know what if any impact it has on loading times.

It might not impact things much as it could be dwarfed by the zip decompression times and file read/write times.

[KevinH]

Based on my timings comparing Sigil 1.8.0 official using Qt5.12.9 versus Sigil master using Qt 6.2.2 with the zip modification date storage, their appears to be little to no noticeable change in loading times of a large epub with many files. So the sha256 hashing is dwarfed by the zip decompression and file writing times. So their is probably no need for a user preference setting. But more testing with even bigger files would be helpful.

[democrite]

Many thanks. Building now and will try to report back later today.

Concerning the file with 1200 XHTML files, agree it is poorly made. It is a dictionary-like reference work, with each entry a separate file. It also contains links such as see also (some other entry) though at least such is tied to header id rather than just file, so I can merge entries by letter. In such a case, some advanced merging such as filtering in some dialog that shows file name, header title, and TOC entry would be helpful. In other cases, I often see file names by chapter/article DOI by certain publishers, and that too would be nice there rather than needing to look at each file to determine what chapter it belongs to. Other publishers, somewhat fairly common, may make each chapter section a separate file. There are various such cases where restructuring, merging, and renaming files often make an EPUB easier to work with.

Such restructuring, I often do it since it makes later editing, often for corrections or additions such as linking references to other sections or bibliographic entry much easier when the file name is something useful such as section/chapter, plus with front and back matter ordered in the file list before and after. Certain publishers also organize chapters by:

[OPS/OEBPS]/[DOI NUMBER]/[IMAGES]/[IMAGE FILES]

or some other that is difficult to navigate when later editing, so restructuring I've gotten into the habit for all. Plus as CSS is often in the same folder as XHTML with many publishers, a separate folder makes it much easier to find later.

As far as indexing and possibly keeping some internal in memory structure of files, IDs, headers, etc., I am aware earlier editors such as Emacs and Vi(m) may use ctags, though as you know, many editors have been created since. Many offer advanced indexing and are able to do such quick dialogs to navigate to file, class, symbol, function, etc., or show usages, with some background indexing, perhaps only needing to update entries of changed files, or dynamically indexing in the background. JetBrains perhaps offers the most advanced indexing and navigation, with CLion for C++ if you wish to try and see how useful it really is. EPUBs can get complex enough where such navigation I think someday would be very nice.

Of the testing of that one EPUB, I just timed it by watching the clock and did do it three times and it seemed consistent. I'll later try with other EPUBs.

A small detail, might you consider changing the macOS Sigil icon to be of the Big Sur plus style? You can find one here, or perhaps easily make one. Perhaps minor though for now I copy the icon after each build.

https://macosicons.com

[KevinH]

I think the Sigil icon is fine as it is. I am just not worried about making Sigil look more native as it is a cross platform app.

Are you aware of the goto link or style right functions and Find and Replace across all xhtml files. It already will take to directly to the destination of a link, or to its class destination.

Again, indexing is not really valuable as files change even in the background as indexing would force unneeded cache flushes back to disk.

If you want indexing you should set and use Open With to take advantage of your editor of choice that has that feature. Or use the external XEditor settings to open multiple files in an external IDE.

Take care,

Kevin

[democrite]

Of using an external editor, maybe I am not sure how it works; I see only one file open a time, and not necessarily the one selected. Perhaps a command to send the enclosing folder of the whole EPUB to an editor? If such could also be automated, perhaps AppleScript is too much to ask, but from the command-line, that could make it one step from calibre or some other source.

Some benchmarks. Done with AppleScript tell application "Sigil" open …, using the built-in timer of Script Debugger. Unsure how optimized SHA is for Intel vs Apple Silicon.

470 MiB - 6700 images, 20 XHTML

w: 10.55, 11.49, 10.55, 10.51, 10.41
w/o: 6.49, 6.50, 6.50, 6.51, 6.45

175 MiB - 5500 images, 250 XHTML

w: 10.95, 17.67, 19.14, 10.68, 10.60
w/o: 
8.80, 8.88, 9.04, 9.05, 9.99

330 MiB - 2700 images, 40 XHTML

w: 7.97, 6.20, 6.15, 8.14, 6.62, 6.12
w/o: 3.33, 3.68, 3.38, 3.36, 3.34, 3.34

40 MiB - 380 images, 30 XHTML

w: 2.46, 1.59, 1.65, 1.44, 1.39
w/o: 1.05, 1.05, 1.06, 1.03, 1.05


[KevinH]

Well those differences seem pretty significant. Perhaps you are right and there is no optimized sha256 on macOS arm64.

I will go ahead and add a user preference to enable / disable it.

But looking at what file_info is stored in zip and is freely available:

typedef struct unz_file_info_s
{
    uLong version;              /* version made by                 2 bytes */
    uLong version_needed;       /* version needed to extract       2 bytes */
    uLong flag;                 /* general purpose bit flag        2 bytes */
    uLong compression_method;   /* compression method              2 bytes */
    uLong dosDate;              /* last mod file date in Dos fmt   4 bytes */
    uLong crc;                  /* crc-32                          4 bytes */
    uLong compressed_size;      /* compressed size                 4 bytes */
    uLong uncompressed_size;    /* uncompressed size               4 bytes */
    uLong size_filename;        /* filename length                 2 bytes */
    uLong size_file_extra;      /* extra field length              2 bytes */
    uLong size_file_comment;    /* file comment length             2 bytes */

    uLong disk_num_start;       /* disk number start               2 bytes */
    uLong internal_fa;          /* internal file attributes        2 bytes */
    uLong external_fa;          /* external file attributes        4 bytes */

    tm_unz tmu_date;
} unz_file_info;

I could create a key from the pre-calculated crc32 value converted to a hex string, and append the basename (not the full path) of the file, and the file size in bytes again converted to a string.

Then I could add and use a crc32 checksum calculator only when saving the epub.

That should pretty much make speed a non-issue. Might be worth a shot.

[KevinH]

Okay, I just pushed using the stored zip crc32, the stored zip uncompressed size, and the file name ( with all paths removed) as the key to store and retrieve zip modification times.

Please give this a try with your multiple test runs. Importing times should return to what they were previously (ie. be faster). Files can be moved around but not renamed and still keep their modification dates on Export. And Export times should be faster since crc32 is faster than an sha256.

Hopefully this version is better and will not need a Preference change.