Data loss in 0.9.5 on OS X (El Capitan)

[Peter Ahlstrom]

On 0.9.5 I'm running into a problem I haven't seen before. (And I do have the latest ActiveTcl installed.) Occasionally (and unpredictably) when saving a file, it loses its resources. Images (some but not all), fonts, even the whole META-INF folder.

Sometimes Sigil replaces the files in the epub with 0-byte empty files. And after that happens, the 0-byte files can't be deleted using the interface—it says it deletes them, but they're still there. And if I try to add a file with the same name, it says I can't add it because it already exists. (Yet if I look at the structure of the zipfile, those 0-byte files do not exist.)

The epub can then be opened back up in Sigil and the 0-byte files are gone, and I can put the missing resources back. But this is a pain, and I usually just end up restoring the epub from a backup.

The data loss has happened twice in the last week, on two separate files.

[KevinH]

I run and test on Mac all of the time and I have never seen any issues like what you describe.

So Please run DiskUtility to verify your harddisk is not corrupted and that the proper permissions exist and that you have enough disk space remaining.

Also, Do you have any anti-virus software running and if so, have you tried disabling it temporarily to see if that is an issue?

Also do you have any programs that try to clean tmp files that may becausing an issue? Some cleaner programs actualy delete tmp files when they should only do that at start-up or shutdown. This can cause missing files in epubs since the epub is being processed unpacked in tmp files.

If those are not the issue, please post an epub someplace and pm me the link here, along with the steps to reproduce/cause the problem?

Thanks,

KevinH

[Toxaris]

Originally Posted by KevinH

Also do you have any programs that try to clean tmp files that may becausing an issue? Some cleaner programs actualy delete tmp files when they should only do that at start-up or shutdown. This can cause missing files in epubs since the epub is being processed unpacked in tmp files.

This was the first thing that came up in my mind as well.

[Peter Ahlstrom]

Sigh! Now I'm trying to recreate the problem, and it hasn't happened yet. Both times it's happened before, it hasn't seemed like it was triggered by any particular thing. I'm doing the same kind of stuff now that I was doing when the error happened, but it's smooth sailing.

I guess for now I'll just pay close attention and see if it ever happens again, and if it does, I'll send you a file.

I did run Disk Utility and it's fine. I don't run antivirus (*cough cough*) and I don't have any kind of tmp cleanup utility.

[KevinH]

There are a number of bug reports about temp files mysteriously being deleted on Mac OSX el capitan creating issues for Office that uses .docx files (zip files). Just like epub are .zip files. It seems a return of an old issue.... on Mac OS X, there is a dirhelper daemon that looks for files not accessed within 3 days inside the official temp directory and will delete them even if the app using them is still running.

Here is an official apple discussion about this nonsense when it hit under Leopard.

http://lists.apple.com/archives/coco.../msg01595.html

Notice in that issue when it says if you unzip an archive with files that have a modification date of over 3 days old, dirhelper may simply delete them when it runs!!!

This is unheard of behaviour that only impacts software that unzips possibly old files inside the correct temp directory such as Sigil and Office do. For the bug to hit, it must be 3:35am and the dirhelper daemon is run and you have unmodified files unpacked from a zip by the os that uses the zip extended date information to set the original file date.

The file that determines dirhelper's time of launch is:

/System/Library/LaunchDaemons/com.apple.bsd.dirhelper.plist

So please verify your system clock and the time for dirhelper to run from that plist file to rule out any nonsense caused by this strange policy by Apple. Why this bug hits El Capitan again is beyond me?

Related Office bug reports on el capitan can be found here:

https://discussions.apple.com/thread...art=0&tstart=0

and here:

http://answers.microsoft.com/en-us/m...9953eb0?page=3

Not sure the best way to deal with this since lots of Sigil files are unzipped in the correct temp directory to remove issues with attacking temp files creation using races.

It has never hit me, but my clock is correct and I simply do not use Sigil at 3:35am so I have not seen the issue.

Please check that plist and let me know what it says and if your data losses happened near the time set in that plist according to your computers clock, so we can rule this out.

[KevinH]

I ran some tests to see what modification dates are set for epubs "unzipped" on Sigil as follows (this is under Mac OS X 10.10 not 10.11) (it should work similarly under Linux).

1. Open Sigil.app and then open any old epub you have around that you have not touched yet today

2. In Terminal.app type the following command:

 stat `find $TMPDIR -name "*"` | grep Sigil

The dates you see for the Sigil temp files are in this order (on Mac)
- date last accessed
- date last modified
- date the last file inode was changed
- date of creation of the file (not typically available on Linux)

When I run this test on Mac OS X Yosemite I can see that each and every file access, modification, inode change and birth date are today's date. This is because the unzip utility used inside of Sigil does not know how to unpack or set unix file date/time information.

So even if dirhelper runs, nothing Sigil related will be older than 3 days and so should not be deleted out from under Sigil.

If I instead cd into $TMPDIR and create my own dir there and run the command line unzip on the epub, it is smart enough to unpack old dates and properly set access, modified, and birthed and so temp may very well end up with files "older" than 3 days even though they were just created (unzipped). These files would be candidates for being deleted out from under the application that created them in temp when dirhelper is launched.

So because Sigil's built in unzip library is "stupid", Sigil should be fine even if dirhelper runs and removes temp files.

I would love to see a similar test run on El Capitan (Mac OS X 10.11) to see how it handles things to verify that this is not the source of your issues. So if anyone has El Captian and would try this, I would appreciate knowing if any of the dates shown are older than 3 days when using Sigil to edit an epub.

Thanks,

KevinH

[Peter Ahlstrom]

Kevin, this is all very interesting. I appreciate you going down the rabbit hole with the tmp idea even though I said I would drop it until I saw the error again.

Theoretically, what would happen if an ePub file is left open in Sigil over the weekend and then worked on again 3 days later? Do the unzipped files remain in the tmp folder until the ePub is closed in Sigil? Because I am sure I have done this—I left a file open in Sigil for more than 3 days, then came back and made some changes and saved it. Thinking back, that was most likely the circumstance both times this error popped up for me.

I just ran the stat command above on my El Capitan, and some of the timestamps seem to correspond to when I opened the currently-open ePubs yesterday, with others corresponding to the last time I saved the files yesterday. The directories seem to have a most recent timestamp of the moment I ran stat. Here are some of the results:

16777220 23411102 drwxr-xr-x 3 USER GROUP 0 102 "May 11 12:02:51 2016" "May 10 13:26:37 2016" "May 10 13:26:37 2016" "May 10 13:26:37 2016" 4096 0 0 /var/folders/zx/greeked/T//Sigil-PmPJ5r/META-INF

16777220 23411113 -rw-r--r-- 1 USER GROUP 0 259 "May 10 17:26:18 2016" "May 10 13:26:37 2016" "May 10 13:26:37 2016" "May 10 13:26:37 2016" 4096 8 0 /var/folders/zx/greeked/T//Sigil-PmPJ5r/META-INF/container.xml

16777220 23411144 -rw-r--r-- 1 USER GROUP 0 1420 "May 10 17:23:52 2016" "May 10 17:26:18 2016" "May 10 17:26:18 2016" "May 10 13:26:37 2016" 4096 8 0 /var/folders/zx/greeked/T//Sigil-PmPJ5r/OEBPS/Text/Acknowledgments.xhtml

16777220 23413723 -rw-r--r-- 1 USER GROUP 0 227171 "May 10 17:26:19 2016" "May 10 14:37:29 2016" "May 10 14:37:29 2016" "May 10 14:37:29 2016" 4096 448 0 /var/folders/zx/greeked/T//Sigil-QgtWuM/OEBPS/Images/cover.jpg

[KevinH]

Hi Peter,
Thanks for checking. And yes, if you left Sigil running with an epub loaded, long enough (ie over a weekend) then Mac OSX dirhelper can and will delete any files that ave not been updated within 3 days.

So leaving Sigil open and running over a weekend on Mac OS X will result in exactly the behaviour you are seeing. Missing files, etc.

So that is probably what you are seeing. It would be better to do a save-as and exit Sigil if you plan on leaving it for more than a day or two (at least on Mac OS X). Not sure how linux or Windows handles cleanup of temp files when a user does not reboot and keeps the program running over multiple day periods.

They must have some mechanism to clean up.

Thanks for checking that for me.

KevinH

[eschwartz]

Typically Linux uses a tmpfs (ramdisk) for /tmp -- which means on reboot everything gets wiped clean.
(Some distros don't use a tmpfs, in which case tmpfiles.d is used to empty /tmp as one of the tasks run on system startup.)

To me, that seems a much more reasonable way of dealing with orphaned temp files than using inaccurate date heuristics to purge random files in the course of normal operations.

[Peter Ahlstrom]

This is frustrating. I know I'm not the only one who leaves programs and documents open for days at a time (for sure I know that my boss does the same thing). I will certainly try to close the files more often, but this is so foreign to the way I work—I generally keep a file open (in whatever program) until I'm totally done with it.

The issue didn't cause the deletion of any of the html files in the ePub, so those are clearly treated differently when saving, even though they also appear in the tmp folder. Maybe there should be some kind of error checking when saving? If the tmp folder is gone, update the on-disk zipfile (i.e. using zip -u) with any text files left open in the program, instead of creating a new zipfile to replace the old one?

To eschwartz: But if the machine never gets rebooted, then the tmp files would build up.

[eschwartz]

And most temp files will get deleted by the (well-behaved) program that created them... when they are done using the files.

The rest probably don't take up enough space to worry about running low on space, at least until the next reboot.
It is certainly better than the aforementioned heuristics. Better to delete too little than too much.

[KevinH]

Sigil actually does clean up its temp files when it exits.

[KevinH]

Peter,
If you work that way with other programs (like office) you should be able to just change the plist file I cited earlier you can set it to a much longer days between if you want to keep things around longer. As with Linux, this temp dir will be cleaned upon reboot. Also Sigil will properly clean up after itself too.

There may be ways for a user to set its own temp directory via the environment var TMPDIR, but I am not sure where Qt gets its Tempdirectory location from.

If you can somehow override the temp directory location, you can set it to your own folder in your home directory and not worry about Mac OSX cleaning anything underneath you.

KevinH

[KevinH]

Hi Peter,
All open html tabs are cached in memory and therefore always available and written back to disk just before it is saved. That approach will not work for tabs not open, and missing source files can cause segfaults and other strange behaviour.

Your best bet if you work this way in many programs is to simply change that plist file to be 5 or even 7 days.

/System/Library/LaunchDaemons/com.apple.bsd.dirhelper.plist

I will look at adding an option in a future release of Sigil for a user to specify his/her own existing temp directory or default to the system one. That way you can create your own temp folder specific to Sigil and then you can keep Sigil running as long as you want.

Probably won't be for the next release but should happen in the one after that.

KevinH

[Peter Ahlstrom]

Thanks. I'll go ahead and edit the plist. (I missed it when you mentioned it the first time.)

EDIT: Oh great, in order to change anything in the /System directory under El Capitan, I have to reboot into recovery mode and turn off System Integrity Protection. So...this will have to wait until I'm prepared for a reboot. Thanks, Apple.