Losing content

exaltedwombat · 06-04-2015, 05:56 AM

Twice recently, when working on two different ebooks in Sigil (not simultaneously) I have lost large chunks of content. Running the epubcheck plugin revealed a number of orphaned images - they were in the images folder but not used in the book. Half of a chapter proved to have gone missing. Another time, the contents of two whole chapters - everything between <body> and </body> - had gone.

I have a suspicion this may have happened after performing a batch rename of html files - selecting a range and entering "chapter_01" to achieve automatic chapter numbering in the filename.

These were both books that had started life as PDF, converted to EPUB in Calibre then brought into Sigil for editing. The code was pretty clean - just the usual misinterpretations of line ends/paragraphs and intrusive page headers/numbers.

Just me, or is anything known?

Latest Sigil version, Windows 7.

Toxaris · 06-04-2015, 07:42 AM

The only time I witness this behavior is after a wrongly defined regex and accidentally let Sigil autocorrect the errors.

exaltedwombat · 06-04-2015, 07:54 AM

Thanks, Toxaris. Nothing like that going on in this case.

DiapDealer · 06-04-2015, 08:22 AM

Quote:

Originally Posted by exaltedwombat

Another time, the contents of two whole chapters - everything between <body> and </body> - had gone.

I've seen splits result in the original file having nothing between the <body> tags. But in those cases, the content was all there in the newly created file.

But that's not a bug, that's just splitting at the beginning of a file before any of the content (where it doesn't make sense to split).

rubeus · 06-04-2015, 10:11 AM

Using Sigil from 0.3.4, hat some errors but nothing as you described

KevinH · 06-04-2015, 10:13 AM

Try doing a file save-as to a new file name just before invoking the batch rename. If you see the issues afterwards then please post the just saved file (privately if need be) so that we can see what is going on.

KevinH

theducks · 06-04-2015, 11:52 AM

Never seen this happen on proper (even if it is really messy

) code
Possibly (not really a bug. GIGO

)

Lousy (malformed?) XHTML code.

validate before split/global-S&R. Validate AGAIN after a split/global-S&R BEFORE SAVING

(java) Scripted Code in the body. Can confuse things

NOW Using Sigil 0.8.6 (W7, XP) but have used every version in the past (except .1

)

xovo · 07-16-2015, 04:23 PM

I've encountered the arbitrary loss of book content from time to time without being able to pin down the cause over the years. Today I finally found one cause in particular that is similar to your problem exaltedwombat:

If I have a poorly formed EPUB that gives me the "This EPUB contains errors" warning, renaming a CSS file in the Styles folder will cause content from the specific poorly formed file to be dropped. My guess is it has to do with Sigil's automatic updates to the stylesheet links in the <head> of the files when a stylesheet is renamed.

I run Sigil 0.8.6 on Windows 8.1.

KevinH · 07-16-2015, 05:52 PM

Hi,
Again, if you can recreate this reliably, please post a sample document and the exact set of steps taken to cause this to happen, and I would be happy to look at fixing this.

Thanks,

KevinH

Turtle91 · 07-16-2015, 11:53 PM

Quote:

Originally Posted by xovo

If I have a poorly formed EPUB that gives me the "This EPUB contains errors" warning, renaming a CSS file in the Styles folder will cause content from the specific poorly formed file to be dropped.

Yes, I have had this happen as well. If I get this warning (usually when opening an epub that has been Kindleunpacked via Calibre) I immediately go and fix the header of the (usually) single html file. It usually has just <html><head> instead of the

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>

When I update the header Sigil is happy and there is no loss of data.

p.s. I'm not saying this happens all the time from Kindleunpack...most of the time they are fine... I only mention it because that is the process I use to de-stinkify my books... I fully believe that some of the files I get from Amazon aren't the easiest for kindleunpack to work with.

xovo · 07-17-2015, 12:27 PM

Quote:

Originally Posted by KevinH

Hi,
Again, if you can recreate this reliably, please post a sample document and the exact set of steps taken to cause this to happen, and I would be happy to look at fixing this.

Thanks,

KevinH

Hi KevinH,
Here's a simple way to recreate the problem in the attached css_loss.epub file:

1. Open the file in Sigil.
2. Open the css_loss.html file. It will be 2737 lines long.
3. Remove the slash at the end of the self-closing <a> tag in line 17. (Anything that will cause Sigil to note that "EPUB saved, but not all HTML files are well formed" will work. Removing a closing tag somewhere is the quickest.)
4. Open the Styles folder. Rename one of the CSS files.
5. The css_loss.html file will be truncated close to the line that causes the HTML to be malformed. For example, if you remove the closing <li> tag from line 127 of the css_loss.html file and then rename one of the CSS files, the css_loss.html file will be truncated to line 132.

I run Sigil 0.8.6 on Windows 8.1. If you have the time to check it out that's great, but no worries otherwise. I manage to avoid doing this to myself 99% of the time.

KevinH · 07-17-2015, 12:43 PM

Hi,

One further piece of information - what are your preference settings for auto-fixing the file on import or save with Tidy? Have you enabled or disabled that?

In general, it is never a good idea to run an automated update utility on a malformed file (ie. on a file an xml parser can not properly parse - and so barfs on it). The only thing I can do without allowing auto-fixing is to prevent use of any set of automated changes (renaming files, deleting files, creating toc, editing metadata, etc) if any html file is not well-formed.

This will require a very fast well-formed check that can be quickly run over all html files before allowing the command to proceed.

Not sure what else I can do. The upcoming Sigil 0.9.X series that replace Tidy with googles gumbo html5 parser which is self-reparing by the very definition of html5 parsing. I will check your test case against current master to see what impact it may have.

Thanks!

KevinH

xovo · 07-17-2015, 04:41 PM

I keep the open and save clean source options turned off.

DiapDealer · 07-17-2015, 05:39 PM

Just checked, and it doesn't seem to matter if the Clean Source settings are checked/unchecked. Once I rename the css file and hit enter, the css_loss.html file's content disappears before my eyes.

KevinH · 07-20-2015, 11:16 AM

Hi,
When using Sigil master which uses google's gumbo parser (self-repairing using official browser parsing rules), text is not lost.

So I am going to hold off on fixing this because hopefully it will be a non-issue when the Sigil 0.9.X series is finally released. If not, I will make changes then.

Thanks for the bug report and the repeatable test case,

KevinH

06-04-2015, 05:56 AM	#1
exaltedwombat Guru Posts: 878 Karma: 2457540 Join Date: Nov 2011 Device: none	Losing content Twice recently, when working on two different ebooks in Sigil (not simultaneously) I have lost large chunks of content. Running the epubcheck plugin revealed a number of orphaned images - they were in the images folder but not used in the book. Half of a chapter proved to have gone missing. Another time, the contents of two whole chapters - everything between <body> and </body> - had gone. I have a suspicion this may have happened after performing a batch rename of html files - selecting a range and entering "chapter_01" to achieve automatic chapter numbering in the filename. These were both books that had started life as PDF, converted to EPUB in Calibre then brought into Sigil for editing. The code was pretty clean - just the usual misinterpretations of line ends/paragraphs and intrusive page headers/numbers. Just me, or is anything known? Latest Sigil version, Windows 7.

07-20-2015, 11:16 AM	#15
KevinH Sigil Developer Posts: 8,483 Karma: 5703586 Join Date: Nov 2009 Device: many	Hi, When using Sigil master which uses google's gumbo parser (self-repairing using official browser parsing rules), text is not lost. So I am going to hold off on fixing this because hopefully it will be a non-issue when the Sigil 0.9.X series is finally released. If not, I will make changes then. Thanks for the bug report and the repeatable test case, KevinH Last edited by KevinH; 07-20-2015 at 11:30 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
PW losing responsiveness	pegasi&prefects	Amazon Kindle	8	01-01-2015 05:18 AM
Losing the will	ehani	Amazon Kindle	40	02-17-2014 08:26 AM
Losing Wedding Ring Less Stressful Than Losing Mobile Device	xg4bx	General Discussions	41	07-04-2012 07:16 AM
Getting Full Content from Partial Content Feeds	thread314	Calibre	5	05-05-2012 10:49 AM
Amazon losing the e-book pricing battle: Will content trump distribution?	kjk	News	0	02-05-2010 11:32 AM

06-04-2015, 07:42 AM	#2
Toxaris Wizard Posts: 4,520 Karma: 121692313 Join Date: Oct 2009 Location: Heemskerk, NL Device: PRS-T1, Kobo Touch, Kobo Aura	The only time I witness this behavior is after a wrongly defined regex and accidentally let Sigil autocorrect the errors.

06-04-2015, 07:54 AM	#3
exaltedwombat Guru Posts: 878 Karma: 2457540 Join Date: Nov 2011 Device: none	Thanks, Toxaris. Nothing like that going on in this case.

06-04-2015, 10:11 AM	#5
rubeus Banned Posts: 272 Karma: 1224588 Join Date: Sep 2014 Device: Sony PRS 650	Using Sigil from 0.3.4, hat some errors but nothing as you described

06-04-2015, 10:13 AM	#6
KevinH Sigil Developer Posts: 8,483 Karma: 5703586 Join Date: Nov 2009 Device: many	Try doing a file save-as to a new file name just before invoking the batch rename. If you see the issues afterwards then please post the just saved file (privately if need be) so that we can see what is going on. KevinH

06-04-2015, 11:52 AM	#7
theducks Well trained by Cats Posts: 30,909 Karma: 60358908 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	Never seen this happen on proper (even if it is really messy ) code Possibly (not really a bug. GIGO ) Lousy (malformed?) XHTML code. validate before split/global-S&R. Validate AGAIN after a split/global-S&R BEFORE SAVING (java) Scripted Code in the body. Can confuse things NOW Using Sigil 0.8.6 (W7, XP) but have used every version in the past (except .1 )

07-16-2015, 04:23 PM	#8
xovo Junior Member Posts: 3 Karma: 10 Join Date: Jul 2015 Device: Kindle Fire, iPad 2, Nexus	I've encountered the arbitrary loss of book content from time to time without being able to pin down the cause over the years. Today I finally found one cause in particular that is similar to your problem exaltedwombat: If I have a poorly formed EPUB that gives me the "This EPUB contains errors" warning, renaming a CSS file in the Styles folder will cause content from the specific poorly formed file to be dropped. My guess is it has to do with Sigil's automatic updates to the stylesheet links in the <head> of the files when a stylesheet is renamed. I run Sigil 0.8.6 on Windows 8.1.

07-16-2015, 05:52 PM	#9
KevinH Sigil Developer Posts: 8,483 Karma: 5703586 Join Date: Nov 2009 Device: many	Hi, Again, if you can recreate this reliably, please post a sample document and the exact set of steps taken to cause this to happen, and I would be happy to look at fixing this. Thanks, KevinH

07-17-2015, 12:43 PM	#12
KevinH Sigil Developer Posts: 8,483 Karma: 5703586 Join Date: Nov 2009 Device: many	Hi, One further piece of information - what are your preference settings for auto-fixing the file on import or save with Tidy? Have you enabled or disabled that? In general, it is never a good idea to run an automated update utility on a malformed file (ie. on a file an xml parser can not properly parse - and so barfs on it). The only thing I can do without allowing auto-fixing is to prevent use of any set of automated changes (renaming files, deleting files, creating toc, editing metadata, etc) if any html file is not well-formed. This will require a very fast well-formed check that can be quickly run over all html files before allowing the command to proceed. Not sure what else I can do. The upcoming Sigil 0.9.X series that replace Tidy with googles gumbo html5 parser which is self-reparing by the very definition of html5 parsing. I will check your test case against current master to see what impact it may have. Thanks! KevinH

07-17-2015, 04:41 PM	#13
xovo Junior Member Posts: 3 Karma: 10 Join Date: Jul 2015 Device: Kindle Fire, iPad 2, Nexus	I keep the open and save clean source options turned off.

07-17-2015, 05:39 PM	#14
DiapDealer Grand Sorcerer Posts: 28,358 Karma: 203720150 Join Date: Jan 2010 Device: Nexus 7, Kindle Fire HD	Just checked, and it doesn't seem to matter if the Clean Source settings are checked/unchecked. Once I rename the css file and hit enter, the css_loss.html file's content disappears before my eyes.

Advert

Advert