Oddity in Mend and Prettify

[DNSB]

I was looking at an epub for another person who was running into errors and ran into an oddity using Mend and Prettify.

An example is one line originally has a missing " after the bold while another had a missing " before the bold:

  <p>This is a <span class="bold>test line</span> with a missing " after the bold.></p>
  <p>This is a <span class=bold">test line</span> with a missing " before the bold.></p>

After Mend and Prettify, the lines read:

  <p>This is a <span class="bold&gt;test line&lt;/span&gt; with a missing " after="" the="" bold.=""></span></p>
  <p>This is a <span class="bold"">test line</span> with a missing " before the bold.&gt;</p>

The whole ePub was full of similar items since the author manually added most of the spans in violation of common sense.

[KevinH]

Yes, Xhtml spec requires all attributes to be quoted - either with single or double quotes.

If they had run Sigil's well formed check, it should detect things like unopened or unclosed attributes and warned them but sometimes if a suitable quote is found "nearby", even that gets confused.

The missing end quote means the attribute value actually encompassed the closing > and converted it to & lt ; to prevent its loss since > are not allowed inside attribute quoted strings without them being xml encoded. It repeated that and converted bare text to attributes and ended that attribute later.

The missing quotes around an attribute are really hard to detect at times and this was one of them. Luckily, all text has been either encoded, made into empty attributes or saved in some way. So you can fix it by hand.

That said, gumbo follows the same whatwg spec for xhtml so Preview was seeing that code and internally doing pretty much the same thing as gumbo showed.

Please try Sigil's built in well formed check or epubcheck on the initial broken attribute code to see if either would better warn you of the breakage properly.

[KevinH]

Actually, I tested your example and Preview clearly shows the following error message.
So Preview is interpreting that text the exact same way gumbo did in Mend and Prettify.

And Sigil's well formed check detected the missing unclosed attribute value and the Validation window showed:

Section0001.xhtml Line 11: Offset N/A  Message: Attribute "class" has unmatched quotes on attribute value.  near column 15

Unfortunately, I did not design the whatwg/gumbo parser but it seems that missing quotes in attribute names is not a common enough occurence to create a better self mend process probably because real html does not need attributes to have quotes or even have any value.

Not much I can do here. Both Preview and Sigil's own Well-formed check indicate there is a problem and Sigils' own well-formed check even correctly identifies what is wrong.

In general making sure everything is well formed and that Checkpoints are made before running Mend or Mend and Prettify or any Sigil Tool is probably the best approach.

[DNSB]

Any of well-formed check, epubcheck or epubveri options works to warn me of the errors.

I triggered the issue when the original ePub did not believe in line breaks so each file was a solid mass of text and I pretty-printed it using mend and prettify from the Tools menu. I dumped that version, reopened the original and pretty-printed it. After that, I ended up using search/replace & regex to do most of the cleanup to get the book ready to be published. The author couldn't remember which web page she used to do the AI assisted conversion from her word processor file to ePub but will not use it again.

[DiapDealer]

I'm fairly certain that BeautifulSoup can correct mismatched and/or missing attribute quotes by parsing and re-serializing. If so, a plugin to correct something like this shouldn't be terribly difficult.

[KevinH]

Certainly worth a shot. We could include it as python3lib code and run it before every Mend and Prettify. But I did not think BS4 when using the lxml parser for xml/xhtml did that? Maybe only the html5lib parsing then reserializing as xhtml?

[DiapDealer]

Originally Posted by KevinH

Certainly worth a shot. We could include it as python3lib code and run it before every Mend and Prettify. But I did not think BS4 when using the lxml parser for xml/xhtml did that? Maybe only the html5lib parsing then reserializing as xhtml?

It very well could be the html5lib parser now that you mention it.

[DiapDealer]

For the record, none of the parsers I've tried this with will fix the unbalanced attribute quotes, and properly escape the naked quote (nor eliminate the extraneous >) in the sample provided.

Seems like a non-starter.

[KevinH]

That is what I feared. The current whatwg spec when it starts to parse an attribute really does not handle these cases at all.

The only way this is detected by our well formed check is that it parses by extracting the tag first (ie. it assumes that the > char truly ends the tag and is not something someone just forgot to xml encode) and with a known tag end in hand the missing end quote can be detected. But the modern whawg tokenizer does not work that way, and so the > char does not end the tag, and the error is treated as a missing xml encoding of the > char and building up the attribute value continues merrily along.

Neither way is guaranteed correct. They just make different assumptions as to the cause of the error.

I would think that missing ending quotes on attribute values in xhtml is much more prevalent than a spurious > char that was not xml encoded by mistake. But I have no data to back that view up.

[BeckyEbook]

I have a proposal to consider: Safe Mend.
Before running Mend, we first run "Well-Formed Check EPUB." If there are no errors, we proceed as usual.
If there are errors, we display the following message to the user: "Mending canceled: XML not well-formed."

Note that this isn’t anything new. This option already exists in Sigil when running "Restructure EPUB to Sigil Norm" or "Merge".

It could be optional, but I think it would be useful.

A reference from three years ago, where we struggled with various types of quotation marks surrounding the value of an attribute.

[KevinH]

Unfortunately, there is no such thing as perfectly safe Mend. Only the user knows what they actually wanted for the code they wrote that does not follow the rules.

Mend by definition just uses the whatwg tokenizing and parsing specification (ie. google's gumbo parer) that forces the given xhtml/html to be well-formed enough that it can can create a DOM tree. All Mend does is take that created internal DOM tree and serialize it so you can see exactly how your code was interpreted by the browser. This is exactly what browsers do.

Mend properly handles most errors and almost never actually loses text. It is and was much much safer than Tidy ever was.

So using Mend is generally just fine for fixing most errors. So running a well-formed check and only then running Mend if it passes being well-formed is typically overkill and also no guarantee of correctness. But treating your code the exact same way a browser does is valuable so you can see what is different.

So it is much much better to get into the habit of creating a Checkpoint first before running any Sigil tool and using Sigil's diff function to examine the changes.

Or at least checking each page in Preview to verify there are no current parsing warning pink warnings at the top of each page.

Now if you said that passing a wellformed check before running prettify was a good idea, I think I would agree in general and something we can play with to decide.