Entities oddities / 0.9.991 bug?

[KevinH]

The very first time an epub file is loaded we run it through the following code to figure out its encoding, convert it to utf-8, and change the line endings ...

See: Sigil/src/Misc/HTMLEncodingResolver.cpp

// Accepts a full path to an HTML file.
// Reads the file, detects the encoding
// and returns the text converted to Unicode.
QString HTMLEncodingResolver::ReadHTMLFile(const QString &fullfilepath)
{
    QFile file(fullfilepath);

    // Check if we can open the file
    if (!file.open(QFile::ReadOnly)) {
        std::string msg = file.fileName().toStdString() + ": " + file.errorString().toStdString();
        throw (CannotOpenFile(msg));
    }

    QByteArray data = file.readAll();

    if (IsValidUtf8(data)) {
        data.replace("\xC2\xA0", " ");
    }

    return Utility::ConvertLineEndings(GetCodecForHTML(data)->toUnicode(data));
}

I think this is the culprit. It is what is special casing the nbsp. We could remove this manual conversion and instead pass it through PreserveEntities here instead to always set the files on first input to have only the entities the user specified.

I think it was a holdover from an earlier time that we never saw since we used to always run mend on every file to do the universal updates which always ran things through PreserveEntities.

How do you want to handle this? If we add in PreserveEntities code here instead of manuall setting that one, at least the epub will present itself with the entities the user expects.

KevinH

Originally Posted by DiapDealer

Has anyone else been able to duplicate this issue? Is the no-break-space being special-cased here?

[KevinH]

Actually running it through PreserveEntities without gumbo being involved will not help. Anything that was an entity will stay an entity.

So perhaps we simply just delete this forced entity conversion and let the user decide when to run Mend to get only the entities they want in every file.

KevinH

[DNSB]

Originally Posted by KevinH

Actually running it through PreserveEntities without gumbo being involved will not help. Anything that was an entity will stay an entity.

So perhaps we simply just delete this forced entity conversion and let the user decide when to run Mend to get only the entities they want in every file.

KevinH

That gets my vote.

What? This isn't a democracy???

[DiapDealer]

Originally Posted by KevinH

So perhaps we simply just delete this forced entity conversion and let the user decide when to run Mend to get only the entities they want in every file.

Works for me, too.

[DiapDealer]

Originally Posted by DNSB

What? This isn't a democracy???

Sure it is. It's just a very small one!

[KevinH]

will do

[KevinH]

change pushed to master