this epub has HTML files that are not well formed or are missing a doctype..

[DiapDealer]

Originally Posted by theducks

There are features of Sigil I like better and there are edit-ebook features I like. So I use both and do NOT want either +fixed+ .

No worries there.

It's not Kovid's duty to change things in Calibre just because Sigil handles things differently. Nor are we going to worry about accommodating every single thing that Calibre does differently.

Calibre (and the Modify Epub plugin is using calibre's own code to generate the Jacket) doesn't use (and will sometimes omit or remove) doctypes and xml declarations. Sigil uses (and will want to insert) doctypes and xml declarations.

Those who use both programs for editing epubs are simply going to have to get used to seeing Sigil's warning about missing doctypes.

There are some differences in the two programs that are simple and unobtrusive to accommodate, but doctypes (and doctype insertions) vs no doctypes (and doctype omission/removal) is not one of them.

[DNSB]

You will also see this with Modify Epub and insert/replace cover.

I just live with the error.

[Tex2002ans]

Originally Posted by JSWolf

I've figures out what the problem is. It's due to the adding of the Jacket. The header used for the Jacket is what's triggering the doctype requestor in Sigil. The existing headers are not being modified.

Nice debugging.

Originally Posted by DiapDealer

And since Calibre is involved, we have to make sure the epub is not being opened in Sigil via the third-party calibre OpenWith plugin. I long ago gave up debugging the on-again-off-again issues that arose when running Sigil via Kiwidude's OpenWith calibre plugin. Especially when calibre's own internal Open With feature has worked fine for years.

Heh... I thought I erased those horrible "Open With" memories out of my head!

I haven't seen you post about those in a while.

[BetterRed]

For years I've been assuming it was something to do with conversion from DOCX, I then use Modify to strip non-dc metadata and covers, and add a jacket (at the back) of EPUBs.

BR

[stumped]

Originally Posted by JSWolf

I've figures out what the problem is. It's due to the adding of the Jacket. The header used for the Jacket is what's triggering the doctype requestor in Sigil. The existing headers are not being modified.

Thanks.
I was about to updateby saying it happens with any random book in my library so no value in puloading specific books. Don't use epub3 , so it's an 3pub2 issue for me
I do use the plugin to deal with covers. I need proper covers or things go bad when covered and read as kepub.

So should I make a request for a fix somewhere and should that be in sigil or in the plug in

[DiapDealer]

Sigil is not going to stop warning about missing doctypes if that helps. Nor is it going to go back to silently making changes to epub code in the background.

[stumped]

Originally Posted by KevinH

A sample epub based on a public domain epub that shows the issue and follows the op's workflow (both before and after modify epub is run) would be quite helpful to see what is actually going on.

Jswolf seems to have solved it.

I looked for any easy way to log what changes were being made which were upsetting sigil but no joy. Could not find any helpful logs to upload.
It happens, on my modify settings, with any randomly chosen epub2 from my library so uploading one of those did not seem useful
The books render ok on all my devices even if I ignore the error report and just read them.

I am happy to log a proper update or fix request cnce advised whether it's sigil needs updating to ignore error or modify epub need updating to stop it upsetting sigil.

Meanwhile yes I can tweak my workflow. It's not understanding what's happening at code level that bugs me.

Usually, I am only tweaking an epub2 format so that it will navigate well when converted to kepub, so I am making sure there is exactly one header tag per file and that there is a cover in the expected place to keep kobo renderer and navigation happy

[stumped]

Originally Posted by DiapDealer

Sigil is not going to stop warning about missing doctypes if that helps. Nor is it going to go back to silently making changes to epub code in the background.

Fair enough.
Out of interest, could I have gotten sigil to tell me where in the book it was seeing the missing doctype? I did not see any way to force a detailed report
I looked at all the files in app data but none seemed to be detailed logs.

[KevinH]

Yes use Sigil's built in Checkpointing and Diffing features. Load the epub and tell it to make no changes. Make a Checkpoint. Reload the epub and allow it to auto fix things. Then compare it against the Checkpoint using the Compare/Diff command of Checkpointing.

To prevent this cycle you need to change your workflow or you will need to get the author of modify epub to add the proper doctype to both the jacket and cover it creates. Or just let Sigil auto fix the missing doctypes and be done with it.

[stumped]

Thanks again.

Maybe I should read an idiots guide to doc types and why they matter, or don't matter , in epub?

[Tex2002ans]

Originally Posted by stumped

Meanwhile yes I can tweak my workflow. It's not understanding what's happening at code level that bugs me.

If you open an XHTML (or HTML) file in Sigil, at the very top, you'll see something like this:

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
  "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">

That DOCTYPE is the code that is being added.

- - -

Note: In your very specific Modify EPUB example, there's a single file—the new cover file—that's missing this DOCTYPE.

- - -

What's That DOCTYPE Code Saying?

"Hey! I'm written in XHTML 1.1."

Programs can use this to determine what the heck is in the file, instead of trying to completely guess.

Reason 1: Rules

This then sets up some "rules" the program can follow. For example:

• In XHTML, everything that opens <p> needs to close </p>.
• In HTML, you can have super ugly stuff like <p> <p> <p> without ever closing it.

If the DOCTYPE is missing, the program will have to guess based on filenames... or just assume it's HTML.

- - -

Sigil, when opening and popping up that warning, is saying:

"Hey! This thing isn't 100% correct according to the specs. Add a DOCTYPE!"

Calibre is saying:

"Meh, it's just HTML or XHTML."

- - -

Reason 2: Named (or Numbered) Entities

"Named Entities" are stuff like:

• & = &amp;
• < = &lt;
• > = &gt;

"Numbered Entities" are stuff like:

• “ = &#8220;
• ” = &#8221;
• † = &#8224;

In EPUB2, both versions are allowed.

In EPUB3, only the numbers are allowed.

What Sigil/Calibre do, when fixing your files, is:

1. Convert all entities into their proper Unicode characters.
2. In EPUB3, convert to the numbered version if needed.

So everything changes into Unicode Characters:

• &quot; → "
• &ldquo; → “
• &rdquo; → ”
• &dagger; → †

And only a few change to numbered form. The most famous one is:

• &nbsp; → &#160; = Non-Breaking Space

That's pretty much the major code changes that Sigil/Calibre do when you say "Fix my files."

- - -

Note: If you want to read even more of the technical details on DOCTYPE, see the HTML specs:

• HTML5 > 8. The HTML syntax > 8.1.1 The DOCTYPE

- - -

Side Note: There's also been tons of topics over the years where we've discussed stuff like:

• Differences between HTML vs. XHTML
• Named vs. Numbered Entities

Just type into your favorite search engine:

whatever term you want Tex2002ans site:mobileread.com

and you'll probably stumble across all those topics over the years.

Originally Posted by stumped

Maybe I should read an idiots guide to doc types and why they matter, or don't matter , in epub?

Just push the button and trust Sigil. :P

It's like going from 99.999% correct to 100% correct.

KevinH (+ Sigil) says follow the specs.

Kovid (+ Calibre) says that .001% doesn't matter in reality.

(This is all going based off memory. It's buried somewhere in those previous DOCTYPE MobileRead topics.)

[JSWolf]

If ePub is not correct with the doctype, why doesn't epubcheck flag a missing doctype as an error?

[Tex2002ans]

Originally Posted by JSWolf

If ePub is not correct with the doctype, why doesn't epubcheck flag a missing doctype as an error?

I don't know. Look at the code. Here's the exact file on the epubcheck github:

• XMLParser.java (Relevant lines are 689 -> 768)

Spoiler:

    if (context.version == EPUBVersion.VERSION_2)
    {

      if (mimeType != null && "application/xhtml+xml".equals(mimeType) && root.equals("html"))
      {
        // OPS 2.0(.1)
        String complete = "<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.1//EN\" \n"
            + "\"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd\">";

        if (matchDoctypeId("-//W3C//DTD XHTML 1.1//EN", publicId, complete))
        {
          matchDoctypeId("http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd", systemId, complete);
        }

      }

      if (mimeType != null && "opf".equals(mimeType) && (publicId != null || systemId != null))
      {

        // 1.2: <!DOCTYPE package PUBLIC
        // "+//ISBN 0-9673008-1-9//DTD OEB 1.2 Package//EN"
        // "http://openebook.org/dtds/oeb-1.2/oebpkg12.dtd">
        // http://http://idpf.org/dtds/oeb-1.2/oebpkg12.dtd
        if ("package".equals(root)
            && (publicId == null || publicId
                .equals("+//ISBN 0-9673008-1-9//DTD OEB 1.2 Package//EN"))
            && (systemId == null || systemId
                .equals("http://openebook.org/dtds/oeb-1.2/oebpkg12.dtd")))
        {
          // for heritage content collections, dont warn about this, as its not
          // explicitly forbidden by the spec
        }
        else
        {
          report.message(MessageId.HTM_009, EPUBLocation.create(path));
        }

      }

      if (mimeType != null && "application/x-dtbncx+xml".equals(mimeType))
      {
        String complete = "<!DOCTYPE ncx PUBLIC \"-//NISO//DTD ncx 2005-1//EN\" "
            + "\n \"http://www.daisy.org/z3986/2005/ncx-2005-1.dtd\">";
        if (matchDoctypeId("-//NISO//DTD ncx 2005-1//EN", publicId, complete))
        {
          matchDoctypeId("http://www.daisy.org/z3986/2005/ncx-2005-1.dtd", systemId, complete);
        }
      }

    }
    else if (context.version == EPUBVersion.VERSION_3)
    {
      if (mimeType != null && "application/xhtml+xml".equals(mimeType)
          && "html".equalsIgnoreCase(root))
      {
        String complete = "<!DOCTYPE html>";
        // warn for obsolete or unknown doctypes
        if (publicId == null && (systemId == null || systemId.equals("about:legacy-compat")))
        {
          // we assume to have have <!DOCTYPE html> or <!DOCTYPE html SYSTEM
          // "about:legacy-compat">
        }
        else
        {
          report.message(MessageId.HTM_004, EPUBLocation.create(path), publicId, complete);
        }
      }
      else if (publicId != null || systemId != null)
      {
        report.message(MessageId.OPF_073, getLocation());
      }
    }

    firstStartDTDInvocation = false;
  }

You also have a few comments in there like:

// for heritage content collections, dont warn about this, as its not
          // explicitly forbidden by the spec

For EPUB3, it's looking for:

<!DOCTYPE html>

and there are also a few comments in there like:

        // warn for obsolete or unknown doctypes

Again, I'd reread that first DOCTYPE topic in the Sigil 1.4.0 changes where KevinH explained his exact technical reasoning. I'm betting that's where Kovid showed up and explained Calibre as well.

Sigil and Calibre create perfectly valid EPUBs, and have forever.

As I explained, it's like a 99.999% vs. 100% thing. No normal human will ever have to worry about this DOCTYPE stuff. Just press "Fix", and Sigil/Calibre will take care of it for you.

[DNSB]

Originally Posted by stumped

Thanks.
I was about to updateby saying it happens with any random book in my library so no value in puloading specific books. Don't use epub3 , so it's an 3pub2 issue for me
I do use the plugin to deal with covers. I need proper covers or things go bad when covered and read as kepub.

So should I make a request for a fix somewhere and should that be in sigil or in the plug in

You would need to ask for a change to the plugin. I think (not absolutely certain but moderately certain) that that plugin uses code built into calibre and there we run into philosophical differences where the creator of calibre does not regard DOCTYPE as a needed option (the built in Polish does the same thing).

At this point, I just live with the problem since it is not a showstopper and it is a lot faster to ModifyEpub or Polish to add the cover and then open in Sigil and mend to fix the issue.

[JSWolf]

This could be the code causing the problem for jackets. It's at the end of jacket.py.

def is_legacy_jacket(html):
    nodes = html.xpath('//x:h1[starts-with(@class,"calibrerescale")]',
            namespaces={'x':'http://www.w3.org/1999/xhtml'})
    if not nodes:
        nodes = html.xpath('//x:h2[starts-with(@class,"calibrerescale")]',
            namespaces={'x':'http://www.w3.org/1999/xhtml'})
    return len(nodes) > 0

def is_current_jacket(html):
    nodes = html.xpath('//x:meta[@name="calibre-content" and @content="jacket"]',
            namespaces={'x':'http://www.w3.org/1999/xhtml'})
    return len(nodes) > 0