[Plugin] epubveri - Simple epubveri wrapper

[Doitsu]

[Plugin] epubveri - Simple epubveri wrapper

Current Version: "0.0.6"

This plugin is a very simple epubveri wrapper that will automatically download the latest epubveri binary and check the current epub.

Preferences:

You can change the following epubveri.json preference settings:

  "usage": false,
  "advisory": false,
  "update_check": true,
  "check_interval": 7,

Change usage to true, to display USAGE message. Change advisory to true to display custom epubveri messages. Change update_check to false, if you want to manually update the epubveri binary file. check_interval is the number of days between update checks. (0 = always check for updates before running epubveri.)


Disclaimer:

Even though it's unlikely that the plugin will somehow corrupt the epub that you're working on or crash Sigil, you may want to save your ePub in Sigil before running this plugin and/or create a checkpoint.

Installation:

1. Select Manage Plugins from the Plugins menu, then select Use Bundled Python, if it isn't already selected.
2. Click Add Plugin and select epubveri_v0.0.6.zip. This will install the epuberi plugin, which you can select via Plugins > Validation > epuberi.

License: GNU General Public License v3 (GPL-3)

[KevinH]

Added to Sigil Plugin Index. Thank you!

[odamizu]

Thanks Doitsu! Does this mean you consider epubveri ready for production use? (The test plugin you sent me so super fast!)

[Doitsu]

Originally Posted by odamizu

Thanks Doitsu! Does this mean you consider epubveri ready for production use? (The test plugin you sent me so super fast!)

I still consider this a beta version, since it doesn't have preference settings yet. But it shouldn't contain any major bugs.

[JSWolf]

Originally Posted by Doitsu

I still consider this a beta version, since it doesn't have preference settings yet. But it shouldn't contain any major bugs.

And with the plugin, it's a lot easier to use it to check for errors and compare it's output with epubcheck.

I only use the CLI now when I want to post output from epubveri and epubcheck to help fixing errors.

[odamizu]

Originally Posted by Doitsu

I still consider this a beta version, since it doesn't have preference settings yet. But it shouldn't contain any major bugs.

Oh, I was referring to whether epubveri (program) was ready for production use, per our DMs, not whether your plugin was ready.

Anyway, I will start using it and compare with epubcheck. Thank you!

[Doitsu]

epubveri is getting very close to displaying the same number of errors and warnings as EPUBCheck. According to the official website:

Originally Posted by Kayadelenium

• 99.8% of the test suite's "this should be flagged" cases are correctly caught, with the exact same error code epubcheck itself would report (606 of 607).
• 100% of the test suite's "this is perfectly valid" cases are correctly left alone — no false alarms on any of the 355.

[odamizu]

That's great! Maybe I will switch to epubveri Its speed is so tempting.

Originally Posted by Doitsu

epubveri is getting very close to displaying the same number of errors and warnings as EPUBCheck ...

[Doitsu]

FYI: I've attached a new plugin version with preference settings. For example, you can now hide USAGE messages and select how often the plugin will check for updates.

[Kayadelenium]

I read both epubveri plugins line by line. Not because anyone reported a problem — because for most people the plugin is epubveri, and I had never looked. Most of what I found turned out to be our documentation's fault rather than yours, so let me start there.

Five things you had no way to know, because we never wrote them down. All five are now in INTEGRATING.md:

• -u decides what the JSON contains, not just what is shown. Both plugins pass it only when the user turns usage on — so with usage off, the summary box reports "0 usages", which is true about the output and false about the book. Passing it always and filtering in your own UI also means toggling the preference costs a repaint instead of a second validation run.
• The version is already in the envelope. tool_version is the second field. Running epubveri -V separately costs a process launch per validation and ties you to a human-readable line we have never promised.
• Prefer musl on Linux. Both plugins pick unknown-linux-gnu. The musl builds are static and run on any distribution; a gnu build on an older or non-glibc system fails to start with a message that explains nothing. The doc now carries the whole platform-to-asset table.
• data.advisory_basis. With --advisory on, a user sees NEXT-005 and ADV-001 side by side, and they make different claims: NEXT-* becomes an ordinary error the day epubcheck catches up, ADV-* never will. Every advisory finding carries spec-ahead or spec-silent so you can show them differently. We shipped the field and forgot to document it.
• location is a full container path. This plugin reduces it to a basename before basename_to_id(). Two files in one book can share a basename, so the cursor can land in the wrong file.

And one that is worth fixing first: exit code 2 still has a full report. We documented it as though it meant "nothing to read", which is wrong. There are three shapes:

• bad command line — stdout empty, message on stderr
• the file does not exist — full JSON on stdout, a PKG-018 fatal, stderr empty
• the input cannot be processed, e.g. a directory — full JSON, status "error", reason in inputs[].error, stderr empty

The plugin reports "Return code: {} epubveri {}".format(returncode, stderr), so in the two commonest cases the user sees Return code: 2 epubveri and nothing else, while epubveri had explained both in plain words. Parsing stdout first, and falling back to stderr only when there is no JSON to parse, is the whole fix.

Three small things in the plugin itself, found by reading rather than running:

• plugin.py line 582 — bk.add_result('error', path, None, ...) in the except branch for bk.readfile(). There is no path in scope anywhere in the module, so that line raises NameError: a handled error becomes a crash. issue.path looks like what was meant.
• No timeout on either network call. urlopen() and urlretrieve() both block indefinitely if a connection stalls. Your calibre plugin already gets this right with timeout=30 and 180 — it is the better of the two here.
• Version resolution reads /releases[0], which includes drafts and prereleases. The calibre one already filters those out.

One suggestion, with its limits stated. Since 0.12.4 every release ships SHA256SUMS.txt and a signed build attestation, and neither plugin checks either — so that work currently reaches nobody. Verifying the archive is about twenty lines and the doc now has the recipe. Be clear about what it buys, though: both files come from the same host over the same connection, so it is no protection against someone who controls GitHub. What it does catch is a truncated download, which otherwise installs as a binary that fails in a way neither of us could diagnose, and a release where one archive was replaced and the checksum file was not.

None of this is urgent and none of it came from a user. If any of it is easier as a patch than as a description, say so and I will write it.

[Doitsu]

Originally Posted by Kayadelenium

None of this is urgent and none of it came from a user. If any of it is easier as a patch than as a description, say so and I will write it.

Thanks for the thorough code review!


Since I might misinterpret some of your comments and I'm rather busy this week, I'd really appreciate it, if you updated plugin.py and main.py as you see fit and attached the updated files in this thread. I'll then replace both files in the plugin zip files and upload them to the Sigil and Calibre threads.

[JSWolf]

@Kayadelenium, are you going to do a code review of the calibre version of the plugin?

[Kayadelenium]

Doitsu — I spent today on this. Rather than patch plugin.py and main.py, I have written my own, from Sigil's own BSD-licensed plugin API. I could not have taken your code into it in any case: epubveri is dual-licensed, so its repositories cannot accept outside code until there is a contributor agreement, and there is not one yet.

Your plugins are why it exists. They proved the idea, and reading both is what taught me what an editor integration actually needs — most of what I found was my own documentation's fault rather than anything in your code.

What mine does, in case any of it is useful to you:

• Verifies the downloaded archive against the release's SHA256SUMS.txt before running it. epubveri has shipped that file since 0.12.4.
• Gives both network calls a timeout, so a hung connection cannot freeze the editor.
• Parses stdout first, whatever the exit code. Exit 2 has three shapes and only one of them answers on stderr: a bad command line. A missing file and an unreadable input both return a full JSON envelope with stderr empty — the first as a PKG-018 fatal, the second with status "error" and the reason in inputs[].error.
• Uses epubveri's location as the bookpath, which is a full container-relative path. Two folders can hold the same filename.
• Escapes everything that goes into Sigil's result XML, including the file path — as you already do for the message. Sigil's launcher builds that line by raw interpolation and escapes none of it.

Two things I would rather ask than decide on my own.

The name. Both are called epubveri, and Sigil keys plugins by name — so a user cannot have both installed, one replaces the other, and they even share the same preferences file under plugins_prefs/epubveri. That is your users as much as mine. I am happy to rename mine to keep yours where it is; tell me what you would prefer.

Settings in a Sigil plugin. I could not find a way to give one a preferences screen — Manage Plugins shows name, version, author, type, engine and platforms and nothing else, so the only routes I found were editing the JSON by hand or shipping a second plugin for one checkbox. I took neither and show everything instead, labelled. You have written more of these than anyone; is there a way I have missed?

It is at https://github.com/veripublica/epubveri-plugins, GPL-3, and unlike the epubveri repository it needs no contributor agreement — so anything in it is yours to take, and you are very welcome to send changes.

[Doitsu]

Originally Posted by Kayadelenium

Settings in a Sigil plugin. I could not find a way to give one a preferences screen — Manage Plugins shows name, version, author, type, engine and platforms and nothing else, so the only routes I found were editing the JSON by hand or shipping a second plugin for one checkbox. I took neither and show everything instead, labelled. You have written more of these than anyone; is there a way I have missed?

Thanks for rewriting the plugin from scratch! I quickly tested it and the only thing that you might want to change is change autostart to true.

<autostart>true</autostart>

As for preference settings, you could code you own preference dialog from scratch and have it displayed when the plugin is run for the first time. For an example, see the cssRemoveUnusedSelectors plugin.

IMHO, this would be overkill though, since most Sigil users are used to manually editing .json files, and preference dialogs are only useful for settings that will be changed a lot.

Since your plugin is superior to mine, I really think that you should create a new thread for it. I'll then ask KevinH or DiapDealer to delete mine.

Otherwise, I could simply remove my plugin.zip file and add a link to the official release page.

The same goes for the Calibre version.

[Kayadelenium]

Doitsu — thank you, and not only for the kind words. You tested it the same day, gave me a concrete fix, answered the settings question, and offered to stand down, all before I had asked for any of it.

autostart is in. I read PluginRunner.cpp before taking it and you are plainly right: without it the runner opens a dialog whose only content is a Start button, and a validation asks the user nothing, so that was a click for nothing. I left autoclose alone — line 553 already accepts and closes for any plugin of type validation.

And your answer on preferences settled it. I had already put the switch in the JSON and was uneasy about it; hearing from the person who has written the most of these that a dialog is overkill for a setting nobody changes twice is what I needed. The file now gets "autoupdate": true written into it on the first run, so at least it says that a choice exists.

First release is up: https://github.com/veripublica/epubv...g/sigil-v0.1.0

It is 0.1.0 rather than 1.0.0 on purpose. Two people have run it, on macOS, for one day, and nobody has tried it on Windows — where about three quarters of epubveri's downloads go. Three defects turned up today just from running it in Sigil that twenty-six unit tests had not caught, and that rate has not flattened yet.

Please do not have your thread deleted, and please do not take your plugin down yet — either of them. Mine is one day old, has been run by two people on one operating system, and nobody has tried it on Windows. If it turns out to be broken there, your plugin is what those users still have. Taking it down now would remove the fallback at exactly the moment it is most likely to be needed.

The thread itself I would keep whatever happens: it holds your work and the discussion in it, and deleting that takes away the record of where this came from.

When mine has been used for a while and has held up, I will say so here and you can decide then, with something to decide on. Until then the offer to rename mine stands, if two plugins with one name is awkward while both are installable.

On calibre: I have not started. What is in the repository is a skeleton — the half that talks to epubveri works, the half that talks to calibre has never been run inside it. I am going to try to look at it tomorrow. Yours is the only working one and should stay that way for now.