A question about how plugins are distributed and updated

[Kayadelenium]

@KevinH, @Doitsu — a question rather than a proposal, and I'll say my interest up front so it isn't hiding anywhere: I write epubveri — an EPUB validator that runs as a single binary with no Java and reports epubcheck's message IDs — and Doitsu has written a Sigil plugin that wraps it. Anything that makes plugin distribution easier eventually helps me too. I'm not asking anyone to do work on my behalf.

What prompted it: Doitsu's plugin already keeps the epubveri binary it wraps up to date by itself — so the binary end of that chain has releases, tags and a version history, while the plugin doing the updating is a forum attachment with none of them. He built the mechanism and then had nowhere to point it at himself. That asymmetry is more my curiosity than anyone's problem, but I'd rather ask how it is meant to work than assume.

Kevin, if you have a minute:

• Has a GitHub-based index or distribution for plugins ever been considered? I noticed the official plugins (ReadiumReader, BibiReader, EpubJSReader) link to

  releases/latest

  while most third-party ones link to forum threads, so the two paths already exist side by side. If that split is deliberate, I'd like to understand the reasoning — it's probably a constraint I can't see from outside.
• Is there anything about the current arrangement that actually costs you time, or does it work well enough that it isn't worth changing? A 13-year-old index that you still maintain suggests it works; I don't want to solve a problem nobody has.

Doitsu: from the author's side, does keeping a plugin as a forum attachment cause you any friction — version history, bug reports, telling users which version they're on?

If it turns out something here is worth changing, I've thought about it a bit and I'm happy to help with the boring parts. But I'd rather hear the history first, since anything I'd suggest has probably already been discussed at some point.

[Doitsu]

Originally Posted by Kayadelenium

Doitsu: from the author's side, does keeping a plugin as a forum attachment cause you any friction — version history, bug reports, telling users which version they're on?

Both Calibre Editor and Sigil will display the plugin version number. Since most of my plugins are simple wrappers, I rarely get bug reports.

As long as you don't change the file names of new epubveri packages, the plugins should automatically download the latest versions from Github.

BTW, if you want to take over the epubveri plugin development, let me know and I'll ask the mods to delete my Sigil/Calibre epubveri plugin threads.

[KevinH]

The Sigil plugin list on MR is not automated. It links to the first post in a thread in our Sigl plugins subforum. Each plugin gets its own thread where the plugin developer can post releases, answer user questions etc. A few plugin devs do use github to host their plugin but still usually link it here in their own threads.

We found this approach was simple to maintain, the plugin zip files are typically small. And not high volume. The most issues I see come from Github based plugins where users confuse release zips with source zips, and zip copies of the repo source. This happens because most Sigil users are not github users. So creating issues for any plugin is not easy for them. They typically ask their questions here on MR in the plugin's own thread.

What types of things do you have in mind?

[Kayadelenium]

Thank you both — and Kevin, that's exactly the constraint I couldn't see from outside. You're right about it, and I've fixed the same hole on our own download page.

But I think I asked my question badly, because your answer settles one half of it and the half I actually care about is the other one.

Distribution to users and development are not the same problem. A forum thread is a fine way to hand someone a zip — one attachment, no ambiguity, and they can ask a question in the same place. I have no argument with that. What a forum thread cannot be is a record: there is no history to diff, no way to see what changed between two versions or why, and nothing a person arriving in three years can read to understand a decision made in 2019. That is the part I was asking about, and it survives your answer intact.

And there is a safety argument I'd make even if the record one didn't land. A Sigil plugin is code that runs on a user's machine, inside their editor, with access to their books. Today it reaches them as an attachment uploaded by an account, with no history, no review step, and no way to tell what changed since the version they had. I don't say that to cast doubt on anyone here — Doitsu's plugins are good and I've read one of them. I say it because "it has been fine so far" and "it is safe by construction" are different things, and the difference is exactly the one we insist on for ourselves: our releases are built by CI from a tagged commit, not uploaded by me, and the npm package carries a provenance attestation so anyone can check which commit produced it. We do not ship a release we have not tested. It would be strange to hold our own artefacts to that and not notice that the plugins wrapping them are held to nothing.

On the confusion problem, since you named it — I think it is a property of GitHub's release page rather than of GitHub, and it has answers that cost your users nothing. A single download page on sigil-ebook.com could name the one right file, and could pick it by the visitor's operating system without asking. Better still, Sigil could fetch from the index itself, so there is no page to visit and no wrong file to click. That last one solves your problem more completely than the forum does, and it is the reason I asked in the first place.

I'm offering that as thinking, not as a plan, and you know your users and I don't. If none of it is worth the trouble, that is a real answer and I'll take it.

Doitsu — on filenames, you have it in writing now: the eight release archives are named after the Rust target triple, they are documented, and they will not be renamed; a new target gets added alongside, never substituted. That's in

docs/INTEGRATING.md

rather than in a reply to you, on purpose — the calibre plugin and whoever comes next should read the same page instead of each getting a private assurance from me.

One thing on your side, as information rather than a request: resolving through the

releases

list takes its first element, which includes prereleases and drafts.

releases/latest

excludes both. We have never published a prerelease; if we ever did, the current code would pick it and then decline to treat it as an update, because the tag doesn't parse as three integers. It fails safe, but silently.

And thank you for the offer to hand over the plugin threads — genuinely, but no. Two honest reasons. It would make us slower: there is Sigil, there is calibre, and there is whatever comes next, and a plugin per host means learning every host while the validator stops moving. And your plugins work and you maintain them; users get a better result from the person who wrote them. What I can usefully do is keep the surface underneath stable and say so in advance, and I'd be glad to help on the plugin side when I have the time — with that caveat meant honestly.

[DiapDealer]

We make it pretty clear that people who install and use third party plugins with Sigil do so at their own risk. We couldn't really vet/review them if we wanted (and I don't want). There are countless plugins for Sigil on GitHub whose devs have never come here and never interacted with our online community.

I appreciate your interest, but I just don't have much interest in using my limited Sigil coding time to make some sort of web/sigil/github integrated system for users to find/review/download/install plugins. Or in providing a mechanism for users to see what has changed in new versions of 3rd-party plugins. We will likely continue to leave that up to the plugin devs to do (or not do).

[KevinH]

Agreed. No automated host download system is needed. Many plugins are not listed here but exist in private repos. What changed (dev-wise) is typically highlighted by each plugin dev in their latest MR plugin threads, normally in response to a bug report.

And neither DiapDealer nor I want to get into the 3rd party verification game. Since plugins are mainly pure python and typically short and involve no binaries, if I want to try a plugin from an untrusted source I unzip and read the code to see actually what it does.

But luckily, we have many long term contributors here in our MR forum that are now trusted sources for plugins and things. Our current plugin list is comprised mainly from those users who chose to help out and give back to our community, just like Sigil.

[Kayadelenium]

Thank you — all three of you. That answers it, and clearly.

I asked because I didn't know how the arrangement had come about, and now I do: a small community of long-term contributors, plugins short enough to read, and nothing sitting there waiting to be fixed. That's a good answer, and I'd much rather hear it than invent a problem nobody has.

One thing I'll take away for my own side: "unzip it and read the source" doesn't cover a plugin that downloads a compiled binary, which is exactly what mine does. That's my gap to close, not Sigil's.

And if a plugin — or anything around one — is ever useful to you, please ask me without hesitation. I'm glad to help whenever I can.

[DiapDealer]

Originally Posted by Kayadelenium

One thing I'll take away for my own side: "unzip it and read the source" doesn't cover a plugin that downloads a compiled binary, which is exactly what mine does. That's my gap to close, not Sigil's.

In fairness: anyone who can "unzip it and read the source" will be able to grasp that a binary will be downloaded upon first running the plugin (and from where that binary is being downloaded). That user can then decide for themselves whether or not they trust Doitsu (who has extensive history providing useful plugins) or your compiled release artifact (from where the plugin downloads it, and from where they can verify what the binary claims to do).

Others who might not be capable of grokking the plugin code are free to stop at the first line of Doitsu's plugin thread:

This plugin is a very simple epubveri wrapper that will automatically download the latest epubveri binary and check the current epub.

Everyone should be sufficiently leery of semi-anonymous code that downloads binaries to their machines. But we can't make them be so no matter what we might try (they download a Sigil binary on trust after all). So we leave it up to them to heed the warnings or not.