[GUI Plugin] epubveri library

[Kayadelenium]

epubveri library checks every EPUB in your calibre library, or just the books you select, and tells you which defects recur across it. It reads your books and reports; it changes nothing.

Why it exists. The epubveri plugin for Edit Book validates the book you have open, but it can only tell you about a book you already decided to open. In a large library the question you actually have is which ones, and nobody opens every book to find out. This plugin answers that question, from calibre's main window.

What you get. Not a list of broken books but a list of defects, each with the number of books carrying it, most widespread first. The shape of it, sketched (your library's rows and numbers will be its own):

MESSAGE   WHAT IT SAYS                              BOOKS   FINDINGS
RSC-005   element "..." is not allowed here           ...        ...
OPF-003   resource is not declared in the manifest    ...        ...
RSC-007   referenced resource is missing              ...        ...

That shape is the point. A library rarely has a long list of unrelated problems; it has a handful of producer defects repeated across many books. Understand one row and you have understood all the books under it.

From a row to the fix. Select a row and Show affected books filters calibre's book list to exactly the books carrying that defect. The report window stays open while you do it, so you can look at the books, come back and pick the next row. Open one of them in Edit Book and the epubveri editor plugin puts your cursor on the problem.

Did it get better? The last two whole-library scans are kept on disk, so the report survives a restart, and Compare with the previous scan shows each defect's book count before and after, largest change first. If the two scans were made by different epubveri versions the window says so, because a changed rule and a repaired book look the same in a comparison.

Also in the toolbar menu

• Validate the whole library or the selected books. Neither half of the button starts a scan on its own; you always choose which.
• Show the last report, without re-reading a single book.
• Show books with no EPUB: the books a scan counted but could not check, usually a conversion that never happened.
• Clear the marks it left, which removes this plugin's marks and nobody else's.
• Export as CSV, either the report as shown (one row per defect) or one row per book and defect, with how many times that book trips it.

Fast enough to be worth doing. The scan runs as a background job, so calibre stays usable, and cancelling still shows you what it found. It validates several books at once; under Preferences / Plugins / Customize you choose how many, and the default uses most of your processor's cores while leaving some free for calibre and the rest of the system. Every report records when it ran, how long it took, how many books at once, and the epubveri, plugin, calibre and OS versions, so a scan time posted here is one other people can compare against their own.

What is ranked. Errors and fatals always, warnings by default. Usage notes and epubveri's own advisory findings are collected but not ranked unless you tick the box in the report window; ticking it re-sorts the report and never re-scans the library. None of them changes a verdict.

Nothing is bundled, and nothing is trusted blindly. The plugin contains no binary. On first use it downloads the epubveri build for your platform from epubveri's own GitHub releases, verifies it against that release's published SHA256SUMS.txt before extracting anything, and checks it again before every scan. Automatic update checks can be switched off in the settings.

Installing. Download the zip attached to this post, then Preferences / Plugins / Load plugin from file, and restart calibre. If the Validate library button does not appear, add it under Preferences / Toolbars & menus. calibre 6.0 or later is required.

The two plugins, side by side

                    epubveri                  epubveri library
                    (Edit Book)               (main window)

checks              the book you have open    the whole library, or
                                              the books you select
reads               the book as the editor    the EPUB file as it sits
                    holds it, unsaved         on disk
                    changes included
answers             WHAT is wrong             WHICH books have a problem
shows               every finding, with       one row per defect, ordered
                    file, line and column     by how many books carry it
clicking a row      puts the cursor on it     filters the book list to
                                              the books that row affects

Same validator underneath, same findings, same message IDs. The Edit Book plugin has its own thread: https://www.mobileread.com/forums/sh...d.php?t=375293

Where to report what. Anything about this plugin (the scan, the report, the settings, installing) belongs in this thread or at https://github.com/veripublica/epubveri-plugins/issues. A finding you believe is wrong is a question about epubveri, and it belongs in the epubveri thread, https://www.mobileread.com/forums/sh...d.php?t=374286. A book on which epubveri and epubcheck disagree is the most useful thing you can send.

Links

• Releases and changelog: https://github.com/veripublica/epubv...ugins/releases (take the release tagged calibre-library-)
• Source, GPL-3: https://github.com/veripublica/epubveri-plugins
• epubveri itself: https://github.com/veripublica/epubveri

Plugin Download

[DNSB]

I tested on my Intake library with 77 books and on my Main library with ~18,000 books. The Intake library took ~38 seconds to report multiple errors and the second took ~3 hours to report 1 error.

The books in the Intake library have not finished my update metadata, correct errors, edit CSS workflow.

This is using Calibre 9.14 running from source with Windows 11 as the OS.

[BetterRed]

Moderator Notice

It's not an Editor plugin, it's a GUI Plugin - thread title changed.

BR

[Kayadelenium]

38 seconds for 77 books is a perfectly normal duration, but the software likely crashed with 18,000 books. I need to add logging for debugging.

Tomorrow is the first day of school, and I'll likely be at the kids' school. I'll look into the error this evening and try to release a new version.

[Kayadelenium]

Originally Posted by BetterRed

It's not an Editor plugin, it's a GUI Plugin - thread title changed.

BR

Actually, this isn't a GUI; it's a validation plugin. Would you consider creating a separate category for plugins like epub-data or epubcheck?

Neither "GUI" nor "editor" really fits our category perfectly.

Consider that I wrote this not just for the forum, but within Calibre's own application.

[BetterRed]

Originally Posted by Kayadelenium

Actually, this isn't a GUI; it's a validation plugin. Would you consider creating a separate category for plugins like epub-data or epubcheck?

Not my decision, take it up with Kovid Goyal.

BR

[Comfy.n]

Hi,

Currently the friendly name for GUI Plugin is "Extend calibre generally":



IMO, this convention fits this plugin, but a new category can be arranged, I guess.

I had sent you a PM about this earlier today.

[DNSB]

Originally Posted by Kayadelenium

38 seconds for 77 books is a perfectly normal duration, but the software likely crashed with 18,000 books. I need to add logging for debugging.

Tomorrow is the first day of school, and I'll likely be at the kids' school. I'll look into the error this evening and try to release a new version.

It didn't crash. It just took rather long to run. The one error reported was a book in which I has switched to using an SVG wrapper for a full page image and did not update the manifest.

Using the numbers from the Intake library compared to the Main library, the time is about right. The Main library has a higher percentage of omnibuses since I have a habit of converting a completed series into an omnibus volume.

Edit: Reran the epubveri library test while not running a collection of VMs running in the background and the time for the check dropped to 1hr 29 min. Having that mix of Windows and Linux VMs updating was not that big a CPU load but they keep the disk usage about 80%. Gave the computer something to do while I was watching the CFL Labour Day classic games.

[Kayadelenium]

epubveri library 0.1.1 — https://github.com/veripublica/epubv...library-v0.1.1

New — the logging I promised in #4. A scan now writes a progress line every 2.5% of the way through, carrying the position, the time so far and the seconds per book, so a run that stops for any reason says where it got to. Books that cannot be read are named as they happen, a cancelled scan says where it stopped, and the opening lines name the validator binary being used.

The log stays about forty lines whether the library holds a hundred books or twenty thousand. One line per book was the obvious version and the wrong one — DNSB, your 18 000 books would have produced 18 000 lines, written on the job thread inside the loop being timed. What shipped costs nothing measurable: 120 books took 14.32 s with it and 14.34 s without.

DNSB, on whether an hour and a half was normal — the last line of the log now gives the rate, so a run answers that about itself. And I measured the part that is mine: over 120 books this plugin's own overhead, everything that is not the validator, is 2 ms per book — 1.6% of the total. So the number your library reports is essentially a fact about your machine and your books, not about the plugin. 0.30 s/book on Windows against 0.125 s/book here is the platform, not the wrapper.

Also: the example report in the README used to print the real counts from the library I designed it against. Those read as a claim about EPUBs in general, which they are not — every library's rows are its own — so the columns stayed and the numbers became placeholders.

On the category question I raised in #5: Comfy.n's answer in #7 settles it. "Extend calibre generally" is what this does, and I will leave it there rather than take it further.

The zip and its checksums are on the release page. Install with Preferences → Plugins → Load plugin from file, then restart calibre.

[Kayadelenium]

epubveri library for calibre 0.2.0 — https://github.com/veripublica/epubv...library-v0.2.0

The scan uses more than one core now, and you choose how many. Until this release it validated one book at a time, whatever your machine had, which on a large library was the whole of the wait.

There is a new setting in Preferences → Plugins → Customize: Books validated at the same time. Its maximum is your processor's physical core count less two — two are kept for the rest of the system, calibre included — and it defaults to Automatic, which is that maximum, or the number of books if you have selected fewer.

How much faster it gets is a question about your machine, so I am not going to give you a number of mine to measure yourself against. Not every core is worth the same: a laptop that mixes fast and efficient cores gains noticeably less than its core count suggests. What I can tell you is that the limit is the processor rather than the disk — that was measured, after I had assumed the opposite — so a slow drive is not what holds a scan back, and putting the library on a faster one will not help.

If a scan is not getting quicker, or the machine becomes unpleasant to use while one runs, that setting is the number to lower.

DNSB — this is the release your 18 000-book run prompted. I would genuinely like to know what it does for you, and whether Automatic picked a sensible number on your machine; that is the one thing I cannot test from here.

Cancel still keeps what it has done, and now takes slightly longer. Pressing it stops new books starting immediately, but the ones already running are allowed to finish rather than being killed — work you have already paid for should not be thrown away. So expect a short pause before the report appears, roughly as long as the slowest book still in flight.

The report itself is identical whatever the setting: same books in the same order, same rules in the same order. That is asserted by tests rather than hoped for.

And it is filed as a GUI plugin now. It used to declare itself an Edit book tool so that it would sit next to its companion in calibre's plugin list — one product, one heading. BetterRed and Comfy.n settled that the other way in this thread, and they were right: it is a library-side plugin and belongs under Extend calibre generally. It now says so. The two plugins name each other in their descriptions instead, which is where that belonged anyway.

The zip and its checksums are on the release page.

[DNSB]

I left the cores at automatic and the total time for 17703 books dropped to 10 min 48 sec using 14 cores. A very decent improvement! I was going to try 22 cores but 14 is the maximum I could set since only 16 physical cores.

[Kayadelenium]

That is a massive reduction in time compared to the 1 hour 30 minutes. Did you encounter any problems? Is there anything else in this plugin that you find lacking, incorrect, or buggy, or anything you would like to improve?

Note: I have to reserve two cores for the system; otherwise, the system will render itself inoperable.

[DNSB]

I didn't see any issues. I compared the error output to the one from my older test run and no differences noted. No issues noted other than that system fans speeding up as the CPU use jumped to about 21%. An optional verbose CSV report with book title/errors would be nice but not a big deal.

[maddz]

OK, one thing - can you make the output pop-up persistent? When you click on a row and click show affected books, the pop-up disappears, and to get it back you need to re-run the plug-in which is annoying for a large library (it takes my Mac over 8 minutes to check 7912 books).

It needs to stay visible until you explicitly dismiss it, or re-run the plug-in, or close calibre.

[Kayadelenium]

epubveri library 0.4.0 — download

Two reports from this thread, and something they led to.

The report window stays open now. maddz — you were right that losing it to one click is expensive on a 7912-book library. It closed because it was a modal window, which forced it to: there is no point filtering the library view behind a window you cannot leave. It is modeless now, so you can click a row, look at the books, come back and click the next one. If you close it anyway, Show the last report in the toolbar menu brings it back without re-reading a book.

A per-book CSV. DNSB — Export… now offers two files: the report as you see it, one row per defect, or one row per book and defect, with the book's title and how many times that book trips that rule. That last column is why it was not a one-line change: a report row held a set of book ids, which can say a defect affects 181 books and cannot say one of them carries it forty times.

Then the part nobody asked for. Once a scan was worth keeping, it seemed worth keeping two. The last two whole-library scans of each library are saved, so:
- the report survives calibre closing, not just the window;
- Compare with the previous scan subtracts them — one row per rule, books before, books after, and the change.