Calibre 3 and rar on Debian Linux

[norbusan]

Hi all,
something seems to have changed wrt to rar in recent versions 3. The rar comics I have don't open anymore, and the error message says at the end:

ImportError: No module named unrardll

And indeed, there is no unrardll on Linux.

Is this intended, is there a known work-around for that?

Thanks and all the best

Norbert

[kovidgoyal]

Install the official calibre binary which comes with unrardll, instead of using a distro package. And unrardll is most definitely available on linux, your distro package simply forgot to declare a dependency on it. https://github.com/kovidgoyal/unrardll

[eschwartz]

IIRC you are a Debian user, and Debian normally patches out calibre's rar capability because "it isn't free enough". So perhaps they neglected to update their patch to calibre v3.

[norbusan]

Thanks for both replies. Indeed, unrar is in Debian, but there is no .so shipped, only the binary. Thus, it cannot be found.

Concerning the Calibre 3 packages, no rar code is removed, that was only done in the Calibre 2 packages and I have built and published packages for Calibre 2 without stripping rar support for quite some time.

I will contact the maintainer to update the packaging to get also a library version.

Thanks

Norbert

[eschwartz]

https://anonscm.debian.org/cgit/coll...d6d44a91dd12a8

Yeah, they just dropped the code to patch out the unrar plugin, but failed to replace it with a dependency... seems like an odd oversight. But I certainly can't tell from the commit message whether that means Debian intends thinks it's okay to ship a python2-unrardll package.

[norbusan]

Sorry, you seem to miss a few points about Debian packaging: First, there *cannot* be a dependency from Calibre onto unrar in any way, since unrar is not DFSG free. The unrar package is in the nonfree section.

And then, even if there is a dependency on unrar, this package (and no! package) ships the shared library necessary (libunrar.so).

There is already a bug report to unrar requesting inclusion of the shared library, but it has not been fixed by now, meaning that there is *no* libunrar.so available via standard Debian installation methods, including nonfree.

In absence of any reaction from the maintainer I will build a deb package of *only* libunrar.so and ship it in my calibre apt repository for those who want to use it.

All the best

Norbert

[eschwartz]

1) libunrar.so is not sufficient, that is the library version of unrar but you actually need the python binary extension python2-unrardll ("/usr/lib/python2.7/site-packages/unrardll/unrar.so") which will in turn link to libunrar.so because shared libraries.

2) I guess Debian could put both packages (libunrar and python2-unrardll) in the nonfree section, but calibre would still need to depend on it (python2-unrardll), because if it isn't installed, then you will still get that error message, which is unfriendly to users.

If Debian wishes to both handle the DFSG (no comment) and allow people to install from the nonfree section to get RAR comic handling in calibre, they will need to provide their own downstream patch to supply graceful error messages, and hopefully add an optional dependency on python2-unrardll. (Sadly, your comment "in any way" seems to imply that Debian would expect their users to simply learn through the grapevine that python2-unrardll has anything to do with calibre.)
calibre itself will of course not support that error message itself, since downstream packagers are not "supposed" to strip out parts of calibre functionality just because of their political differences.

[norbusan]

Indeed, both are needed, that is true.

I have made unrar packages that include the libunrar.so(.5) and sent the patches to the maintainer. Next I will package python2-unrardll and will upload it to the non-free archive.

With the maintainer of Calibre we will discuss how to handle the errors in case python2-unrardll is not installed.

Discussions about DFSG are useless - I am at many times not happy either, but that doesn't help ;-)

Thanks

[norbusan]

Another comment why all this came up - afais there is *nothing* mentioned in the calibre source code that python2-unrardll is needed, useful, or whatever.

$ grep -i unrardll `find . -type f`
./src/calibre/utils/unrar.py:    from unrardll import extract
./src/calibre/utils/unrar.py:    from unrardll import names
./src/calibre/utils/unrar.py:    from unrardll import comment
./src/calibre/utils/unrar.py:    from unrardll import extract_member

I think it would make sense to document this dependency somewhere ...

[kovidgoyal]

Umm all calibre dependencies are documented on the linux download page. Not only are they documented, they are documented in a machine readable format, i.e. a json file, that is stored under version control so it is trivial to see how the dependencies change with time.

[norbusan]

Indeed, there it is https://github.com/kovidgoyal/build-...s/sources.json, never saw that before, thanks. I just looked into the actual sources, my fault.

[eschwartz]

Originally Posted by norbusan

Indeed, both are needed, that is true.

I have made unrar packages that include the libunrar.so(.5) and sent the patches to the maintainer. Next I will package python2-unrardll and will upload it to the non-free archive.

With the maintainer of Calibre we will discuss how to handle the errors in case python2-unrardll is not installed.

Discussions about DFSG are useless - I am at many times not happy either, but that doesn't help ;-)

Thanks

Don't see any bugreports in my admittedly inexpert search of the Debian bugtracker.

Note also that while you are fixing things, since you seem to be somewhat invested in Debian packaging for calibre ...

It would be nice if Debian would stop shipping downstream manpages for binaries that no longer exist at all, and start shipping the desktop files, icons and other XDG stuff that calibre does try to install. The fact that Debian has essentially forked calibre's system integration entirely with a vastly inferior downstream version is... well, I would say troubling except I don't actually have to deal with it on a personal level.
It also causes occasional bugreports both for Debian/Ubuntu and occasionally mis-aimed at calibre itself, when people want to know why there are problems with the desktop files/icons.

Also if you could bug someone to re-enable the update notifier, that would be much obliged. There is a checkbox to disable the notifications if someone doesn't want to get told when Debian is running behind upstream releases, and more importantly, it also handles the plugin notifications. Which were erroneously patched out based on the absolutely shoddy logic that "It uses a totally non-authenticated and non-trusted way of installing arbitrary code."
calibre plugins are curated by the moderators here before being linked to the plugin index, so the only possible danger ever was a man-in-the-middle attack between MobileRead --> calibre-ebook.com --> user. I suppose some Debian busybodies believe anything that wasn't downloaded directly from the Debian repositories is somehow horribly dangerous or at least highly suspicious.
And calibre-ebook.com has served the plugin downloads via HTTPS for a long time, so any MiTM injecting malicious code while Kovid was running the scraper for updated plugins, would be noticed if all calibre users were getting malware.
And MobileRead itself is now also HTTPS-enabled so that isn't a problem either.

See this bug -- and from the way the reporter described it, I am not at all sure he even uses the software, because he certainly didn't check to see how the feature operates. He might be shocked to learn that the question_dialog does indeed require user interaction, and appears whenever a plugin is installed whether via the plugin index or manually loaded from a file.
It is of course patently obvious that he never bothered asking upstream whether any security checks were done, or requesting such -- just reflexively disabling things willy-nilly.

[norbusan]

Hi

thanks for your long email, and I am more than happy to work with you and the Calibre team for better packaging.

But mark one thing: I'm not the official maintainer of Calibre in Debian, but I am trying to contribute!

Originally Posted by eschwartz

Don't see any bugreports in my admittedly inexpert search of the Debian bugtracker.

Here it is https://bugs.debian.org/cgi-bin/bugr...cgi?bug=720051

Originally Posted by eschwartz

It would be nice if Debian would stop shipping downstream manpages for binaries that no longer exist at all

Good point, I will look into it. Just one recommendation: If you have complains, a bug report to the BTS always helps. Maintainers often forget about things and need a reminder. Anyway, I put it on my TODO list for calibre.

Originally Posted by eschwartz

start shipping the desktop files, icons and other XDG stuff that calibre does try to install.

Can you be a bit more specific? The current packages do ship desktop files:

/usr/share/applications/calibre-gui.desktop
/usr/share/applications/ebook-viewer.desktop
/usr/share/applications/lrfviewer.desktop

and I don't see anything being removed after the call to install. If you can point out what is missing from the curretn calibre 3.1.1 packages, I will surely fix that!

Originally Posted by eschwartz

The fact that Debian has essentially forked calibre's system integration entirely with a vastly inferior downstream version is... well, I would say troubling except I don't actually have to deal with it on a personal level.

Can we please try to remain on a normal level without insults. Packaging isnot trivial, and it works best if *both* sides work together without insulting each other. Thanks.

As I said, I don't see what you are pointing at, esp wrt to system integration. If you have detailed complains, not general insults, I am happy to look into them.

Originally Posted by eschwartz

Also if you could bug someone to re-enable the update notifier, that would be much obliged.

That is a difficult point. Please put yourself for 1min into the position of a distributor with releases etc etc. Debian/stable (and for that matter *ANY* distribution) will have a certain version shipped in the stable release, with no updates. That is how Linux (and not linux, too) distributions work. You need to understand that an update notifier does not help at all.

Just to give you an example: I am the upstream devloper of TeX Live (4Gb of TeX, fonts, software) distribution, and I wrote our own update manager. At the sam etime I am packaging TeX Live for Debian. And there the update manager is not included, because *it*does*not*make*sense*. Users cannot update, root *should* not update as it breaks the whole Debian packaging infra.

Distributors and upstream have different requirements and targets. Calibre upstream has to accept this, too. It is better to work together instead of pointing fingers.

Originally Posted by eschwartz

absolutely shoddy logic that "It uses a totally non-authenticated and non-trusted way of installing arbitrary code."
calibre plugins are curated by the moderators here before being linked to the plugin index, so the only possible danger ever was a man-in-the-middle attack between MobileRead --> calibre-ebook.com --> user.

See the above. Instead of finger pointing, lets try to be civil and see what can be done. First of all, the update functionality is still here, nobody is blocked from using it (and I use it regularly), only the notifier is gone.

This is a general requirement in Debian that programs should *not* phone home if possible, unless started by the user. The user might be in an environment where certain things are prohibited, not welcome, and just automatically phoning home is not optimal.

Again, as with the above, distributors and upstream have different requirements. To go back to my example of TeX Live: I often thought about phoning home from the TeX Live manager, it could give us valuable information, but refrained from doing that, first because I don't want that anything happens on the user computer without explicit action (and second because Karl Berry would object ;-) )


So all in all, I don't see any serious problems here, and we can surely work out any differences. But packaging in many cases means compromising, and that means compromising for both sides.

Anyway, if you read down to hear, thanks for your long attention, and all the best

Norbert

[norbusan]

First thing, I have reworked the man pages and removed outdated, and generated man pages for the shipped programs that have a --help option via help2man.

See https://git.preining.info/calibre.git/

Sent this to the maintainer.

Thanks

[kovidgoyal]

help2man will only work for the simpler commands. For the more complex commands, it will not generate useful help.

So for at least calibredb and ebook-convert you should instead have the man pages point to the calibre user manual.

https://manual.calibre-ebook.com/gen...calibredb.html
https://manual.calibre-ebook.com/gen...k-convert.html