Calibre 3 and rar on Debian Linux

[norbusan]

Thanks, that is an option. The other is generating man pages for the sub commands via help2man for calibredb <subcommand>, an approach that is taken by several applications. Or, combination of both ;-)

[kovidgoyal]

If you decide to do that, please do not hard code the list of sub-commands. Otherwise we will be having a similar discussion with somebody else in a few years.

[eschwartz]

Thanks for fixing the downstream manpages.

Note that you should probably be getting the command list the same way calibre's build process gets them. With that in mind, the following shell command prints out a newline-separated list of binaries calibre installs to /usr/bin:

cd /path/to/calibre-sourcecode/
printf "import sys, os; sys.path.insert(0, os.path.abspath('.')); import setup; setup.initialize_constants(); \nfor i in setup.basenames['gui'] + setup.basenames['console']: print(i);" | python2

Originally Posted by norbusan

Hi

thanks for your long email, and I am more than happy to work with you and the Calibre team for better packaging.

But mark one thing: I'm not the official maintainer of Calibre in Debian, but I am trying to contribute!

That's okay. I just don't have time or inclination to figure out what it takes to contribute packaging instructions for Debian, nor figure out the bugtracker.
I already do all that as a contributing *user* in Arch Linux, plus I don't really grok Debian much.

Can you be a bit more specific? The current packages do ship desktop files:

/usr/share/applications/calibre-gui.desktop
/usr/share/applications/ebook-viewer.desktop
/usr/share/applications/lrfviewer.desktop

and I don't see anything being removed after the call to install. If you can point out what is missing from the curretn calibre 3.1.1 packages, I will surely fix that!

Consider the differences in the filelists for the debian package calibre v3.1.1 and the Arch Linux package calibre v3.1.0.

Downloading and extracting calibre_3.1.1+dfsg-1_all.deb:data.tar.xz -- what do I see?

Those aren't the desktop files calibre installs, they have different content from the four desktop files shipped in calibre's codebase (or technically, the install target of calibre's python build system will generate them programmatically from the codebase, making use of e.g. an up-to-date list of gui_scripts and associated supported mimetypes):

/usr/share/applications/calibre-ebook-edit.desktop
/usr/share/applications/calibre-ebook-viewer.desktop
/usr/share/applications/calibre-gui.desktop
/usr/share/applications/calibre-lrfviewer.desktop

Debian doesn't even have a forked desktop file for the E-Book Editor at all, which means it won't show up in application menus, and uses downstream desktop files for the other GUI software.
A good policy would be to always rely on upstream desktop files, to save on duplication of effort (unless upstream supplies not desktop files, in which case oops).

Additionally, /usr/share/pixmaps/calibre.png is sourced from manual/resources/logo.png which is wrong, especially because calibre doesn't ship any pixmaps file at all, and instead ships a set of various icons (both application icons for the three distinct GUI applications, and for various ebook mimetypes) in various sizes, in /usr/share/icons/hicolor

Also missing from the debian package is /usr/share/mime/packages/calibre-mimetypes.xml (since Debian provides their own /usr/share/mime/packages/calibre.xml which contains unique content that calibre does not have, so I have no idea why it exists at all or where it came from).

That is a difficult point. Please put yourself for 1min into the position of a distributor with releases etc etc. Debian/stable (and for that matter *ANY* distribution) will have a certain version shipped in the stable release, with no updates. That is how Linux (and not linux, too) distributions work. You need to understand that an update notifier does not help at all.

Ha, do my Arch Linux biases show? If we don't have the latest release available (at a minimum in the testing/staging repos awaiting smoketests or some large soname rebuild) within days of upstream, we consider that something is wrong. There is *no* "stable" Arch Linux release, we are rolling-release.

And the update notifier in calibre is not AFAIK split out between the plugin update notifier and the calibre release update notifier, but either way there are patches in the Debian sources that seem to be trying to wholly get rid of the plugin indexer/installer/updater as well... I admit I haven't tested those patches but merely depended on my reading of the description/patched code.

I am also in part reacting to the perceived attitude of the downstream Debian person who filed a debian-specific, non-upstreamed security bug about the plugin system and decided the best thing was to silently disable things which to most people here are a large part of why calibre is so useful.
So, mea culpa.

Just to give you an example: I am the upstream devloper of TeX Live (4Gb of TeX, fonts, software) distribution, and I wrote our own update manager. At the sam etime I am packaging TeX Live for Debian. And there the update manager is not included, because *it*does*not*make*sense*. Users cannot update, root *should* not update as it breaks the whole Debian packaging infra.

Distributors and upstream have different requirements and targets. Calibre upstream has to accept this, too. It is better to work together instead of pointing fingers.

Sure, and if I was working on the assumption that only the calibre update notifier itself was being disabled, I wouldn't really care.

See the above. Instead of finger pointing, lets try to be civil and see what can be done. First of all, the update functionality is still here, nobody is blocked from using it (and I use it regularly), only the notifier is gone.

If the notifier is disabled and the buttons to access the Get Plugins feature are patched out of the codebase, I am not sure how you do that short of installing it via the ZIP file.

This is a general requirement in Debian that programs should *not* phone home if possible, unless started by the user. The user might be in an environment where certain things are prohibited, not welcome, and just automatically phoning home is not optimal.

Well, calibre does phone home with a unique installation ID, but I didn't argue against patching that out per se, checking for plugin updates seems a lot more reasonable to me.
I do see what you mean though.

So all in all, I don't see any serious problems here, and we can surely work out any differences. But packaging in many cases means compromising, and that means compromising for both sides.

Well, I am coming from the perspective of an Arch Linux user. One of our core beliefs is that upstream software should be built and distributed "as upstream intended", with minimal patching when necessary to ensure it actually builds, or when the patch exists in an as-yet-unpublished-release, etc...

Anyway, if you read down to hear, thanks for your long attention, and all the best

Norbert

All the best.

[norbusan]

Hi

thanks for all your detailed answers.

[QUOTE=eschwartz;3547545]the following shell command prints out a newline-separated list of binaries calibre installs to /usr/bin:[/QUODE]

thanks. BTW, the text for the web site/help, can it be also programmatically extracted so that I can generate man pages from it?

Originally Posted by eschwartz

Those aren't the desktop files calibre installs

Indeed, I checked and see that the installation does not work out, as mime type and desktop files are not installed.

The error is happening in the desktop-integration install process, and xdg-icon-resource breaks. I was suprised because --root (--staging-root) is passed to the install, and it wasn't forwarded to the desktop integration install code.

Reading through setupp/install.py I also see a comment that one needs to set XDG_DATA_DIRS, which I did, without any change.

I will try to fix this so that the correct desktop files are installed.

Originally Posted by eschwartz

/usr/share/pixmaps/calibre.png ... /usr/share/mime/packages/calibre-mimetypes.xml

Both of the above is again a consequence of the failed desktop integration installation. I will try to fix that and remove the locally used desktop/mime file.

Originally Posted by eschwartz

There is *no* "stable" Arch Linux release, we are rolling-release.

Well, so it be, diferent targets.

Originally Posted by eschwartz

If the notifier is disabled and the buttons to access the Get Plugins feature are patched out of the codebase, I am not sure how you do that short of installing it via the ZIP file.

Preferences -> Pluigns -> Get new plugins

is not patched out, works well! Using it.

Originally Posted by eschwartz

One of our core beliefs is that upstream software should be built and distributed "as upstream intended", with minimal patching when necessary to ensure it actually builds, or when the patch exists in an as-yet-unpublished-release, etc...

In principle this is the same in Debian ;-)

Thanks again for your suggestions, I will work on fixing as much as I can together with Martin.

All the best

Norbert

[norbusan]

Originally Posted by kovidgoyal

If you decide to do that, please do not hard code the list of sub-commands. Otherwise we will be having a similar discussion with somebody else in a few years.

Thanks, yes I want to generate the list of man pages or the man page automatically. If the content of th help is easily programmatically available that would be easy.

Thanks

Norbert

[norbusan]

Originally Posted by norbusan

Hi
Indeed, I checked and see that the installation does not work out, as mime type and desktop files are not installed.

The error is happening in the desktop-integration install process, and xdg-icon-resource breaks. I was suprised because --root (--staging-root) is passed to the install, and it wasn't forwarded to the desktop integration install code.

Reading through setupp/install.py I also see a comment that one needs to set XDG_DATA_DIRS, which I did, without any change.

I will try to fix this so that the correct desktop files are installed.

Indeed, there are many bugs. The problem is that xdg-mime and xdf-desktop-menu does not honor XDG_DATA_DIRS and hard code the location of gnome dirs to /usr/ something, which cannot be overriden. Thus the build fails.

I have fixed that now locally in my repo by changing the linux.py installation routine for debian to use 'install' instead of xdg-mime/xdg-desktop-menu for the debian builds.

Best

Norbert

[kovidgoyal]

The user manual sections on the command line interface are all programmatically generated. See manual/custom.py

You are welcome to submit a patch to refactor that code to make it easily useable for generating man pages as well, if you want.

Probably the best way to do that, is to run sphinx-build with the man output type and just take the man pages for the CLI aprt from there.

[kovidgoyal]

And with this commit

https://github.com/kovidgoyal/calibr...d8947eda602072

You can simply run

cd manual && sphinx-build2 -b man . /tmp/man

to generate the man pages in /tmp/man

[norbusan]

Thanks for the patch, but I can't get it to run, getting error

Exception occurred:
  File "conf.py", line 20, in <module>
    import init_calibre

[kovidgoyal]

You need to have calibre installed on the machine where you are running sphinx-build2. The calibre installation process installs init_calibre.py into your system python path.

[eschwartz]

Originally Posted by norbusan

Indeed, I checked and see that the installation does not work out, as mime type and desktop files are not installed.

The error is happening in the desktop-integration install process, and xdg-icon-resource breaks. I was suprised because --root (--staging-root) is passed to the install, and it wasn't forwarded to the desktop integration install code.

Reading through setupp/install.py I also see a comment that one needs to set XDG_DATA_DIRS, which I did, without any change.

I will try to fix this so that the correct desktop files are installed.

See the package() function I contributed to the Arch Linux PKGBUILD: https://git.archlinux.org/svntogit/c...ckages/calibre

XDG_DATA_DIRS= works fine, though you will first have to create some directories by hand because xdg-utils is a very buggy collection of tools.

Preferences -> Pluigns -> Get new plugins

is not patched out, works well! Using it.

This is... actually a very good point. I'm not entirely convinced it is reasonable to remove the more discoverable option away, and I don't understand the justification of removing one and not the other, but... so be it.

[eschwartz]

Originally Posted by norbusan

Thanks for the patch, but I can't get it to run, getting error

Exception occurred:
  File "conf.py", line 20, in <module>
    import init_calibre

It works fine for me, calibre should install a file /usr/lib/python2.7/site-packages/init_calibre.py which when using a distro calibre installation allows any python project to make use of the 'calibre' module after initializing it with 'import init_calibre' (this is necessary because calibre still relies on a forked version of html5lib).
I think Debian deletes that file though, and I am not sure why.

This is not really suitable for the actual package recipe at build time of course since it uses bits from the installed version of calibre rather than the source code in the build directory.

Kovid, when viewing the generated manpages I see garbage like:

You can also use \fBcalibre\-debug\fP to run standalone scripts. To do that use it like this:
.INDENT 0.0
.INDENT 3.5
\fBcalibre\-debug\fP myscript.py \-<U+200B>\- \fB\-\-option1\fP \fB\-\-option2\fP file1 file2 …

That "<U+200B>" is highly disruptive when viewed with man(1).

Also, I've never really tried formatting a manpage and I don't know much about Sphinx, but e.g. "ebook-convert input_file output_file [options]" should be dedented and be preceded by the "SYNOPSIS" header and followed by the "DESCRIPTION" header.
The ebook-convert page has:

Below are the options that are common to all conversion, followed by the
options specific to every input and output format.
.INDENT 0.0
.IP \(bu 2
\fI\%Look and Feel\fP
.IP \(bu 2
\fI\%Heuristic Processing\fP
.IP \(bu 2
\fI\%Search and Replace\fP
.IP \(bu 2
\fI\%Structure Detection\fP
.IP \(bu 2
\fI\%Table of Contents\fP
.IP \(bu 2
\fI\%Metadata\fP
.IP \(bu 2
\fI\%Debug\fP
.IP \(bu 2
\fI\%AZW4 Input Options\fP
.IP \(bu 2
\fI\%CHM Input Options\fP
.IP \(bu 2
\fI\%Comic Input Options\fP
.IP \(bu 2
\fI\%DJVU Input Options\fP
.IP \(bu 2
\fI\%DOCX Input Options\fP
.IP \(bu 2
\fI\%EPUB Input Options\fP
.IP \(bu 2
\fI\%FB2 Input Options\fP
.IP \(bu 2
\fI\%HTLZ Input Options\fP
.IP \(bu 2
\fI\%HTML Input Options\fP
.IP \(bu 2
\fI\%LIT Input Options\fP
.IP \(bu 2
\fI\%LRF Input Options\fP
.IP \(bu 2
\fI\%MOBI Input Options\fP
.IP \(bu 2
\fI\%ODT Input Options\fP
.IP \(bu 2
\fI\%PDB Input Options\fP
.IP \(bu 2
\fI\%PDF Input Options\fP
.IP \(bu 2
\fI\%PML Input Options\fP
.IP \(bu 2
\fI\%RB Input Options\fP
.IP \(bu 2
\fI\%RTF Input Options\fP
.IP \(bu 2
\fI\%Recipe Input Options\fP
.IP \(bu 2
\fI\%SNB Input Options\fP
.IP \(bu 2
\fI\%TCR Input Options\fP
.IP \(bu 2
\fI\%TXT Input Options\fP
.IP \(bu 2
\fI\%AZW3 Output Options\fP
.IP \(bu 2
\fI\%DOCX Output Options\fP
.IP \(bu 2
\fI\%EPUB Output Options\fP
.IP \(bu 2
\fI\%FB2 Output Options\fP
.IP \(bu 2
\fI\%HTML Output Options\fP
.IP \(bu 2
\fI\%HTMLZ Output Options\fP
.IP \(bu 2
\fI\%LIT Output Options\fP
.IP \(bu 2
\fI\%LRF Output Options\fP
.IP \(bu 2
\fI\%MOBI Output Options\fP
.IP \(bu 2
\fI\%OEB Output Options\fP
.IP \(bu 2
\fI\%PDB Output Options\fP
.IP \(bu 2
\fI\%PDF Output Options\fP
.IP \(bu 2
\fI\%PML Output Options\fP
.IP \(bu 2
\fI\%RB Output Options\fP
.IP \(bu 2
\fI\%RTF Output Options\fP
.IP \(bu 2
\fI\%SNB Output Options\fP
.IP \(bu 2
\fI\%TCR Output Options\fP
.IP \(bu 2
\fI\%TXT Output Options\fP
.IP \(bu 2
\fI\%TXTZ Output Options\fP
.UNINDENT
.INDENT 0.0

which is just noise when appearing in a manpage...
The OPTIONS header has a confusing set of [brackets] around it.
calibre's help system doesn't really have the information about what type of <parameters> each --flag accepts anyway, so the fact that the manpages don't have that either isn't really a problem.

Other than that, I'd use the manpages as-is. It's certainly useful right now!

(Now I would like a way to generate it as part of python2 setup.py resources, then install it during the install phase. Pretty please?)

[norbusan]

Originally Posted by eschwartz

I think Debian deletes that file though, and I am not sure why.

Indeed, thanks, how strange. Need to clean that up, too.

Originally Posted by eschwartz

(Now I would like a way to generate it as part of python2 setup.py resources, then install it during the install phase. Pretty please?)

Indeed indeed!!! That is what would be nice.

Thanks

Norbert

[kovidgoyal]

@Eli: I'm not going to add a dependency on sphinx to the calibre build process. I will create a separate command to build the man pages conveniently, but that's about it.

As for formatting of the man pages -- that's upto sphinx, I'm not going to spend time debugging it.

[norbusan]

Ok, after installing init_calibre.py, and moving it from the instal location in /usr/lib/pyton2.7/site-packages/ (which is not searched here) to one level up, I could run the sphinx-build command successfully.

This generated calibre.1, but not man pages for other, like calibredb. But the calibre.1 man page is great, thanks!