Calibre 3 and rar on Debian Linux

[norbusan]

Originally Posted by eschwartz

because xdg-utils is a very buggy collection of tools.

Indeed, they are a huge compilation of ad-hoc stuff that tries to be generic without working out. :-(

Thanks for the link to the arch build script, not so much different from Debian nowadays ;-)

Best

Norbert

[kovidgoyal]

No need to do that, in the next release the source tarball will contain the man pages for all the commands, in all the languages.

[norbusan]

Oh, great, thanks a lot!

[eschwartz]

Originally Posted by kovidgoyal

@Eli: I'm not going to add a dependency on sphinx to the calibre build process. I will create a separate command to build the man pages conveniently, but that's about it.

Seems reasonable. But if it subprocesses calibre-debug it still requires a separate calibre installation before it can run, and I think I do want to put a python2-sphinx makedepends in my AUR package (which needs to bootstrap everything). Arch usually adds in extra makedepends when it is needed for docs.

We can copy it across ourselves.

Also, I tried building it, and I don't think it likes working in parallel.
(Was this supposed to cache the doctrees so they aren't repeatedly generated?)

*
* Running man_pages
*

	Creating man pages in /home/eschwartz/git/calibre/man-pages...


**************** Building translations for: en
[Errno 2] No such file or directory
[eschwartz@vostro ~/git/calibre]$ python2 setup.py man_pages

*
* Running man_pages
*

	Creating man pages in /home/eschwartz/git/calibre/man-pages...


**************** Building translations for: en

Exception occurred:
  File "/usr/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 17] File exists: '/home/eschwartz/git/calibre/man-pages/doctrees'
The full traceback has been saved in /tmp/sphinx-err-McF2d_.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
sphinx-build2 -D language=en -b man -q -d /home/eschwartz/git/calibre/man-pages/doctrees /home/eschwartz/git/calibre/manual /home/eschwartz/git/calibre/man-pages/en
Traceback (most recent call last):
  File "/usr/bin/calibre-debug", line 20, in <module>
    sys.exit(main())
  File "/home/eschwartz/git/calibre/src/calibre/debug.py", line 320, in main
    run_script(args[1], args[2:])
  File "/home/eschwartz/git/calibre/src/calibre/debug.py", line 238, in run_script
    execfile(ef, g)
  File "/home/eschwartz/git/calibre/manual/build.py", line 108, in <module>
    build_man_pages(language, base)
  File "/home/eschwartz/git/calibre/manual/build.py", line 81, in build_man_pages
    sphinx_build(language, base, builder='man', bdir=language)
  File "/home/eschwartz/git/calibre/manual/build.py", line 36, in sphinx_build
    subprocess.check_call(ans)
  File "/usr/lib/python2.7/subprocess.py", line 186, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '[u'sphinx-build2', u'-D', u'language=en', u'-b', u'man', u'-q', u'-d', u'/home/eschwartz/git/calibre/man-pages/doctrees', u'/home/eschwartz/git/calibre/manual', u'/home/eschwartz/git/calibre/man-pages/en']' returned non-zero exit status 1

# Sphinx version: 1.6.3
# Python version: 2.7.13 (CPython)
# Docutils version: 0.13.1 release
# Jinja2 version: 2.9.6
# Last messages:

# Loaded extensions:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/sphinx/cmdline.py", line 305, in main
    opts.warningiserror, opts.tags, opts.verbosity, opts.jobs)
  File "/usr/lib/python2.7/site-packages/sphinx/application.py", line 226, in __init__
    self.builder = self.create_builder(buildername)
  File "/usr/lib/python2.7/site-packages/sphinx/application.py", line 306, in create_builder
    return self.registry.create_builder(self, name)
  File "/usr/lib/python2.7/site-packages/sphinx/registry.py", line 82, in create_builder
    return self.builders[name](app)
  File "/usr/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 83, in __init__
    os.makedirs(self.doctreedir)
  File "/usr/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 17] File exists: '/home/eschwartz/git/calibre/man-pages/doctrees'

As for formatting of the man pages -- that's upto sphinx, I'm not going to spend time debugging it.

Makes sense I suppose, thanks for fixing the U+200B issue though.

[kovidgoyal]

parallel builds work fine for me. Reduce the time it takes to build the man pages by a factor of three. And yes, it is supposed to share doctrees. If you cant get the parallel build working you can switch to calling sphinx-build2 with the -j option instead to let it handle parallelization

[kovidgoyal]

This should fix the aprallel build error: https://github.com/kovidgoyal/calibr...c35b071f7b95ea

[eschwartz]

Now I get:

*
* Running man_pages
*

	Creating man pages in /home/eschwartz/git/calibre/man-pages for 39 languages...
sphinx-build2 -D language=en -b man -q -Q -d /home/eschwartz/git/calibre/man-pages/doctrees /home/eschwartz/git/calibre/manual /home/eschwartz/git/calibre/man-pages/en


**************** Building translations for: el

Exception occurred:
  File "/usr/lib/python2.7/site-packages/docutils/nodes.py", line 2177, in dupname
    node['names'].remove(name)
ValueError: list.remove(x): x not in list
The full traceback has been saved in /tmp/sphinx-err-yj86QF.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
sphinx-build2 -D language=el -b man -q -Q -d /home/eschwartz/git/calibre/man-pages/doctrees /home/eschwartz/git/calibre/manual /home/eschwartz/git/calibre/man-pages/el
Traceback (most recent call last):
  File "/usr/bin/calibre-debug", line 20, in <module>
    sys.exit(main())
  File "/home/eschwartz/git/calibre/src/calibre/debug.py", line 320, in main
    run_script(args[1], args[2:])
  File "/home/eschwartz/git/calibre/src/calibre/debug.py", line 238, in run_script
    execfile(ef, g)
  File "/home/eschwartz/git/calibre/manual/build.py", line 111, in <module>
    build_man_pages(language, base)
  File "/home/eschwartz/git/calibre/manual/build.py", line 83, in build_man_pages
    sphinx_build(language, base, builder='man', bdir=language, very_quiet=True)
  File "/home/eschwartz/git/calibre/manual/build.py", line 38, in sphinx_build
    subprocess.check_call(ans)
  File "/usr/lib/python2.7/subprocess.py", line 186, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '[u'sphinx-build2', u'-D', u'language=el', u'-b', u'man', u'-q', u'-Q', u'-d', u'/home/eschwartz/git/calibre/man-pages/doctrees', u'/home/eschwartz/git/calibre/manual', u'/home/eschwartz/git/calibre/man-pages/el']' returned non-zero exit status 1

# Sphinx version: 1.6.3
# Python version: 2.7.13 (CPython)
# Docutils version: 0.13.1 release
# Jinja2 version: 2.9.6
# Last messages:
#   reading sources... [ 45%] generated/el/ebook-polish
#   reading sources... [ 47%] generated/el/ebook-viewer
#   reading sources... [ 50%] generated/el/fetch-ebook-metadata
#   reading sources... [ 52%] generated/el/lrf2lrs
#   reading sources... [ 54%] generated/el/lrfviewer
#   reading sources... [ 56%] generated/el/lrs2lrf
#   reading sources... [ 58%] generated/el/template_ref
#   reading sources... [ 60%] generated/el/web2disk
#   reading sources... [ 63%] glossary
#   reading sources... [ 65%] gui
# Loaded extensions:
#   sidebar_toc (unknown version) from /home/eschwartz/git/calibre/manual/sidebar_toc.pyc
#   sphinx.ext.viewcode (1.6.3) from /usr/lib/python2.7/site-packages/sphinx/ext/viewcode.pyc
#   custom (unknown version) from /home/eschwartz/git/calibre/manual/custom.pyc
#   sphinx.ext.autodoc (1.6.3) from /usr/lib/python2.7/site-packages/sphinx/ext/autodoc.pyc
#   alabaster (0.7.10) from /usr/lib/python2.7/site-packages/alabaster/__init__.pyc
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/sphinx/cmdline.py", line 306, in main
    app.build(opts.force_all, filenames)
  File "/usr/lib/python2.7/site-packages/sphinx/application.py", line 339, in build
    self.builder.build_update()
  File "/usr/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 326, in build_update
    self.build(['__all__'], to_build)
  File "/usr/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 344, in build
    updated_docnames = set(self.env.update(self.config, self.srcdir, self.doctreedir))
  File "/usr/lib/python2.7/site-packages/sphinx/environment/__init__.py", line 583, in update
    self._read_serial(docnames, self.app)
  File "/usr/lib/python2.7/site-packages/sphinx/environment/__init__.py", line 602, in _read_serial
    self.read_doc(docname, app)
  File "/usr/lib/python2.7/site-packages/sphinx/environment/__init__.py", line 723, in read_doc
    pub.publish()
  File "/usr/lib/python2.7/site-packages/docutils/core.py", line 218, in publish
    self.apply_transforms()
  File "/usr/lib/python2.7/site-packages/docutils/core.py", line 199, in apply_transforms
    self.document.transformer.apply_transforms()
  File "/usr/lib/python2.7/site-packages/docutils/transforms/__init__.py", line 171, in apply_transforms
    transform.apply(**kwargs)
  File "/usr/lib/python2.7/site-packages/sphinx/transforms/__init__.py", line 187, in apply
    self.document.note_implicit_target(node)
  File "/usr/lib/python2.7/site-packages/docutils/nodes.py", line 1383, in note_implicit_target
    self.set_name_id_map(target, id, msgnode, explicit=None)
  File "/usr/lib/python2.7/site-packages/docutils/nodes.py", line 1330, in set_name_id_map
    self.set_duplicate_name_id(node, id, name, msgnode, explicit)
  File "/usr/lib/python2.7/site-packages/docutils/nodes.py", line 1369, in set_duplicate_name_id
    dupname(node, name)
  File "/usr/lib/python2.7/site-packages/docutils/nodes.py", line 2177, in dupname
    node['names'].remove(name)
ValueError: list.remove(x): x not in list

And I have manpages for en, is, and vi.

...

Also, I would appreciate if instead of using 'import init_calibre' you could use the same logic the setup/*.py uses to get at calibre.constants calibre.linux calibre.utils.localization etc. whatever, and don't use calibre-debug for subprocesses. Currently this only works at all if you have already installed calibre to / which means no building manpages until the second time you build calibre.
It dies quite fast with:

*
* Running man_pages
*

	Creating man pages in /home/eschwartz/git/calibre/man-pages for 39 languages...
Traceback (most recent call last):
  File "setup.py", line 119, in <module>
    sys.exit(main())
  File "setup.py", line 104, in main
    command.run_all(opts)
  File "/home/eschwartz/git/calibre/setup/__init__.py", line 236, in run_all
    self.run_cmd(self, opts)
  File "/home/eschwartz/git/calibre/setup/__init__.py", line 232, in run_cmd
    cmd.run(opts)
  File "/home/eschwartz/git/calibre/setup/publish.py", line 284, in run
    self.build_man_pages(opts.man_dir or 'man-pages', opts.compress_man_pages)
  File "/home/eschwartz/git/calibre/setup/publish.py", line 307, in build_man_pages
    subprocess.check_call(jobs[0][0])
  File "/usr/lib/python2.7/subprocess.py", line 181, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 168, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib/python2.7/subprocess.py", line 390, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1024, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

[kovidgoyal]

sphinx is pretty fragile, you need to patch docutils to ignore that error. My docutils has:

def dupname(node, name):
    node['dupnames'].append(name)
    try:
        node['names'].remove(name)
    except ValueError:
        pass
    # Assume that this method is referenced, even though it isn't; we
    # don't want to throw unnecessary system_messages.
    node.referenced = 1

[eschwartz]

Uhhhh...

Is that an upstream bugreport anywhere (sphinx or docutils)? I don't think "convince everyone to patch their system packages" scales very well...
Or can that somehow be fixed in calibre's docs?

[eschwartz]

https://github.com/kovidgoyal/calibr...9ebeb581b883e6

Thanks. By the way, you missed an init_calibre in manual/custom.py

[norbusan]

Ah great, staged installs now work with building the man pages. Thanks.

[kovidgoyal]

Originally Posted by eschwartz

Uhhhh...

Is that an upstream bugreport anywhere (sphinx or docutils)? I don't think "convince everyone to patch their system packages" scales very well...
Or can that somehow be fixed in calibre's docs?

I dislike opening bug reports for bugs where I cannot track down the cause. This is most likely a bug in sphinx. But sphinx is such a mess that I really dont have the time/desire to debug it.

And no one needs to build the man pages, since they are included in the source tarball.

[eschwartz]

With the latest commits to fix building language=el manpages, I was able to complete a manpage build. But I wasn't able to duplicate it in a clean build, and it seems that the reason is because generate_docs() is not automatically called before setup_man_pages().

I think the reason this worked outside of my clean build, is because at one point in the past I had experimented with building the HTML manual and "manual/generated/${langs[@]}" existed as a result.

Spoiler:

# Sphinx version: 1.6.3
# Python version: 2.7.13 (CPython)
# Docutils version: 0.13.1 release
# Jinja2 version: 2.9.6
# Last messages:

# Loaded extensions:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/sphinx/cmdline.py", line 305, in main
    opts.warningiserror, opts.tags, opts.verbosity, opts.jobs)
  File "/usr/lib/python2.7/site-packages/sphinx/application.py", line 168, in __init__
    confoverrides or {}, self.tags)
  File "/usr/lib/python2.7/site-packages/sphinx/config.py", line 150, in __init__
    execfile_(filename, config)
  File "/usr/lib/python2.7/site-packages/sphinx/util/pycompat.py", line 150, in execfile_
    exec_(code, _globals)
  File "/usr/lib/python2.7/site-packages/six.py", line 699, in exec_
    exec("""exec _code_ in _globs_, _locs_""")
  File "<string>", line 1, in <module>
  File "conf.py", line 55, in <module>
    ge = {'generated/' + x for x in os.listdir('generated')} | {
OSError: [Errno 2] No such file or directory: 'generated'

By the way, thanks *a lot* for being so patient and fixing all these things I keep bringing up.

[norbusan]

Just to report back on this issue: I have updated the calibre package with man pages as generated from the build process, and installation of the desktop integrations files (.desktop, .xml, icons) as well.

If anyone wants to take a look: http://www.preining.info/debian/pool/main/c/calibre/ (please only the 3.3.0 packages)

Thanks for all you help and explanations.

[norbusan]

Dear all,
Calibre 3.4.0 is now in Debian with the mentioned changes, and I have been added to the development team. If you find more things to improve, please let me know.

My next plans are separate packages for the unrar stuff, unfortunately the rar maintainer is absolutely unresponsive :-(

Best

Norbert