calibre-0.6.31, mechanize and lxml

[taurnil]

Having this odd problem compiling calibre. All the dependencies (latest versions) are in place. If I compile mechanize then lxml, the calibre make fails on this;

####### Building extension progress_indicator #######

*
* Running gui
*

Traceback (most recent call last):
  File "./setup.py", line 99, in <module>
    sys.exit(main())
  File "./setup.py", line 85, in main
    command.run_all(opts)
  File "/usr/src/calibre/setup/__init__.py", line 158, in run_all
    self.run_cmd(self, opts)
  File "/usr/src/calibre/setup/__init__.py", line 151, in run_cmd
    self.run_cmd(scmd, opts)
  File "/usr/src/calibre/setup/__init__.py", line 154, in run_cmd
    cmd.run(opts)
  File "/usr/src/calibre/setup/gui.py", line 36, in run
    self.build_forms()
  File "/usr/src/calibre/setup/gui.py", line 59, in build_forms
    from calibre.gui2 import build_forms
  File "/usr/src/calibre/src/calibre/gui2/__init__.py", line 15, in <module>
    from calibre.ebooks.metadata.meta import get_metadata, metadata_from_formats
  File "/usr/src/calibre/src/calibre/ebooks/metadata/meta.py", line 9, in <module>
    from calibre.ebooks.metadata.opf2 import OPF
  File "/usr/src/calibre/src/calibre/ebooks/metadata/opf2.py", line 14, in <module>
    from lxml import etree
ImportError: No module named lxml

So I go back and recompile lxml. Now when I compile calibre, the make fails on this;

####### Building extension progress_indicator #######

snip

g++ -c -pipe -fPIC -O2 -Wall -W -D_REENTRANT -DNDEBUG -DQT_NO_DEBUG -DQT_CORE_LIB -DQT_GUI_LIB -I. -I/usr/src/calibre/src/calibre/gui2/progress_indicator -I/usr/include/python2.6 -I/opt/lunar/qt/4/mkspecs/default -I/opt/lunar/qt/4/include/QtCore -I/opt/lunar/qt/4/include/QtGui -I/opt/lunar/qt/4/include -I/usr/X11R6/include -o sipprogress_indicatorQProgressIndicator.o sipprogress_indicatorQProgressIndicator.cpp
g++ /usr/src/calibre/build/pyqt/progress_indicator/qt/QProgressIndicator.o /usr/src/calibre/build/pyqt/progress_indicator/qt/moc_QProgressIndicator.o -shared -Wl,-O1 -Wl,-rpath,/opt/lunar/qt/4/lib -Wl,--version-script=progress_indicator.exp -o progress_indicator.so sipprogress_indicatorcmodule.o sipprogress_indicatorQProgressIndicator.o -L/opt/lunar/qt/4/lib -L/usr/X11R6/lib64 -lQtCore -L/usr/lib/mysql -L/usr/lib -L/usr/lib/postgresql -L/opt/lunar/qt/4/lib -lpthread -lQtGui -L/usr/lib/mysql -L/usr/lib -L/usr/lib/postgresql -L/opt/lunar/qt/4/lib -L/usr/X11R6/lib64 -lQtCore -L/usr/lib/mysql -L/usr/lib -L/usr/lib/postgresql -L/opt/lunar/qt/4/lib -lpthread -lXext -lX11 -lm -lpthread
Traceback (most recent call last):
  File "./setup.py", line 99, in <module>
    sys.exit(main())
  File "./setup.py", line 85, in main
    command.run_all(opts)
  File "/usr/src/calibre/setup/__init__.py", line 158, in run_all
    self.run_cmd(self, opts)
  File "/usr/src/calibre/setup/__init__.py", line 149, in run_cmd
    cmd.pre_sub_commands(opts)
  File "/usr/src/calibre/setup/install.py", line 128, in pre_sub_commands
    import calibre.utils.config as c
  File "/usr/src/calibre/src/calibre/__init__.py", line 20, in <module>
    import mechanize
ImportError: No module named mechanize

I do not understand what is happening here. Using Python-2.6.4 and lxml/mechanize are using the normal python build process.

[taurnil]

Found part of the problem and does not appear to be calibre related but a particular file that gets stomped on by lxml and mechanize. By default these two install the same file in the same place;

/usr/lib/python2.6/site-packages/easy-install.pth

So if I compile mechanize, it contains this;

import sys; sys.__plen = len(sys.path)
./mechanize-0.1.11-py2.6.egg
./ClientForm-0.2.10-py2.6.egg
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)

and then compiling lxml, it will contain this;

import sys; sys.__plen = len(sys.path)
./lxml-2.2.4-py2.6-linux-x86_64.egg
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)

So it is a chicken/egg thing with these and not a calibre issue and a matter of these two living together.

[kovidgoyal]

merge them manually.

[taurnil]

Originally Posted by kovidgoyal

merge them manually.

Yeah that's what I had to do. Also python-dateutil stomps on the same file and adding those worked fine.

[kovidgoyal]

That file is actually created by a package called setuptools, which is used by these packages. It sounds like your setuptools is broken.

[taurnil]

Originally Posted by kovidgoyal

That file is actually created by a package called setuptools, which is used by these packages. It sounds like your setuptools is broken.

That would seem to be the case, however after going through setuptools and the other python dependencies of calibre; I cannot find any reason why compiling those from source insists on overwriting easy-install.pth. Even watching the compile process it eventually says "adding blah blah" to easy-install.pth.

Installing them from source is pretty straight forward;

 python setup.py build && python setup.py install

I even tried running setuptools-0.6c11-py2.6.egg as a script file, as described in the documentation with the same results. So either there is some undocumented magic incantation for building/installing from source, or I have found a bug.