Building Calibre on OSX Lion?

[fab672000]

Hi, I read the calibre documentation, has setup a bzr branch of calibre, has set the CALIBRE_DEVELOP_FROM env var to point the source as described in the documentation & checked the env variables are defined. I also installed QT4.7 and checked that my python libs are in the python path (2.6).

I then tried to run python setup.py build (resp. install)
even in sudo modes, all would fail.
I could build separately the c related code libs.

Does the build work in Lion / SL 64 bit mode ?
TIA,
Fab

Below the trace:
==========

*
* Running build
*


####### Building extension speedup #######
Linking speedup


gcc /Users/fab/devl/calibre/calibre/build/objects/speedup/speedup.o -o /Users/fab/devl/calibre/calibre/src/calibre/plugins/speedup.so -Wall -arch i386 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.5.sdk -bundle -undefined dynamic_lookup


ld: library not found for -lbundle1.o
collect2: ld returned 1 exit status
ld: library not found for -lbundle1.o
collect2: ld returned 1 exit status
lipo: can't open input file: /var/folders/g_/prj7fpf55zqfsncg1xkl_cj00000gp/T//cc8qHnDD.out (No such file or directory)
Error while executing: gcc /Users/fab/devl/calibre/calibre/build/objects/speedup/speedup.o -o /Users/fab/devl/calibre/calibre/src/calibre/plugins/speedup.so -Wall -arch i386 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.5.sdk -bundle -undefined dynamic_lookup

Traceback (most recent call last):
File "setup.py", line 99, in <module>
sys.exit(main())
File "setup.py", line 85, in main
command.run_all(opts)
File "/Users/fab/devl/calibre/calibre/setup/__init__.py", line 163, in run_all
self.run_cmd(self, opts)
File "/Users/fab/devl/calibre/calibre/setup/__init__.py", line 159, in run_cmd
cmd.run(opts)
File "/Users/fab/devl/calibre/calibre/setup/extensions.py", line 298, in run
self.build(ext, dest)
File "/Users/fab/devl/calibre/calibre/setup/extensions.py", line 353, in build
self.check_call(cmd)
File "/Users/fab/devl/calibre/calibre/setup/extensions.py", line 373, in check_call
subprocess.check_call(*args, **kwargs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 504, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['gcc', '/Users/fab/devl/calibre/calibre/build/objects/speedup/speedup.o', '-o', '/Users/fab/devl/calibre/calibre/src/calibre/plugins/speedup.so', '-Wall', '-arch', 'i386', '-arch', 'x86_64', '-isysroot', '/Developer/SDKs/MacOSX10.5.sdk', '-bundle', '-undefined', 'dynamic_lookup']' returned non-zero exit status 1
[f

[DoctorOhh]

Originally Posted by fab672000

Hi, I read the calibre documentation, has setup a bzr branch of calibre, has set the CALIBRE_DEVELOP_FROM env var to point the source as described in the documentation & checked the env variables are defined.

I'm not a Mac user, neither do I run from source, but I suspect you are doing a lot of extra work to create the development environment. I don't believe "building" calibre is part of the process.

Are you following these steps?

Good Luck, I'm sure another developer will push you in the correct direction.

[kovidgoyal]

All you need to do is check out the source and set CALIBRE_DEVELOP_FROM there is no need to build anything.

[fab672000]

I thought that there would be a build command to generate an app wrapping the python sources under mac os x ? How do you guys proceed to generate a package ?

Also, I simply tried to run calibre from sources by typing from the command prompt:
~/devl/calibre/src/calibre $ python startup.py
Traceback (most recent call last):
File "startup.py", line 19, in <module>
from calibre.constants import iswindows, preferred_encoding, plugins, isosx
ImportError: No module named calibre.constants

TIA,
Fabien

[ldolse]

No one needs to bother building a package on OS X except Kovid, or maybe someone who needs to modify one of Calibre's very few C++ components.

The typical Calibre dev is just working on the python code, and in that case using the CALIBRE_DEVELOP_FROM variable is with calibre-debug is all that's required (follow the steps in Dwanthny's link) - that tells Calibre to use your source directory instead of it's bundled python code.

The only other valid reason I could think of for trying to get the whole build system working is if you were trying to fork Calibre. Maybe if you explain to us why you feel it necessary to build the whole package we could provide some better direction.

[KevinH]

Hi,

Originally Posted by ldolse

The only other valid reason I could think of for trying to get the whole build system working is if you were trying to fork Calibre. Maybe if you explain to us why you feel it necessary to build the whole package we could provide some better direction.

Having volunteered on many open source projects over the years, I find this response very troubling.

The whole point of GPL is that the source is available to build and volunteers are encouraged to build their own ...

If this is a GPL project then please provide the instructions necessary to build distribution packages for all platforms that you release on.

Yes, most changes are simply changes to the python code and that is fine. But there are now binary C/C++ files in this build are their not?

Yes, they can be messy and yes they can require lots of extra dependencies, but some sort of build instructions, scripts, or make files should be made available since they have to exist.

Why all of the secrecy on how to build release dmgs, etc?

KevinH

[ldolse]

I agree with you conceptually, but on the flip side Calibre's approach in this case actually encourages development and contribution more than any other open source projects I've worked on. The fact that you don't need to build a bunch of dependencies makes Calibre incredibly easy to contribute too.

I don't think it's a matter of secrecy that building the package from scratch is difficult, though Kovid may be better able to comment on that - I think it's just more about convenience and priorities.

I think it may also be partly Apple's fault - most open source projects completely fail at building easy to use Apple installer packages even when there are Windows and Linux binaries, and I think it's at least somewhat common that the source tree doesn't provide the capability to build a DMG package, but I may be wrong.

BTW, last time I built Calibre from source was on Linux - and that was very easy - the other issue with Mac is you often need a homebrew/Macports/fink recipe for each dependency in order to get a successful build - that can be incredibly tedious to put all those pieces together.

[fab672000]

Well, for having humbly contributed to (and also authored) few GPl'd projects in the past, I find that a bit singular too but that's ok and again thank you guys for making calibre real and as nice just the way it is.

So all I wanted (as I'm used too with open source projects) is the capability to build an app that can differ from main branch just because sometimes I'd like something different or simply something just right now, even if it is not as polished.

That said, i think I can indeed then use the approach of running from sources if that is the only way to go.

I'm not even familiar yet with python but has a quite decent experience in c,c++,java,C# and recently ObjC languages, so I guess my contributions would not be terrific yet anyway, but could be progressively more interesting.
I'm not even planning to contribute to new features for now, only a humble perspective of tweaking the app or finding simple bug fixes, or simply sometimes just getting curious

[ldolse]

I think for what you want working from source the way the rest of us do is more than sufficient. The compiled code/package only changes about once a year if that - the rest of the time the python code is all that changes. The primary practical reason to tinker in that area is if there was a third party C library that you wanted to integrate into Calibre or modify. And if you needed to do that you'd absolutely get help, but if it's unnecessary to go to that level then I'd recommend keeping things simple.

[DoctorOhh]

Originally Posted by KevinH

Having volunteered on many open source projects over the years, I find this response very troubling.

I can see the merits in your position and I'll admit my ignorance in this area but it seems to me you are wrong.

Originally Posted by KevinH

The whole point of GPL is that the source is available to build and volunteers are encouraged to build their own ...

For anyone who wishes the source is available to build. Volunteers to this project are encouraged to contribute via the development environment outlined in the manual.

Originally Posted by KevinH

If this is a GPL project then please provide the instructions necessary to build distribution packages for all platforms that you release on.

How does this have any bearing on whether this is a GPL project? The source is there to contribute to the project or build as you wish. There is no requirement in the GPL that folks have their hand held through building

Originally Posted by KevinH

Why all of the secrecy on how to build release dmgs, etc?

No secrecy exists here, the source is well documented and other groups have used the basic project in endeavors of there own without hand holding or knowing any secret handshake or secret code.

Originally Posted by fab672000

I'm not even familiar yet with python but has a quite decent experience in c,c++,java,C# and recently ObjC languages, so I guess my contributions would not be terrific yet anyway, but could be progressively more interesting.
I'm not even planning to contribute to new features for now, only a humble perspective of tweaking the app or finding simple bug fixes, or simply sometimes just getting curious

I look forward to seeing and testing your future contributions.

[fab672000]

Originally Posted by dwanthny

I'm not a Mac user, neither do I run from source, but I suspect you are doing a lot of extra work to create the development environment. I don't believe "building" calibre is part of the process.

Are you following these steps?

Good Luck, I'm sure another developer will push you in the correct direction.

Thanks but I was there already and could not find on the mac os x platform a 'calibre-debug' script of any sort (ran a find . -name'calibre*' from from calibre root bzr branch)
Am I missing something ?

[KevinH]

Hi,

Many even large-scale projects build just fine on Mac OS X with only minor changes to what is done on Linux. I have built everything on Mac OS X including the original OpenOffice.org project which had a huge cpp codebase with its own completely unique build system. I have never used or needed fink / Mac OS X Ports, etc since they are just pure overhead for someone who can actually build from source properly and install their own libraries in /usr/local using the standard config, make, make install most projects use.

Packaging on Mac OS X is actually easier than on almost any other platform because it actually encompasses the program, icons, libraries, frameworks, etc needed into one .app which is actually just a glorified folder/directory with its own internal structure. No need to install libraries in special locations, update dylib environment variables, or worry too much about dependencies (since they can be packaged right inside the .app directory).

Packaging for Mac OS X can be easy - see Sigil's build tree for an example.

So again, in the next source archive, please include whatever scripts, hacks, makefiles, or even just a list of required steps, versions and dependencies as a text file currently used to complete the build from pure source into the package for Mac OS X. Regular Mac OS X developers can then help to simplify and enhance those scripts and make the build and packaging even easier.

That is the power of open source.

KevinH

Originally Posted by ldolse

I agree with you conceptually, but on the flip side Calibre's approach in this case actually encourages development and contribution more than any other open source projects I've worked on. The fact that you don't need to build a bunch of dependencies makes Calibre incredibly easy to contribute too.

I don't think it's a matter of secrecy that building the package from scratch is difficult, though Kovid may be better able to comment on that - I think it's just more about convenience and priorities.

I think it may also be partly Apple's fault - most open source projects completely fail at building easy to use Apple installer packages even when there are Windows and Linux binaries, and I think it's at least somewhat common that the source tree doesn't provide the capability to build a DMG package, but I may be wrong.

BTW, last time I built Calibre from source was on Linux - and that was very easy - the other issue with Mac is you often need a homebrew/Macports/fink recipe for each dependency in order to get a successful build - that can be incredibly tedious to put all those pieces together.

[DoctorOhh]

Originally Posted by fab672000

Thanks but I was there already and could not find on the mac os x platform a 'calibre-debug' script of any sort (ran a find . -name'calibre*' from from calibre root bzr branch)
Am I missing something ?

I have not run from source and I am unfamiliar with OSX, but could what you are looking for be part of the Command line tools? To install the command line tools, go to Preferences->Advanced->Miscellaneous.

[ldolse]

Originally Posted by fab672000

Thanks but I was there already and could not find on the mac os x platform a 'calibre-debug' script of any sort (ran a find . -name'calibre*' from from calibre root bzr branch)
Am I missing something ?

You won't get calibre-debug unless one of two things happen:

• Download and Install the Calibre OSX Package
• Successfully build Calibre from source

Dwanthny is correct that you need to install the command line tools so that Calibre installs symlinks to the inside of the Calibre.app directory wherever you happened to install it to.

[kovidgoyal]

Setting up an OS X build environment takes literally days of work for calibre. If you want to build calibre from source, run linux. I have absolutely zero interest in documenting build procedures for platforms that go out of their way to make building things difficult.

CALIBRE_DEVELOP_FROM provides a zero friction method of running calibre from source. Use it.

@fab672000: You do not need calibre-debug, as long as you have correctly set CALIBRE_DEVELOP_FROM, you can just run calibre normally. But if you do want it, on OS X, you need to install the command line tools via Preferences->Miscellaneous. A step that is clearly stated in the docs: http://manual.calibre-ebook.com/deve...nt-environment

@KevinH: The provided sources are fully buildable if you know a lot about the OS and python and a couple of dozen other things and are willing to to spend a couple of days doing it.