Interesting. Is there some reason you chose to do the build on the Raspberry Pi instead of cross-compiling? I suppose setting up the cross compile environment is a fairly painful initial effort.
There are build scripts for automatically building calibre and all its dependencies including python and Qt here:
https://github.com/kovidgoyal/build-calibre
Someday I should get around to adding linux ARM builds to those scripts. All that's needed is creating a docker image setup for cross-compiling and adjusting a few of the build scripts as appropraite for cross compiling.