Thanks!! that did it.
Related to this, I think I found a bug. When I set prefix=/ebooks or whatever, and use /ebooks/ to get to the home page the links to the categories always get prefixed with /calibre instead of /ebooks resulting in page not found. But when I go to /ebooks/browse, the prefixing works. Seems specific to running with wsgi as I haven't noticed this problem with reverse proxying.
Thanks for all the work going into calibre!! My Sony reader would be worth a lot less without it