It most definitely isn't a good idea
WHat exactly are you trying to use cairo for? You might me able to do whatever it is with pypdf, poppler or podofo, all of which are distributed with calibre. And use Qt to render svg instead of rsvg, which is also distributed with calibre.