You can force fontconfig's searchpath via env vars to work around the config mismatches somewhat, without having to do anything crazier.
IIRC,
FONTCONFIG_FILE points to the "master" fonts.conf (i.e., what's usually /etc/fonts/fonts.conf). This can point to a bogus path if you just need to discard everything (which was my own use-case, but that was a pretty edge-casey one ;p).
There may be finer-grained approaches, check fontconfig's docs

.