While the garbage collection could be tuned, I don't think it would help. The actual rendering of the PDF seems to be the culprit, and that is part of the (C language) dependent library MuPDF. Some corner cases will probably always trigger a high memory usage, seems to be one of them.
Some day (TM) I hope I have the time to make the rendering backends work in subprocesses that can be better controlled in terms of memory usage. A bit like the Chrome/-ium browser does.