I'm actually currently testing with "Programming the iPhone User Experience" which is an (old) Oreilly title, and it seems to be pretty quick in all cases.
When you change the font size or face, the eReader has to reload the chapter with the new font size/face so it can, among other things, give you a new chapter page count, etc.
If the chapters are somehow giving the device any trouble, it would make sense that performance would degrade in any scenario where the chapter is required to be loaded/reloaded. I would be very interested in finding out why this particular title is giving you problems. If you have some time, please email me at shasan at kobo dot com and we can dig in to your particular case.