Ditto for another O'Reilly book I purchased ( Secure Programming Cookbook for C/C++ isbn=9780596103897 ). In fact the symptoms were pretty much spot on compared with the OP. This book is just shy of 900 pages.
In my case I can confirm that it appears to be css related having tried a few tweaks like removing the fonts and references to them...
I finally opted to rename the core.css file within the epub package (it's just a zip file that I uncompress first) so that it wouldn't be used at all and response/load times are reduced from minutes to seconds; particularly with initial load, content navigation. Chapter changing and page turning is noticeably quicker too. There is some evidence of changes in formatting, as you might expect, but really wasn't too much different and still very readable.
Incidentally, removing the fonts reduced the epub file size from 3,431KB to 1,144KB which is a nice saving too.