That slow update rate may be SDL waiting for the framebuffer device to report a ready status. I ignore that in my video and animation code, and just do updates at a rate that looks good (eink overclocking, of a sort)....
My hacked up SDL from back in December went MUCH faster. I may have commented the code that was waiting for fbdev ready...