The key word here is "almost".
There are two caches at work: the document viewer's cache, and the plugin's cache. The plugin loads and pre-renders pages ahead and behind the page you are on in the current document. The document viewer will cache pages it has requested from the plugin as well, for /all/ documents. The most recently viewed pages remain in memory, while the older ones get purged as more memory is needed.
So whenever you load a document after not viewing it in awhile, the pages will need to be re-rendered and given to the viewer by the plugin... since neither the plugin or the viewer has a copy.
Say it takes about 1.2 seconds to render each page in a specific document. When you load it, part of the 2-3 seconds to load is rendering that first page. Now, you spend 2 seconds skimming the page and hit next. It should be pretty much instant to start the refresh, since page 2 is pre-rendered. But if you start spamming the next button as fast as the screen can refresh, you aren't letting the 1.2 seconds required to load the next page pass by. In that case, you will get a little lag. So the lag will be hit more if you skim comics, than if you spend a few seconds reading each page. You can test this by loading a document, waiting 30 seconds or so, and then spamming next. The first couple pages will be instant, and then you will start hitting lag.
Now, this 1.2 seconds is an arbitrary number, and can be affected by the file format you use to store the image, size of the file outside of a ZIP, and even in some cases, some /variation/ of the file format you use to store the image. A couple rules of thumb to avoid slow performance is to use JPEG over PNG, and avoid using Progressive JPEG. Large PNGs and Progressive JPEGs are both slow to load. PNGs can get in the 2-4 second range when they get large, and even 1MB PNGs load slower than a JPEG of the same file at half the size, despite the simpler compression. Progressive JPEGs start in the 5 second range, and easily pass 10 seconds /per page/. And unfortunately, there isn't anything I can do to address it in the plugin.