Regarding your first sentence, describing what is there is not a claim about what should be.
Note that scroll mode is a misnomer. It should probably be renamed to something like continuous. I use "scroll mode" as the default, but I seldom or never scroll. Conversely, you can still scroll in paged mode if a page is longer than the screen. As far as I'm concerned if you're scrolling (panning) you're doing it wrong, but to anyone reading: you do you.
The simple fact is that it would take several minutes to preprocess a PDF or DjVu, while doing it only one page at a time is speedy but also has some very real disadvantages. For one page at a time your choice is either a longer page or a page that'd look like the end of a chapter every few pages. I think a longer page is the lesser evil.