Just a couple of notes:
1) All of LaTeXs (actually TeXs) layout algorithms are designed for a fixed page size. Now while in theory this should not be a problem, as you can pass in the page size at compile time, in practice almost all LaTeX macro packages make certain assumptions about minimum page size. Just one example: The RevTeX package used for formatting of articles to be submitted to the Physical Review Journals format the abstract assuming a large page size. Just try using the geometry package with a revtex file.
2) TeX (as opposed to HTML) is designed for multi-pass compiling. This is poses two problems:
a) Your hypothetical latex viewer will have to figure out how many times to run pdflatex on the source, not an easy task, witness the number of build systems for LaTeX
b) This will be really, really slow compared to laying out a reflowable format
3) Packaging a TeX distribution is no trivial task and the smallest Text distributions I've ever come across are still pretty large
4) The real improvements of formatting something using TeX come when you either position things by hand or write special purpose algorithms. Positioning things by hand is out in a reflowable context and writing custom algorithms in TeX is a nightmare, believe me, I've done it.
All in all, I doubt this will work. But don't let me discourage you from trying
