@mobusr: That's simply not how either CM, LS, nor the Calibre plugin are engineered (and for good reason, as knc1 gave an example of). And I stand by my point that the ends probably wouldn't justify the means
. (i.e.: way too much work to achieve a broken-by-design design that wouldn't net any severe speed advantages, but would net us severe downsides).
(Plus, the Kindle DB is *NOT* accessible over USBMS, which makes transparent sync unachievable without a proxy).
As for LS, without doing completely stupid things in the name of speed, the current major hotspot is ccat, and we have no real control over it. Bypassing it (which means reinventing a better wheel than Amazon's) and doing the SQL ourselves would *possibly* net some minor speed advantages, but would increase complexity and decrease maintainability of the code. You're welcome to try your hand at it, LS is modular enough to make that switch doable.
Note that if you use Librarian to manage your library & collections, the first point is moot, because the json will be generated on the library host machine, if I'm not mistaken. The ccat thing still stands, though.
(With a bit of creativity, the same effect can be achieved with the Calibre plugin, since you can use the exact same kind of variables to build the collections as you can to build the directory structure. With the added bonus that both Calibre and the plugin maintain their own cache to avoid IO-bound performance penalties on subsequent runs).
As for CM, there's an extra layer of indirection through the java framework, so who knows how that affects performance. Just try it with LS, it should already be a magnitude faster.
As knc1 ended on, either Librarian + LS, Calibre + Plugin + LS, or LS on its own is already smart enough to do what you want in an efficient manner.