calibre is designed to keep the entire contents of an ebook that it is
processing (except for images) in memory. This makes
transformations on the text and formatting easy enough to implement
that they actually end up getting implemented. The downside of this
approach is that calibre cannot handle very large documents. I
suggest you find a way to split up your document outside calibre,
or use some other tools to convert it.