Lucky you, GPT 6.1 Sol was released yesterday evening, so I asked it to invent an intermediate in-memory format with compressed blocks and an index (not unlike StarDict but with lz4 instead of gzip) and convert input to that format in a streaming fashion, and then also benchmark memory usage and performance against the baseline. I deployed it to
https://datyoma.codeberg.page/pbdt/v2-alpha/ but the source code is too horrible to publish. I still recommend using the CLI option though.