FBReader currently reads the entire file into memory during initialization of a new e-book. I don't know how much space this requires, but it is certainly more than the original file size.
On a Desktop machine (Linux or Windows), it should be possible to configure enough virtual memory to import any reasonably sized file with FBReader. Although there may be a 32-bit file/memory limit (typically 1 GB or more). However, I don't know how to increase the swap space under Windows. If you can send the file to the developers, they may be able to help. Try posting to the
FBReader Google Group (even if you can't supply the CHM file). It is also possible that FBReader is working as it should but importing a 200 MB file takes a long time even on a fast desktop. So I suggest leaving it for ~10 minutes to see if it eventually comes up.
On the iLiad, memory is very limited. On the Nokia 770 (and N800), it is possible to add swap space on a MMC card, but I have not heard of anything similar for the iLiad. I did hang the iLiad on a 4 MB CHM file with FBReader - the same file loaded fine on a Windows XP Desktop. I was able to read both a 4 MB and a 7 MB CHM file on a Nokia 770, although the 7 MB file took more than 5 minutes to import.