I don't think it is a Calibre issue. It is a computer "horsepower" issue.
I have never had a conversion take more than a few minutes (less than 10).
If your server is doing the conversions before it sends the file, it better have some "cajones" if you know what I mean.
Otherwise, just do the conversion ahead of time and offer several formats. Don't try to do them "on the fly".
Happy happy, joy joy.