I'll make it easiest
Format shifting in the US is considered creation of a derivative work.
IF you do not have permission of the copyright holder to shift formats, then what you are doing is technically illegal.
An example of this is modern CDs do contain permission to rip for personal use (or in some cases will contain the MP3s themselves). If you check on record company websites , you see they give blanket licenses to do such.
Right or wrong in your or my opinion, you asked whether it was legal to download a book you own.
No.
It is not.