Well, that kind of depends on what format you'd like to use, doesn't it? The iBook library, to my knowledge, is in ePub, I'd think
Sigil is your best choice for editing that. For PDF, personally, I'd use LaTeX, but that's quite complicated- maybe anyone else has a better idea for that.