From my (little) experience, converting from PDF to ePub (and vice versa) is a PITA. Even with Caliber, stuff are getting funky, in a bad way.
Maybe you could copy/past all you PDF into Word and do a ePub from there?
Building it by hand via Sigil would be longuer but more efficient, I think!
Good luck!