Surprising post: your observation is opposite to the usual experience and the official FAQ
https://manual.calibre-ebook.com/con...-pdf-documents . Converting to PDF usually works better than PDF to anything.
Now if your PDFs produce something reasonable, by all means, go for it :-) However, you might have more control on the output if you convert to a fully editable format (docx, odt or RTF), fix all issues using your word processor of choice, and then convert to epub.