If you're on Windows, run
WinDirStat on your library folder. It can be a little slow but it'll show you where the space is going.
I'd put money on the PDF files you mentioned.
For ePubs, you can run Polish > Losslessly compress images and see if that helps.