Regarding formulas, only a few basic mathematical functions can be expressed in HTML which will resize like the rest of your text in an e-book format: this page here for which symbols you'll be able to show successfully.

For more complicated formulas, you'll probably be better off just creating an image of the equation and inserting it into the text like a normal picture or graphic, with the typical "img" HTML tags.

Think of Sigil as primarily a text-editor tool which can insert the occasional picture: it can't create Excel-type graphs, and it can't build complicated equations like MatLab, but it can display any images or screenshots you feed it.

Ultimately, though, the end result will depend on how well the images will display on your e-reader (and if you have a lot of formulas or graphs with very small font, whether you have a zoom-in/zoom-out capability on the device). I'm afraid my own expertise is with the Kindle, not the PocketBook, but these forums are a great resource - I'm sure someone here will be able to tell you which picture sizes will fit your e-reader screen resolution the best.

Hope that helps!