Not easily. Probably the only thing you will be abe to do is call calibre command line applications (such as ebook-convert) and let them run then get the output:
Code:
import java.io.*;
Process p = Runtime.getRuntime().exec("ebook-convert.exe");
p.waitFor();
System.out.println(p.exitValue());
The other option you have if is you want to interact with the calibre library for instance then you don't need calibre and can work with the metadata.db and files themselves.