The main problem with annotations is that there are all sorts of standards, depending on which format you use.
PDF should be fine, at least, since PDF annotations are usually embedded in the PDF itself. Note that if you use calibre's Open With to open a PDF, it will open the copy in the library, and the annotations will be saved there. If you use the PDF on your iPad, you will need to add the iPad-updated copy of the PDF to the book record in calibre. Edit Metadata ==> Add format. It will
automatically replace the current copy of that format with the newly-added copy of that format.
...
On the topic of cloud syncing, copying the entire Library folder is exactly what you need to back up your library. You can also backup the configuration directory if you want to preserve plugin/GUI settings, but it isn't necessary.
You can use Google Drive, as long as you sync a copy of your library into your Google Drive folder. Do NOT keep the only copy of your library in Google Drive!
If something happens to your library, it will require some work to reimport the books backed up in Google Drive, but at least you'd have the books, right?
Though I would still recommend finding another service like Dropbox or Box.com or Copy.com, many of which offer referral programs. Though I believe Copy.com is closing down.
See my Dropbox referral link? I have collected 17.38 GB of storage because of referrals on MobileRead.