Okay, so I'm an idiot - I installed the plugin and didn't back up the collections.json file. Despite the fact that I haven't actually created anything yet (just ran the preview file) when I hit the "preview without saving collections" button I get:
calibre, version 0.7.44
ERROR: Create Kindle Collections plugin: Unable to open Kindle file: E:\system\collections.jso
When I look in the actual Kindle folder it doesn't even have a "system" folder, let alone the collections.jso file.
Does anyone have any suggestions? Feel free to call me a n00b for not backing it up!