Well, ... technically, ... now that you mention it, ... you can already.
If you don't mind the caveats.
1: like MP3 files the MP4 and FLV files do not show up in the Add Files list unless you set the File Type filter to All Files.
2: like MP3 files the MP4 and FLV file will be copied into the Calibre Database.
So, if you don't mind the duplicate file in the file system, then go for it.
3: like MP3 files, MP4 files can be put into an M3U file.
[ the spec doesn't say so, but Media Player doesn't mind at all ]
Then you get the management without the duplication.
But you better not move / delete / rename the file, or the M3U is broken.
3.5: M3U files for Calibre use _must_ use absolute paths or URLs.
The reason this works is that many file formats use the same tagging structure internally. Many Audio Books formats are just special cases of Audio Video Files formats. (same structure but without the video stream).
The AudioBookReader plugin can actually read metadata from file four different file formats:
* M4B, M4A, MP4, 3GP, 3G2, QT
* AA, AAX
* F4B, F4A, FLV
and three variations of the play list file format:
* M3U, M3U8, RAM